Skip to main content
Logo
Overview

AWS SNS to http subscription receiving in python3 http server and Flask

July 23, 2021
4 min read

I wanted an easier way to test and manipulate a notification using an AWS SNS subscription. Mostly I do a quick SMTP subscription to the topic. I wanted quicker and more direct feedback and also manipulate the incoming notification. I used this as reference https://gist.github.com/iMilnb/bf27da3f38272a76c801

NOTE: the code will detect if it is a subscription or notification request and route appropriately.

run a server on port 5000

Terminal window
$ mkdir snsread
$ cd snsread/
$ vi snsready.py
$ python3 -m venv venv
$ . venv/bin/activate


(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install Flask
...


(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install requests
...


(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

test using curl from public client

Terminal window
$ curl -I ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Fri, 23 Jul 2021 21:38:20 GMT

when testing curl request python server shows

99.122.138.75 - - [23/Jul/2021 21:38:20] “HEAD / HTTP/1.1” 200 -

when testing publish direct from SNS topic the python server shows

205.251.234.35 - - [23/Jul/2021 21:41:26] “POST / HTTP/1.1” 200 -

Add subscription in topic rr-events-02 as http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000

server shows during subscription

205.251.234.35 - - [23/Jul/2021 21:41:26] “POST / HTTP/1.1” 200 -

topic > publish message

server shows

Terminal window
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


NOTE: reference code was not exactly matching my specific notifications so need some tweaking  as well as some json.loads(json.dumps())) love.

initial successful from actual Cloudwatch alarm sent to SNS

Terminal window
(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py
 * Serving Flask app 'snsread' (lazy loading)
...
incoming...
headers:
json payload:
js: {"AlarmName":"linux-server-errors","AlarmDescription":"ERROR in /var/log/messages","AWSAccountId":"310843369992","NewStateValue":"ALARM","NewStateReason":"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 00:29:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).","StateChangeTime":"2021-07-24T00:34:13.630+0000","Region":"US West (Oregon)","AlarmArn":"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors","OldStateValue":"OK","Trigger":{"MetricName":"messages-errors","Namespace":"messages","StatisticType":"Statistic","Statistic":"MAXIMUM","Unit":null,"Dimensions":[],"Period":300,"EvaluationPeriods":1,"ComparisonOperator":"GreaterThanThreshold","Threshold":0.0,"TreatMissingData":"- TreatMissingData:                    notBreaching","EvaluateLowSampleCountPercentile":""}}
205.251.233.161 - - [24/Jul/2021 00:34:13] "POST / HTTP/1.1" 200 -

Publish message directly from topic in console

Terminal window
(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989


incoming traffic...
*********************
data
----
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 149d65b0-9a32-524a-95ac-3cc5ea934e04
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 946
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate




json payload
------------
None
js
--
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.187 - - [24/Jul/2021 01:36:11] "POST / HTTP/1.1" 200 -

from linux server custom json

Terminal window
[rrosso@fedora ~]$ curl -i -H "Content-Type: application/json" -X POST -d '{"userId":"1", "username": "fizz bizz"}' http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Sat, 24 Jul 2021 00:59:14 GMT


OK

server shows

Terminal window
(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py
 * Serving Flask app 'snsread' (lazy loading)
....


incoming traffic...
data:
{'userId': '1', 'username': 'fizz bizz'}
headers:
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
User-Agent: curl/7.76.1
Accept: */*
Content-Type: application/json
Content-Length: 39




json payload: {'userId': '1', 'username': 'fizz bizz'}
99.122.138.75 - - [24/Jul/2021 00:55:44] "POST / HTTP/1.1" 200 -

initial successful from actual Cloudwatch alarm sent to SNS. injected to server messages with logger

Terminal window
[root@ip-172-31-6-74 ~]# logger "ERROR: WTF6 is going on..."

server shows

Terminal window
(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py
 * Serving Flask app 'snsread' (lazy loading)
...


incoming traffic...
*********************
data
----
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 7ec65853-62c5-5baf-9155-01261344a002
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 1901
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate




json payload
------------
None
js
--
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.240 - - [24/Jul/2021 01:39:34] "POST / HTTP/1.1" 200 -