Little Thing on on AWS Lambda, AWS SNS, and Slack
AWS Lambda can be used in many places. As a fully managed service, it is best fit for light-weight, short and easy task such as event trigger, s3 data filter and migration, and recently a rather interesting self-assigned task slash hobby, status reports on slack. The infrastructure is also light-weight.
Major thing I want to achieve here,
A configuration-based notification, i.e. if there is new notification need, I do not want to create a new event, re-deploy a new lambda code. As a startup that is short of hands, I want to minimize the effort of hooking new notification line to …. let’s say, simply update a json file on s3 without endangering my existing notification line.
General goal is set, our next research topic would be Slack. An application as such definitely provides some sort of API to access its functionalities. Since Slack is popular among distinct business units worldwide, it is reasonably to believe that API is controlled by advance security settings. Advance security settings is a big word that can be translated as “complicated token scope mechanism”. Now my head started spinning just to think about the infinite scrolls of document to read before I can start typing my first line of code.
Luckily, Slack has an easy way out for my simple project here. The scenario in this case was to send a message to a slack channel when someone send a message to AWS SNS. There is nothing smart about this behavior. There is no talk back with an attitude bot. It only delivers a message as told. Hence no need of a Slack sdk to invoke a client object before proceeding on the talking part. A webhook, an api service to receive notification sent by AWS Lambda is good enough. Slack webhook document is a peach as it provides clear step-by-step instruction, (https://api.slack.com/messaging/webhooks#posting_with_webhooks).
Without further ado, let’s dive into the world of AWS to set up this scheme. First of all, we need an SNS topic. AWS SNS Topic serves as an intermittent service between sender and receiver. There are few options we need to think about before creating SNS topic. There are two types of SNS topics. In this scenario, the order the message is received is not crucial, Standard typed SNS is suffice. Although display name is optional, setting up a display name makes email subscription more clear for email subscriber. Display name shows on sender field in an email.
Another important section for SNS topic is access policy. Access policy controls who is allowed to send message to SNS topic. Check “Only the specified AWS accounts” under “Define who can public message to the topic” and enter your AWS account id into the field to restrict access of the SNS topic to everyone from the same AWS account. Do the same thing for “Define who can subscribe to this topic” section to apply the same restriction for subscription privilege.
When a message is post to SNS topic, a Lambda function is triggered. Lambda function is set with two purposes,
1. Decipher SNS message.
2. Forward the message to Slack webhook based a configuration file from S3.
Configuration file should look something like this,
[
{
"job_name": "job name",
"webhook_url": "https://hooks.slack.com/services/xxxxxxx"
}
]Here comes a tricky part. In order to read file object from s3, we need to read and process in memory. We use Python 3.8+ in this tutorial, boto3 library is what we need.
import boto3s3 = boto3.client('s3')
data = s3.get_object(Bucket='xxx', Key='xxx/config.json')
“Body” in data object holds the content of configuration file, we can access content of the configuration file as such,
import jsonconfiguration = json.loads(data["Body"].read())
After successfully loading json configuration file, we want to loop through and find a match for sent message. Say if someone post a message like this,
import boto3
import jsonsns = boto3.client('sns')
message = {
'default': {
'job_name': 'job name',
'message': '{"text":"success"}'
}
}sns.publish(
TopicArn='xxxxxxx',
MessageStructure='json',
Message=json.dumps({'default': json.dumps(message)}),
Subject='xxx is downloaded'
)
We want our slack to show like this,
Lambda code would generally look as such,
Another challenge we face here is the “request” module that is not provided in pre-compile Lambda environment, i.e. we need to wrap up “request” library along with Lambda python file for Lambda to work. We have the choice of using zipped python package or a container image. In this tutorial, we choose to zip up python module along with our lambda code.
Creating a python package is as easy as
pip install --target ./package requests==2.22.0Two things to remember before uploading our zip file to AWS,
- AWS Lambda uses POSIX file system which only means file needs to be set up with proper permission before zipping them up. Using these two commands to update file permission accordingly.
chmod 644 $(find . -type f)
chmod 755 $(find . -type d)2. Zip file structure is important. All module folder should be in the root of zip file as well as lambda python file. File structure for zip folder should look like this,
Now that we are all set on Lambda, we can place Lambda zip file on s3 and create a Lambda function using aws cli that points to s3 location.
We can upload lambda python file and config.json using aws cli as such,
aws s3 cp lambda.zip s3://xxxx/
aws s3 cp config.json s3://xxxx/And create a lambda function using cli,
aws lambda create-function \
--function-name "Lambda Function Name" \
--runtime "python3.8" \
--code '{"S3Bucket":"Bucket", "S3Key":"zip file S3 key"}' \
--role "role arn" \
--handler lambda.handler \
--region "ap-northeast-1" \
--environment file://lambda_environment.json \
--publish \
--output "json" > log.jsonOur Lambda function is ready on AWS now, we can wrap up this tutorial by creating a subscription for the topic. Each time a new notification requirement is initiated, configuration file modification is all we need to perform.
