API Gateway and Service Proxy with SQS
We recently came across a situation where we had to integrate into a 3rd party SMS gateway solution. The integration process was quite straight forward. We had a challenge when we needed to capture the delivery reports. Our solution has the potential to push out over 500K messages for a given instance. The 3rd party vendor uses a call back hook into our application to send us the status of the message.
Well on the providers end its fine for them ( I guess ) 500k outgoing GET requests. But for us its a massive hit given that we can have many instances of different tenants pushing out messages over 500k. This ends up with a exponentially growing number of hits on our web servers that could potentially result in a unintentional DoS attack.
We had multiple alternatives ways to tackle this so we could off load our Web servers from being effected. I choose to offload everything into AWS SQS. That way the delivery reports , which is not mission critical can sit in a queue until one of our workers process them off.
First option was to open up , SQS endpoint and provide it as the call back hook. This would have been fine but the vendor send us two query params. Which potentially makes SQS drop one of them
As you can see , the MessageBody will end up dropping the Status param. Unless the vendor does a URL encoding which they refused to do.
So in order to handle this , I decided tom look into API Gateway and Lambda. But I realized that API Gateway provides a decent services that they call as a Service Proxy. Which was ideal , since this would provide us with the opportunity to mask the service URL’s , even monitor the callback hook responses easily.
So it was quick setup on the AWS API Gateway console. Just that it didn’t work. I setup the flow as following :
GET Method will receive the query params MID and Status
The Integration Request will pass it into a SQS queue in my region
The Integration Request was a bit tricky. AWS’s mapping templates is quite confusing and lacks uses cases and examples for different instances. It supports only application/json content type. And wouldn’t let me specify my Content-Type and Accept Headers.
SQS was being an ass and wouldn’t accept json when calling the API. The request had to be with the Content-Type : application/x-www-form-urlencoded
I was hitting a wall over the weekend trying to figure this out. And finally came across this post. According to that , I was able to push out static values on API gateway console’s Query / Path and Header params. I just had to simply enclose them in single quotes. This was a life saver. Below is how my Integration request looks.
If you look closely you would see the HTTP Header and that I have simply set a non json format for the mapping template but set the content type as application/json. The HTTP Header overrides the mapping template’s content type and my endpoints work as a charm now.
I hope this gets documented properly with AWS. Being able to offload directly outside the main applications queue , is a life saver. Most applications can save a lot of CPU spikes and memory outage by letting your workers handle the load async. So regular users of the application stays uninterrupted.