Last week I got copied into a tweet by Rick Garner. The tweet was from Daniel Lawson, and he had an issue with AWS Elastic Beanstalk. After exchanging a few messages I jumped on a call to debug the issue a little further. Danny was deploying his .NET Core 3.1 through the AWS Toolkit for Visual Studio to the .NET core on Linux Platform on Elastic Beanstalk and had a 502 error after deployment.
The error appeared to be thrown by NGINX.
After some investigation, I discovered this error was caused by the .NET core application failing to start-up.
With Elastic Beanstalk you can retrieve the Logs for your instances through the console. If I select the Request Logs dropdown and then Full Logs it gets the logs from all the instances that make up the Elastic Beanstalk Environment.
I only have one instance and so I get one set of logs. If I download the zip file it contains a file called web.stdout.log this is the output of the .NET logs and I can see there is a line that says:
Unhandled exception. Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementException: User: arn:aws:sts::xxxx:assumed-role/aws-elasticbeanstalk-ec2-role/i-xxxxx is not authorized to perform: ssm:GetParametersByPath on resource: arn:aws:ssm:eu-central-1:xxxx:parameter/app
This is causing an issue with the application. On startup, the app retrieves some configuration from AWS Systems Manager Parameter Store. Since the app doesn’t have permission to use the ssm:GetParametersByPath API, we get an unhandled exception, and the app fails to start.
When I go to the URL of the web app, NGINX (which is used as a proxy requests from the web to the .net core app) sends the requests to port 5000 on the instance, but sadly the app is not listening on the port, because it has failed to start due to the exception.
To fix this, I need to give the app permission to speak to AWS Systems Manager Parameter Store. I go to the IAM console and look for the aws-elasticbeanstalk-ec2-role (which is the name of the role I use for Elastic Beanstalk), and then click on Attach policies.
I filter for the policy AmazonSSMReadOnlyAccess, select the policy and then click Attach policy.
My application is now able to speak to AWS Systems Manager Parameter Store and retrieve my configuration. Ordinarily, updating IAM would immediately take effect, but since my application has failed to start, I need to restart it. To do this, I redeploy the app (there are other ways).
502 gateway errors, probably mean your app has crashed. To solve this, check your logs and look for startup errors, and to be more resilient, it’s probably good practice to avoid unhandled exceptions at startup, which could crash the app.