[Backend #29] Store & retrieve production secrets with AWS secrets manager

Aug 16, 2023

[Backend #29] Store & retrieve production secrets with AWS secrets manager

In this lecture, we will learn how to use another service: AWS secrets manager to manage the environment variables and secrets for our application.
- Join us on Discord: https://bit.ly/techschooldc
- Get the course on Udemy: https://bit.ly/backendudemy
- Buy us a coffee: https://www.buymeacoffee.com/techschool
- Full series playlist: https://bit.ly/backendmaster
- Github repository: https://github.com/techschool/simplebank
---

In this backend master class, we’re going to learn everything about how to design, develop, and deploy a complete backend system from scratch using PostgreSQL, Golang, and Docker.

TECH SCHOOL - From noob to pro
/ techschoolguru
At Tech School, we believe that everyone deserves a good and free education. We create high-quality courses and tutorials in Information Technology. If you like the videos, please feel free to share and subscribe to support the channel or buy us a coffee: https://www.buymeacoffee.com/techschool

Content

0 -> Hello everyone, and welcome to the backend master class

3.28 -> In this lecture, we will learn how to use another service: AWS secret managers

8.24 -> To manage the environment variables and secrets for our application.

12.96 -> If you remember, in the previous lecture,

15.44 -> We have setup a production database on AWS RDS,

19.44 -> And this is the URL we used to access it.

22.48 -> When we deploy the simple bank app to production,

25.2 -> we would want it to connect to this database.

28.08 -> So basically, we must replace the DB_SOURCE variable in the app.env file

32.96 -> with the real production DB URL.

35.52 -> Also, we need to generate a stronger token symmetric key.

39.04 -> We should not use this trivial and easy-to-guess value, right?

42.88 -> So, the idea is, in the github actions deploy workflow,

46.64 -> Before building and pushing the docker image to ECR,

50.16 -> We will replace all environment variables in the development app.env file

54.56 -> with the real production values.

56.64 -> By doing so, when we run the docker container on the server later,

60.56 -> It will have all the correct settings for production environment.

64.24 -> OK, but the question is, where do we store the values of these environment variables?

69.68 -> Of course we cannot put them directly in our github repository,

73.12 -> because it would be very insecured, right?

76 -> One good solution is to use AWS secret manger service.

80.08 -> This service allows us to store, manage, and retrieve any kind of secrets for our application.

85.68 -> It is pretty cheap, just 0.4 dollars per secret per month,

90.16 -> And only 0.05 dollars per 10,000 API calls,

94.48 -> And we also have 30-day free trial period.

97.6 -> As we have planed to use AWS to deploy our app,

100.88 -> It totally makes sense to take advantages of this secret manager service.

105.68 -> OK, let’s create a new secret!

108.48 -> There are several types of secret,

110.64 -> such as credentials for RDS, DocumentDB, Redshift, or other databases.

116.8 -> In our case, we want to store not only the DB credentials,

120.48 -> but other environment variables as well,

123.12 -> So I’m gonna choose “Other type of secrets”.

125.52 -> Then in the next section,

127.44 -> we can add as many key-value pairs as we want.

131.2 -> First, let’s add the DB_SOURCE.

133.68 -> I’m gonna copy the production RDS database URL from the Makefile

138 -> And paste it to this input textbox.

140.64 -> Then let’s click “Add row” to add a new pair of key-value.

144.72 -> The key will be DB_DRIVER,

147.12 -> And the value will be postgres.

149.44 -> Next, a new row for the SERVER_ADDRESS

152.16 -> Its value will be the same as in development: local host port 8080.

157.28 -> Note that this is just the internal address of the container.

161.12 -> Later, when we actually deploy the app to kubernetes,

164.32 -> We will learn how to setup a load balancer and domain name

167.36 -> that will route the API requests to the correct container’s internal address.

172.32 -> OK, now let’s add one more row for the ACCESS_TOKEN_DURATION

176 -> Its value will be 15 minutes.

178.56 -> And finally, the last row for the TOKEN_SYMMETRIC_KEY.

181.92 -> Its value should be a 32-character string,

185.12 -> There are many ways to generate a random string of 32 characters,

189.28 -> Today I’m gonna show you how to do it using openssl command.

193.28 -> It’s pretty simple, we just run: openssl rand

197.28 -> Use the -hex flag to tell it to output a string of only hexadecimal digits.

202.64 -> And finally the number of bytes, let’s say 64 bytes to make a very long string.

208.4 -> Actually this string has 128 characters

211.52 -> because each hex digit only takes 4 bits, or half a byte.

215.92 -> In our case, we only need 32 characters,

218.72 -> So here we use the pipe to chain its output to the next command:

222.56 -> head -c 32, which means, just take the first 32 characters.

228.08 -> And voilà, we now have a random string for the token symmetric key.

232.24 -> Let’s copy and paste it to the secret manager.

235.52 -> OK, now we have added all necessary variables to the secret.

240.08 -> There’s an option to select a custom AWS KMS key to encrypt the data,

245.2 -> But it’s not mandatory.

246.72 -> We can just go with the default encryption key for now.

250.08 -> In the next step, we have to give our secret a name,

253.12 -> so that we can easily refer to it later.

256.24 -> Let’s call it simple_bank.

258.32 -> You can also write some short description

260.88 -> to help remind you about what values this secret stores.

265.12 -> Optionally, we can add some tags

267.52 -> to make it easier to manage, search or filter AWS resources.

272.24 -> And an option to set permissions to access the secret.

276 -> But we can skip all of them for now.

278.4 -> So let’s click Next!

280.4 -> In this step, we’re able to enable automatic rotation for our secret values.

286.16 -> Simply put, we can set a schedule and a lambda function,

290.08 -> Then the secret manager will automatically run that function

293.36 -> to change the secret values when the time comes.

296.48 -> You can use this to frequently update your DB password, or token symmetric key if you want.

302.64 -> To keep this lecture simple, I’m just gonna disable automatic rotation.

307.68 -> In the last step, we can review all the settings of the secret,

311.76 -> AWS also give us some sample code for several languages,

316.24 -> In case you want to fetch the secret value directly from your code for example,

321.12 -> You can use this template, and download the appropriate SDK to do so.

326.64 -> We don’t need to do that in our case, so let’s click Store to save the secret.

332.16 -> And voilà, the secret is successfully created.

336.16 -> In this page, we can click this button to show all the content stored in the secret.

342.4 -> Now the secret is ready, we will learn how to update the Github deploy workflow

347.36 -> to retrieve the secret values and save them to the app.env file.

351.92 -> To develop this feature, I think we will need to install the AWS CLI.

357.2 -> It is a very powerful tool to help us easily interact with the AWS services

362.32 -> via API call from the terminal.

365.76 -> You can choose the suitable package depending on your OS.

369.84 -> I’m on macOS, so I will click on this link to download the installer package.

375.52 -> Then open it to start the installation.

378.16 -> And follow the instructions on the UI.

381.6 -> OK, now the AWS CLI package is successfully installed.

385.6 -> We can run these 2 commands to verify that it is working properly

389.84 -> which aws,

391.36 -> And aws --version

393.52 -> All looking good.

395.12 -> Next, we have to setup the credentials to access our AWS account.

399.44 -> To do that, just run aws configure

402.88 -> We will be asked for the access key ID and secret.

406.8 -> So let’s go back to the AWS console and open IAM service.

412.8 -> Here, we can see the github-ci user that we’ve set up in one of the previous lecture.

418.16 -> On the security credentials tab,

420.56 -> we can see the Access Key ID that’s being used by Github Action.

424.56 -> But for security reason, we cannot see its Secret Access Key.

429.12 -> So we have to create a new one to use locally.

432.64 -> Let’s copy this access key ID

435.04 -> And paste it to the terminal

437.2 -> Next, it will ask for the secret access key,

440.08 -> Let’s show the value and copy it.

442.48 -> Then paste it to the terminal.

444.48 -> It will ask for a default region name.

447.76 -> I’m gonna put eu-west-1, because it’s the main region I’m currently using.

453.36 -> And finally, the output format.

455.84 -> It’s the data format we want AWS to return when we use the CLI to call its API.

461.44 -> For me, I’m gonna use JSON format.

464.72 -> Alright, now if we look at the .aws folder,

468.32 -> We will see 2 files: credentials and config.

472.56 -> The credentials file contains the access key id and secret that we’ve just entered before.

478.8 -> The default at the top of the file is the name of the AWS profile.

482.88 -> You can add multiple AWS profiles with different access key to this file if you want.

488.32 -> The default profile is, of course, the one you will use by default,

492.32 -> Which means, if you don’t explicitly specify a profile name when running a command,

497.2 -> then this default profile’s credentials will be used.

501.44 -> Similarly, the config file contains some default configurations,

505.76 -> In our case, it’s the default region and output format that we’ve entered before.

511.76 -> OK, now we can call the secret manger’s API to retrieve our secret values.

517.68 -> You can run aws secretsmanager help to read its manual.

522.88 -> The sub command we’re gonna use is get-secret-value

526.64 -> So let’s run the help command again with get-secret-value to see its syntax.

532 -> Basically, we will need to pass in the secret ID of the secret we want to get

536.96 -> It can be either an ARN (Amazon resource name),

540.64 -> Or the friendly name of the secret.

543.44 -> You can find the ARN of the secret in its AWS console page.

547.92 -> The friendly name is the name we set when creating this secret, which is simple_bank

553.36 -> Alright, now get back to the terminal

555.92 -> And run aws secretsmanager get-secret-value

560.72 -> Then --secret-id simple_bank

563.52 -> Oops, we’ve got an error:

565.36 -> The github-ci user is not authorized to perform this request.

569.52 -> That’s expected, because we haven’t grant permissions

572.56 -> To allow this user to get the secret value yet.

575.76 -> In the IAM page, we can see the github-ci user is in the deployment group,

580.8 -> Which only has permission to access the Amazon ECR service.

584.88 -> What we need to do now is to give this group access to the Secret Manager service as well.

590.16 -> So, in this user group’s page, let’s open the permissions tab

594.64 -> Then click Add permissions, Attach Policies.

598.96 -> Search for “Secret” in this filter box.

601.68 -> Here it is!

602.8 -> Let’s select this SecretManagerReadWrite policy

605.92 -> And click Add permissions.

608.24 -> Alright, now all users in this group should have access to secret manager service.

613.76 -> Let’s go back to the terminal

615.44 -> And run the get secret value command.

618.16 -> We still get access denied exception.

620.88 -> I think the permission we’ve just added needs sometime to be effective.

625.36 -> So let’s wait a bit,

627.28 -> And let’s try using the secret ARN instead of the friendly name

631.52 -> OK, now the call is successful.

633.92 -> I’m gonna try again with the friendly name: simple_bank

637.44 -> It’s also successful this time.

639.12 -> Cool!

639.6 -> As you can see, the result is in JSON format.

642.96 -> And we have several more information than just the values stored in the secret.

647.84 -> What we need is only the value stored in this “SecretString” field.

652.08 -> To get only this field, we just have to add a --query argument to the previous command

657.28 -> And pass in the name of the field: SecretString

660.72 -> Voilà, now we see only the data stored in the secret,

664.72 -> However, its value is in the form of a string, not a JSON object.

669.92 -> We have to add one more argument: “--output text” to the command

674.16 -> in order to get the output value in JSON format as you can see here.

678.4 -> OK, but now, how can we transform this JSON object

682.16 -> into the environment variable format to store in the app.env file?

686.96 -> Well, there’s a very nice tool called “jq” that will help us with this problem.

692.64 -> JQ is a lightweight and flexible command-line JSON processor.

697.28 -> Let’s see how to install it.

699.28 -> On Linux, jq is already available in the official Debian and Ubuntu,

704.48 -> So we don’t need to do anything.

706.32 -> But it’s not available on MacOS by default, so we have to install it.

711.28 -> In the terminal, let’s run: brew install jq

714.8 -> While waiting for brew to install the package, let’s take a look at its manual.

719.84 -> There are many built-in filters and operators

722.96 -> that we can use to transform the input JSON data.

726 -> The most basic one is identiy, represented by just a dot.

730.4 -> This filter just returns whatever it takes as input unchanged.

735.2 -> Then we have the object identifier index,

737.92 -> Or a dot followed by the name of the field we want to get.

741.84 -> For example, here we run jq ‘.foo’

745.12 -> So with this input JSON,

747.04 -> it will return the value of the field “foo”, which is 42

751.12 -> If the field doesn’t exist as in this example, it will return null.

756.56 -> There are many many other filters, commands, and syntaxes,

760.08 -> Which I think you can discover on your own if you want.

763.92 -> OK, back to the terminal.

765.92 -> Looks like jq has been installed successfully.

769.2 -> Let’s run jq --version to verify.

771.84 -> OK, it’s version 1.6

774.32 -> Now, let’s run the command to fetch the secret values as JSON object

778.64 -> We will have to chain the output of this command with jq

781.92 -> to produce the final desired output file.

784.88 -> First, I want to convert this key-value JSON object into an array,

789.2 -> Where each object will be 1 separate environment variable.

793.04 -> To do that, we will use the to_entries operator of jq.

797.68 -> As you can see in this example,

799.92 -> It transforms 1 object with 2 keys a, b into 1 array of 2 objects.

806.4 -> Each object has a key and value field.

809.92 -> That’s exactly what we want,

812.24 -> So here I’m gonna chain the get secret value command with jq ‘to_entries’

817.92 -> Voilà, now we have 5 different objects, each stored 1 separate environment variable.

824.32 -> Next, we have to iterate through them,

826.48 -> and transform each object into the form of key=value,

830.8 -> since that’s the final format we want to store in the app.env file.

834.96 -> For this kind of transformation, we’re gonna use the map operator.

839.12 -> The way it works is very similar to the map function in python or ruby.

843.92 -> Basically, it iterates through the list of values,

847.44 -> Apply a transform function on each of them,

850 -> And return a new list of the transformed values.

853.2 -> So here, in our command, we can chain this to_entries operator with map

858.64 -> And let’s say if we only want to get the key of the object,

862.48 -> We will use .key as the transform function.

866 -> Then voilà, we’ve got a new array of strings with all the keys.

870.64 -> Similarly, we can get an array of strings with only the values

874.48 -> by using .value as the transform function.

877.68 -> OK, but what we want is a string contains both key and value, separated by an equal sign.

884.56 -> For that, we will need the string interpolation operator.

888.48 -> Basically, it allows us to put an expression inside a string,

892.56 -> by using a backslash followed by a pair of parenthesis.

896.72 -> As in this example, the first expression will just be replaced by the input value,

902.24 -> While the second one will be the input value + 1.

905.6 -> In our case, for this map function, we will first wrap it as a string

910.72 -> Then the first interpolation expression should be .key

914.64 -> Followed by an equal sign,

916.72 -> And then the second interpolation expression will be .value

920.88 -> OK, now we’ve got an array of 5 strings,

923.52 -> each store 1 environment variable in the desired format.

927.12 -> But in the final output, we must get rid of the array,

930.88 -> So to do that, we will use the array object value iterator.

935.12 -> Or a dot followed by a pair of square brackets.

938.72 -> As you can see in this example, by using this,

942.08 -> the array will be gone, and only the objects are printed.

946.64 -> Let’s try it!

948 -> I’m gonna add a pipe chain here, followed by the array iterator.

952.88 -> Now you see, only the 5 strings remained,

956.08 -> the array square brackets and the commas are gone.

959.44 -> The last thing we need to do is

961.28 -> to get rid of the double quote characters surrounding the string.

965.44 -> For that, we just need to pass in the -r (or raw_output) option to the jq command.

971.52 -> With this option, the result strings will be written out without quotes.

976.4 -> Let’s try it!

979.52 -> Awesome, now the output looks exactly as we wanted it to be.

984.32 -> All we have to do is to redirect this output to overwrite the app.env file

990.16 -> Just like that!

991.92 -> OK, let’s check the file to see how it goes.

995.28 -> Excellent!

996.08 -> The whole file content has been replaced with the production environment variables

1000.88 -> Exactly as we stored in our secret.

1003.68 -> The next step we must do is to plug this command to the github CI deploy workflow

1009.04 -> before building the docker image.

1011.44 -> But first, I need to reset all the changes we’ve made to the app.env and Makefile.

1017.12 -> Let’s run git checkout . in the terminal.

1020.4 -> OK, now all the content of the files has been reset to the original version on master branch.

1026.96 -> I’m gonna create a new branch ft/secrets_manager to make new changes.

1032.64 -> In the deploy.yml file, let’s add a new step

1036.4 -> Its name will be: Load secrets and save to app.env

1040.64 -> And it will run the commands that we’ve prepared before

1043.6 -> to fetch the seret values from AWS, transform them, and store in the app.env file.

1049.6 -> We don’t have to install jq because it’s already available in the ubuntu image,

1054.72 -> We don’t have to setup AWS CLI credentials either,

1058.16 -> Because it’s already been set up in the previous step of the workflow.

1062.48 -> So let’s commit this change.

1065.12 -> Push it to github.

1067.52 -> And open this URL in the browser to create a new pull request.

1071.68 -> This is a very simple change, so there’s not much to be reviewed.

1076.56 -> Let’s wait a bit for the unit tests workflow to finish.

1080.08 -> OK, now it’s completed.

1082.72 -> I’m gonna merge this PR,

1084.56 -> And delete the feature branch.

1086.56 -> Let’s open the master branch of this repo,

1089.28 -> The workflows are running.

1090.96 -> Let’s check the details of the deploy workflow to see how it goes

1094.72 -> OK, looks like the load secrets step was already successful.

1098.96 -> And the image is being built and pushed to ECR.

1103.36 -> Alright, everything finishes without any errors.

1107.68 -> Let’s open the AWS console ECR service,

1111.52 -> In the simple bank repository, we see a new image that has just been pushed.

1116.56 -> So it works!

1118 -> But I want to make sure that the image we built

1120.4 -> can actually talk to the production DB when we run it.

1124 -> So I will try to download this image to my local machine and run it.

1128.24 -> Let’s copy this image’s URL

1130.64 -> And run docker pull this URL in the terminal.

1134.16 -> We’ve got an error because docker cannot pull image from this private repository.

1138.96 -> We have to login to the AWS ECR registry first in order to pull or push image.

1144.72 -> To do that, let’s search for aws ecr get login password

1149.36 -> We’re using AWS CLI version 2, so let’s open this page

1154.08 -> This command will allow us to call the AWS API

1157.12 -> to retrieve an authentication token that docker can use to login to our private registry.

1162.64 -> So let’s copy this aws ecr get-login-password command

1166.96 -> And run it in the terminal.

1169.2 -> Voilà, the authentication token is successfully returned.

1173.12 -> Now we will pipe this token to the docker login command,

1176.96 -> Pass in the username AWS,

1179.6 -> and password from stdin argument.

1182.96 -> Finally the URL of our private docker registry.

1186.88 -> We must remove the name of the simple bank repository from the URL

1191.92 -> Alright, login succeeded.

1194.48 -> Now we can pull the production simple bank image to local.

1198.32 -> It’s done.

1199.44 -> Let’s check the images.

1201.36 -> Here it is!

1203.04 -> Now, I’m gonna run this image to see if it’s gonna work well or not.

1207.12 -> Oops, it failed at the run db migration step.

1210.48 -> Error: URL cannot be empty.

1213.36 -> If we look at the starts.sh file,

1215.76 -> We can see that in the db migration step,

1218.4 -> We’re using the DB_SOURCE environment variable.

1221.44 -> But this variable is only defined in the app.env file

1224.8 -> that our app will read when it starts,

1227.52 -> It was not set as the real environment variable of the container

1230.88 -> Before the db migration step is run.

1233.44 -> So that’s why migrate complains that the URL is empty.

1236.96 -> To fix this, we have to load the variables from the app.env file

1240.64 -> Into the container’s environment before running db migrate.

1245.2 -> If i echo DB_SOURCE now, it’s still empty

1248.64 -> To load the variables to the current shell’s environment,

1251.6 -> I will use the source command.

1253.92 -> So source app.env

1256.24 -> Now if I echo DB_SOURCE again, it’s not empty anymore.

1260.4 -> OK, so let’s copy this souce command,

1263.12 -> And paste it to the start.sh file,

1265.52 -> Right before the db migrate command.

1268.32 -> Note that inside the image, the app.env file is stored in the working directory /app

1275.12 -> So here, we have to change the path to: /app/app.env

1280.4 -> And that’s it!

1281.6 -> I think this will fix the issue.

1283.92 -> Let’s check out the master branch,

1286.48 -> Pull latest change from Github.

1288.72 -> And create a new feature branch for the fix.

1291.52 -> I’m gonna call it ft/load_env

1294.72 -> Add the start.sh file that we’ve just updated,

1298.8 -> Commit it with a message: load environment variable before running db migration.

1304.88 -> Then push it to Github.

1307.12 -> Open this link in the browser to create a new pull request.

1311.2 -> Wait a bit for the unit tests to complete.

1314.08 -> Then merge the pull request to master.

1316.4 -> And delete the feature branch.

1318.64 -> Now we have to wait for the deploy workflow to finish.

1321.76 -> While waiting, I’m gonna check out master branch on local,

1324.96 -> Pull the latest change that we’ve just merged from GitHub.

1328.32 -> OK, now let’s see if the workflow has completed or not.

1332.4 -> It’s still running.

1334.4 -> OK, it completed.

1336.96 -> In the AWS console, we can see the new image.

1340.4 -> I’m gonna copy its URL.

1342.56 -> Then run docker pull to pull this image to local.

1346.56 -> Alright, now let’s run it, with a port mapping 8080 to 8080,

1352.08 -> So that we can send an API request to make sure it’s working well.

1356.16 -> This time, the db migration was successful,

1359.12 -> And the server has started listening and serving request on port 8080.

1363.76 -> I’m gonna open Postman,

1365.6 -> And send the first API request to create a new user.

1368.96 -> It’s successful!

1370.16 -> The user has been created.

1372.32 -> Let’s connect to the production DB on AWS using Table Plus

1376.64 -> And check the data of the users table.

1379.28 -> Here we go, the user Alice has been inserted as the first record in this table.

1384.64 -> So now we can say that our production docker image is good and ready to be deployed.

1390.08 -> And with that, I conclude today’s lecture

1392.72 -> about storing and retrieving production configurations using AWS secret manager.

1398.08 -> I hope it was interesting and useful for you.

1401.12 -> In the next lecture, we will learn how to deploy the production image

1404.56 -> that we’ve prepared today to a kubernetes cluster.

1407.92 -> Until then, happy learning and I’ll catch you guys later!

Source: https://www.youtube.com/watch?v=3i1mQ_Ye8jE