AWS re:Invent 2022 - Take these open-source tools on your AWS adventure (BOA202)

Aug 16, 2023

AWS re:Invent 2022 - Take these open-source tools on your AWS adventure (BOA202)

You’ve set out on a grand adventure to learn, build and expand on AWS. Like any good adventure, it has its challenges. Time to gear up! Grab your best tools and gear to help you on your way. In this session, have a look at open-source tools that can help make your AWS adventure easier. See something for security and permissions, something for cost management, and a few more things for building in the cloud—tools like Infracost, IAMLive, and more.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

0.845 -> - [Darko] Welcome to this talk,

3.99 -> taking your open source tools

5.88 -> or taking these open source tools on your AWS adventure,

9.72 -> and thank you. Goodbye.

11.528 -> (chuckles)

12.361 -> That's all the slides we have,

13.68 -> just so you know.

14.951 -> - I worked on those.

16.061 -> (both chuckle)

17.484 -> - We will not bore you with slides.

19.021 -> First of all, this is a Friday crowd.

20.85 -> Thank you all for devoting your Fridays to us.

23.912 -> Woo, yeah Fridays! Yeah!

26.137 -> (audience clapping)

28.65 -> I think we're good now.

29.61 -> That's all I needed for today. Awesome.

32.162 -> So today is in all seriousness, a presentation.

35.52 -> I can go back to, there we go.

37.44 -> We're gonna be talking about some open source tools out

39.81 -> there that will help you on your cloud adventure.

43.02 -> So I'm gonna switch into demo mode, and please bear with me.

49.59 -> This is something not usually seen at re:Invent.

52.92 -> I'm overselling it.

54.96 -> So you've woken up, you've realized cloud

59.63 -> is the next next thing, right?

61.86 -> You've set up on your grand adventure to do cloudy things.

66.57 -> To build, to scale, to fail, to open S3 buckets,

70.65 -> to stumble with IAM,

73.8 -> stumble with costs,

75 -> and also realize that your laptop is not the cloud.

79.41 -> So you woke up, but it is dangerous to go alone.

83.37 -> What do you do? You need to exit your on-premises cave.

89.1 -> Let's go that way.

90.15 -> And you need to go out into the wide world of the cloud.

95.73 -> Hello!

96.78 -> - [Curtis] Hello.

98.224 -> - Hello.

99.057 -> (audience clapping)

102.3 -> - Hello.

103.133 -> - Who's that?

104.58 -> - My name is Curtis Evans.

106.53 -> I'm a principal developer advocate here at AWS.

109.71 -> And as Darko alluded to earlier,

111.12 -> I want to thank you guys so much for being here on a Friday.

113.43 -> I know you've been here and had a long adventure,

116.07 -> pracked, packed, somewhat hectic week,

118.249 -> re:Play last night I'm sure.

120.96 -> But again, thank you for being here today.

122.52 -> We really appreciate it.

123.9 -> - Thank you Curtis. And my name is Darko.

126.18 -> I am what they call a developer advocate,

128.28 -> but I used to be something we used to call

130.74 -> system administrator. Do y'all remember that?

134.82 -> I scream into the camera for a living, thanks to COVID,

138.15 -> but I'm here now to talk to you about

141.604 -> open source tools, command line,

143.4 -> and everything in between to help make your cloud adventure

146.49 -> better, safer, faster, and more fun.

150.45 -> So let's sit in our cloud thing here,

154.41 -> if I can hit this button, and let's take a ride.

158.34 -> We're taking a ride over to your first cloud steps, right?

162.42 -> You're, you've started doing things,

164.437 -> you started building code,

165.27 -> you started provisioning EC2 instances, VPC's, users,

167.356 -> policies, cloud formation, cloud front, API gateways,

171.06 -> Lambda functions...lot of moving things.

176.13 -> What happens when you start creating a lot of things with

179.16 -> a lot of important data?

181.95 -> Those things can be problematic if

183.96 -> somebody else wants to access them.

186.36 -> So you need to secure them.

188.82 -> But security is a problem. You need your shield.

193.47 -> You need to protect your resources.

195.21 -> Everything needs to be locked down.

198.72 -> How many of you, raise hands, when you do some testing when

203.73 -> you start building, how many of you

204.96 -> just use the completely open policy of whatever?

209.64 -> Don't lie.

210.93 -> - [Curtis] Oooh. Lot of hands.

213.282 -> - Chmod 777.

214.71 -> We also used to do that, right?

217.11 -> You just open everything to the world because

219.15 -> security is hard. It is.

220.8 -> You have to define exact things you need to do.

223.26 -> You need to define, oh, my API call needs to do this.

226.47 -> What is the IAM policy for that.

229.17 -> Or I get a user yell at me,

231.007 -> "Darko, I can't do what I need to do."

235.2 -> So how do you figure that out?

236.91 -> How do you get to shield up and ready with relative ease?

241.2 -> So we're gonna talk to you about two open source tools to

244.05 -> help you with that.

245.88 -> And the first one is very special because there's a

249.69 -> gentleman in the audience here, Ian McKay,

253.02 -> who built this tool and he came here to heckle me.

256.5 -> So let me give you a scenario.

260.67 -> You want to build a thing, right?

263.769 -> You have your open policy, list bucket, launch instances,

266.16 -> create users, request route 53 domains,

270.66 -> and your account admin works fine.

272.91 -> But how can you actually understand or define what your IAM

279.24 -> policy needs to look like for

281.19 -> only a user to do only those things?

285 -> There's always a documentation, but you know, you,

289.546 -> there's a lot of documentation out there and sometimes

292.38 -> trying things out can be problematic.

296.19 -> So IAMLIVE, am I pronouncing that right, Ian?

299.76 -> Is it Okay?

301.222 -> IAMLIVE is a tool that will basically intercept API calls

305.88 -> made from your laptop to the cloud and will generate

309.711 -> an IAM policy for you.

313.14 -> Let me show you.

315.393 -> IAMLIVE. Let me actually do this, IAMLIVE.

318.87 -> There's the set any, boom, this is now running.

325.2 -> If I open up another thing here,

330.27 -> I'm opening up another pane here and I do AWS, S3, LS.

338.28 -> That won't work. AWS, S3, profile default.

345.734 -> Default.

349.8 -> We depend on so much (indistinct) here.

353.88 -> Can I show config? I can show config. That's fine.

356.31 -> Vin.

359.73 -> So AWS config.

363.78 -> What did I miss?

364.613 -> What's the problem here?

365.64 -> - [Curtis] Move your profile before the AWS profile

369.81 -> and then the rest of your command.

372.365 -> - What did I do?

373.249 -> - [Curtis] Try that.

374.407 -> - Oh, the profiles look good, right?

376.11 -> - [Curtis] I think it might be the location of where it is.

378.175 -> (indistinct murmuring from the audience)

387.51 -> - Profile, profile name?

391.29 -> - [Curtis] Yeah. Make profile the second parameter

394.41 -> on the command line.

396.038 -> - Ah okay.

397.623 -> Okay. Like this?

400.573 -> - - Profile.

402.456 -> - [Curtis] Yes.

404.28 -> AWS BOA202 admin.

408 -> Unable to parse config file.

410.64 -> Unable to par config file. Why won't they do it?

414.206 -> (indistinct murmuring from the audience)

417.285 -> - It needs to be prefix to profile?

419.51 -> - [Audience Member] You have a duplicate line.

421.902 -> Line two and three.

424.619 -> - Line two and three. Ah, there we go, okay.

427.14 -> So let's try it again.

430.65 -> Ha works beautiful.

434.23 -> - [Curtis] You guys passed the test. Congratulations.

435.063 -> - There we go.

436.663 -> (audience clapping)

439.14 -> That's what, that was intentional,

442.823 -> but I'm trying to see why this doesn't work now.

445.72 -> So AWS profile does this, LS.

448.47 -> Ian, am I doing something wrong?

450.816 -> - [Ian] Probably.

451.649 -> - Probably?

452.598 -> (Darko and Curtis laugh)

453.741 -> So what, what's what's supposed to happen here,

456.083 -> if I do IAMLIVE like this, and I run a command like this,

460.41 -> the thing upstairs should generate a policy, but it doesn't.

466.23 -> Well that's a, that's a bummer.

468.782 -> So this tool would intercept any API call made and just

472.74 -> spit out a policy to do that for you.

475.11 -> But for some reason it doesn't work.

477.87 -> I promise this worked on my laptop,

479.82 -> but it doesn't work anymore. So let me try something.

482.97 -> IAMLIVE, help, set any, this would be,

487.98 -> oh set any would update the thing.

489.96 -> But there is an option that needs to do it.

492.42 -> Like it should be like just that.

495.364 -> - [Ian] You need to do the same profile maybe.

498.158 -> - Maybe same profile. Okay, so if I do this,

500.329 -> so if, let's do this. IAMLIVE.

501.85 -> Ah, I know why. I know why. Thank you.

504.319 -> IAMLIVE like this,

505.17 -> then go back and then just without the profile name,

507.51 -> because by default set any did it on the default profile.

512.46 -> I do this now it doesn't work.

517.083 -> - [Audience Member] Chmod 777.

518.717 -> - Chdmod 777.

519.55 -> (Darko laughs)

522.401 -> - [Curtis] Good suggestion.

524.301 -> - [Audience Member] (indistinct)

526.2 -> - I have to set, I still have to set it again.

527.91 -> So here's the, here's the thing,

528.81 -> troubleshooting it real time.

531.18 -> Boom, boom, I'm live, set any.

533.82 -> If I do set any, what it'll do,

535.81 -> it'll add the thing to my config file, hopefully.

540.24 -> - [Audience Member] Oh I think it's a single dash,

543.295 -> not double dash.

544.193 -> - Let me just see it is CSM enabled true now.

547.35 -> So that's what it done. If I do AWS, S3, LS...

551.01 -> Boom. Ha! there we go. You're welcome.

553.858 -> (audience clapping)

556.5 -> So that thing up there is a policy that gives you permission

560.64 -> to list the bucket. So if I do something else,

563.555 -> AWS S, let's see, EC2, describe instances.

569.22 -> You get another thing, right?

571.77 -> What's another API call you wanna run?

574.027 -> - [Audience Member] (indistinct)

576.368 -> - What?

577.223 -> - [Audience Member] KMS.

578.409 -> - KMS. What KMS?

579.268 -> Let's do this. Let's do this.

581.61 -> Check this out, check this out.

585.13 -> Check this out.

587.911 -> AWS. KMS.

593.49 -> Can we describe something? Describe, describe key?

599.19 -> Key ID.

601.41 -> Is there a key ID? Let's do this. Boom, there we go.

604.474 -> It has added KMS list keys and describe keys.

607.945 -> It has added those things in that policy.

610.2 -> Now, if I want to add a policy, create a policy,

612.48 -> I can just copy paste that and do it for my user.

615.57 -> Now, most likely your users just don't

618.48 -> do use the CLI all the time,

620.4 -> but this should intercept basically

622.77 -> any call you try to make.

624 -> So think about all of the things you would do,

627.54 -> things your your users would need to do.

629.43 -> And this, it's relatively easy to

631.41 -> create this single user policy to help you do that.

636.348 -> IAMLIVE is just a blessing for these things because

638.25 -> sometimes some calls require multiple permissions,

642.54 -> which then require a lot of troubleshooting.

646.23 -> So this is a great proactive way to create good, decent,

651.39 -> IAM policies for your users.

654.87 -> But what about when you already have a policy in place?

660.81 -> And your user complains that they cannot do a thing?

665.25 -> I will let my good buddy Curtis talk to you

667.11 -> about the next security tool.

669.3 -> We still at security, right? Run draft.

672.571 -> Next security tool to help do something else.

674.58 -> So why, what is this tool and what, why, why do we use it?

677.79 -> - [Curtis] Thanks Darko.

679.98 -> Thank you for being here as well, Ian.

681.42 -> Is Nolan Gohan in a house by the way?

684.18 -> I'm about to talk about his tool. Not in.

687.27 -> - He's not in, he's not in.

688.2 -> - Cool. No worries.

689.55 -> So I'm gonna talk to you guys about Access Undenied.

692.91 -> To the point Darko just made, we've already

695.13 -> got an open source tool that's gonna give us

697.17 -> the ability to understand what permissions that users,

700.647 -> entities, what have you, are gonna need

702.93 -> in order to do things.

704.743 -> Access denied is one of those tools that's gonna

707.88 -> give you an opportunity to discover just that.

711.48 -> If there was an operation by way of an API

713.49 -> that you wanna run and the user is not allowed to

716.43 -> do so by way of deny,

718.17 -> this is an opportunity to parse that information,

720.72 -> do some quick analysis, and be able to fix it on the fly.

724.89 -> With the help of my trusted CIS admin here.

727.791 -> Darko, let's create a situation where, say I'm a user,

731.52 -> I've just been given access to an AWS account and I wanna do

734.85 -> something simple like just go out and list buckets

737.34 -> with an S3 API call.

743.46 -> - [Darko] You know what.

744.475 -> - What do we have there?

745.308 -> - [Darko] We can do it now.

746.401 -> - Sorry?

747.234 -> - [Darko] We can't do it. We list the bucket.

748.966 -> - Remove the profile and let's try.

750.847 -> (Darko chuckles)

751.68 -> - [Darko] We tested this out before.

752.513 -> - We're gonna go back and do a...

753.655 -> - [Darko] We forgot to remove the policy.

754.488 -> (Darko laughs)

755.321 -> - No worries.

756.154 -> - So let's do it again.

758.43 -> S IAM profile.

763.11 -> - [Curtis] Yeah, so what we're gonna do

763.943 -> is basically reset the privileges for

765.93 -> the BOA202 user so that they have no ability to go out

770.76 -> and do a list of buckets.

773.183 -> - Can I delete,

774.807 -> can I delete the policy version from the command line?

777.48 -> - [Curtis] Sure. Just, just as easy.

780.334 -> - Just S3 list bucket?

784.879 -> - [Curtis] List policy.

785.712 -> - List policy.

791.57 -> No, no, no.

794.25 -> So smoke and mirrors.

795.39 -> Imagine this user cannot access this bucket.

798.75 -> There's an access denied 404,

801.231 -> 403, doesn't work for them.

802.56 -> - [Curtis] Yep.

803.845 -> - Right? And the user calls Curtis up says,

805.207 -> "Curtis, my thing doesn't work. Figure it out."

810.72 -> How do we figure it out?

811.83 -> - So we're gonna send that user to the administrator.

815.43 -> We're gonna have the administrator be aware of

818.1 -> what the problem is.

819.9 -> As you know, anytime you have an

821.67 -> access denied issue that's gonna be

824.221 -> written to a CloudTrail trail event.

826.8 -> With Access Undenied the administrator's

829.11 -> gonna have the ability to do an analysis

831.21 -> or run an analyzed command by way of extracting that JSON

835.89 -> data on that particular event and get some additional

839.37 -> information about what the problem is and how they can go

842.4 -> about resolving it.

844.397 -> So with that Darko, why don't we go ahead and we,

846.48 -> we've already taken the JSON out of the CloudTrail event

849.51 -> and now we're gonna use the Access Undenied utility

852.45 -> to go and analyze it.

853.86 -> - So we have a file here called "file.JSON".

857.953 -> This file is just a trace from CloudTrail, right?

862.5 -> So if we do Access Undenied and pass it on an events file,

868.17 -> file.JSON.

869.52 -> If we do this...

876.03 -> we could not find the reason for access denied.

878.483 -> (Darko laughs)

880.329 -> - [Curtis] Are you running the profile

881.31 -> that we needed for that one?

883.694 -> - This, oh yes. Profile, right?

885.3 -> It is a profile. It is, we're using the correct profile.

888.3 -> Are we,

889.741 -> - [Audience Member] (indistinct)

895.28 -> I, I think actually the,

896.85 -> this doesn't matter here because we are analyzing this

899.31 -> locally, so it doesn't,

900.48 -> - [Curtis] Well you're gonna have to be the admin user

903.796 -> in order to be able to parse the uh...

906.289 -> - So this is, this is demo number two.

909.478 -> We have two more to go.

911.738 -> (Darko laughs)

913.804 -> I think I said people, this is easy tools, right?

917.469 -> - [Audience Member] (indistinct)

919.44 -> - Yeah, it most likely looks at the live profile.

922.096 -> So what, what, what, what's happening right now...

925.2 -> But it should analyze the file, no?

926.202 -> Events file.

927.93 -> File.

930.749 -> - [Audience Member] (indistinct)

932.07 -> - Yeah, it opens the file.

933.84 -> Right? There we go. So that all works.

936.96 -> File.JSON. Are we in the correct directory? We are.

940.8 -> So there are no reasons for access denied.

945.922 -> - [Audience Member] Do you think it's already allowed?

947.771 -> - It's all, well the thing is, it's allowed.

949.786 -> So that's why we, we need to figure out

951.214 -> how to remove the policy.

952.282 -> So are you gonna figure out how to remove the policy for a

954.66 -> user with us together?

955.68 -> Yes. Okay, let's do this.

960.57 -> Profile. BOA Admin. IAM.

964.35 -> How do we do remove? Remove roll from instance profile.

967.86 -> Remove, delete user permissions battery.

973.26 -> Can we do, I, I don't know, detach roll policy?

978.42 -> - [Curtis] Detach. Yeah

979.253 -> - We are actually adding put,

981.34 -> we're adding, put user policy.

983.64 -> We're adding, literally adding a policy to a user, right?

988.257 -> So we are,

990.39 -> you are doing this and then username is BOA something,

994.2 -> - [Curtis] 202-user.

996.049 -> - 202-user.

996.93 -> And then policy name is S3 list book, the list policy.

1001.967 -> And then we need to do a policy file, policy document file,

1007.04 -> da da da and then policy JSON.

1009.98 -> Now this will add this thing to that document, but I,

1012.26 -> I don't know how to remove this thing. That's the problem.

1016.428 -> If I...I'll delete the file.

1018.86 -> Yeah, I'm just gonna be,

1020.27 -> oh actually you know this policy .JSON...

1025.16 -> Deny.

1026.66 -> So when the user complains why they can't do it,

1030.05 -> it's because of you.

1032.42 -> So I'll do this.

1034.674 -> - [Audience Member] It is delete your policy.

1037.094 -> - It's delete your policy?

1038.15 -> - [Curtis] Yeah I think it's (indistinct)

1038.983 -> - Okay, let's try that.

1042.53 -> Can we do profile

1046.58 -> BOA202, IAM,

1050.58 -> delete user policy,

1052.82 -> user name,

1057.38 -> name, what is the user, BOA202 user,

1061.28 -> and then policy name S3.

1068.835 -> - [Curtis] This policy.

1069.873 -> - This policy?

1073.16 -> Okay, let's try it again.

1075.236 -> (Darko chuckles)

1077.721 -> Let's do this from the, from the top.

1079.163 -> BOA user still works.

1081.98 -> We didn't delete the policy, but I do, like wait...

1087.23 -> profile, admin, IAM,

1092.737 -> delete user policy...

1098.09 -> User name, BOA202.

1104.382 -> User policy.

1107.931 -> - [Audience Member] (indistinct)

1110.258 -> - Yeah. Yeah.

1111.181 -> So I think that would be great, but it should actually,

1113.012 -> this tool should show us this.

1114.225 -> Do this, BOA202 admin,

1118.814 -> (audience murmuring)

1123.259 -> I'll use the dash name. Okay, that's consistency.

1128.07 -> 202, user.

1131.063 -> - [Curtis] Dash user, yeah.

1132.53 -> - Policy name. And you can see that it has a few policy.

1136.497 -> S3,

1138.56 -> list policy.

1141.17 -> Does this work?

1143.417 -> - [Audience Member] Yeah.

1144.641 -> - No such user policy. There we go.

1145.722 -> So this user actually technically should be able to do this,

1147.68 -> shouldn't be able to do this. Okay, beautiful.

1150.02 -> It cannot do it.

1151.52 -> Winner, winner chicken dinner.

1154.67 -> Okay, so let's try this.

1155.78 -> Access denied, once again.

1160.22 -> Boom. See.

1161.15 -> - [Curtis] There we go.

1162.536 -> (audience clapping)

1163.37 -> - Don't clap.

1165.29 -> You are, you are endorsing our bad preparation.

1168.68 -> Do not clap.

1170.752 -> - [Curtis] Gotta love the ability to respond on the fly.

1173.754 -> Well done, Darko.

1175.34 -> - What is this now Curtis?

1177.512 -> - [Curtis] All right, so we have now

1179.501 -> identified that there is a problem related to

1181.37 -> this user unable to go out and do something

1185.09 -> as simple as a list bucket.

1187.52 -> The administrator has taken the JSON entry from the cloud

1190.61 -> trail event, use Access Undenied to analyze it,

1193.79 -> and now has the ability by way of this output from the

1197.66 -> analysis to say simply go and do a list all my buckets,

1202.432 -> permissions policy for this user.

1204.74 -> - [Darko] Yeah, exactly.

1206.512 -> - With that we can go and,

1208.751 -> and now grant that user this access.

1209.84 -> Notice also too that,

1211.82 -> and Darko alluded at the very beginning, you know,

1214.07 -> hopefully those hands that went up earlier about being S3

1217.67 -> star users aren't, you know,

1219.38 -> really into that, because you wanna be able to always grant

1222.438 -> just principle of lease privilege,

1224.51 -> which is what we're gonna do here by way of a put policy for

1226.886 -> that user granting those permissions.

1232.02 -> - [Darko] Boa202, user. Okay.

1235.22 -> - User with a U not what a Y.

1237.35 -> - [Darko] Policy name. S3 list policy

1240.68 -> and then policy document,

1241.91 -> it's gonna be file policy JSON.

1245.9 -> - [Audience Member] You added that (indistinct)

1248.42 -> - Yeah.

1249.44 -> - [Darko] I just change it right now as he was talking,

1251.12 -> I sneakily changed it to allow. Okay, so if we,

1254.45 -> if we look at the policy now it is allow,

1256.94 -> and do you think it will work right now?

1259.403 -> - Do a list bucket.

1261.2 -> - [Darko] Do you think it'll work?

1265.288 -> - My man.

1268.91 -> - [Darko] There we go.

1270.115 -> (audience claps)

1271.248 -> - So that's Access Undenied in a nutshell.

1274.22 -> Know that you can also use this for organizations accounts

1279.268 -> where there may be issues with service control policies that

1281.15 -> are also creating issues.

1282.26 -> So take a look at Access Undenied.

1284.93 -> - [Darko] Yeah. It's available in GitHub.

1286.845 -> It was made by somebody called Ermetic or,

1288.26 -> but you know, the actual name of the person?

1289.67 -> - Yeah, actually we're proud to call him a

1291.62 -> member of our AWS community builder.

1293.24 -> So thank, thank you, wherever you are.

1295.833 -> - So I mean, how would you do this without this tool?

1298.79 -> - Oh boy.

1299.81 -> - Yeah.

1301.312 -> - Management console.

1302.145 -> - Yeah, management.

1302.978 -> You like, did you ever have a user come to you and says,

1305.217 -> "Hey, my thing doesn't work."

1308.06 -> Right? And you know, that's a problem.

1311.12 -> So you can go through the management console,

1312.83 -> go to CloudTrail and just scroll

1314.24 -> and just search "deny",

1315.92 -> okay, there's a deny here and you,

1317.96 -> you look for it and you add a policy and then you find

1322.25 -> another deny, but is that the old deny or the new deny?

1324.86 -> You don't know.

1325.693 -> So Access Undenied definitely helps streamline this

1327.677 -> so you don't have to worry about that as much.

1332.388 -> Okay, time to move on.

1336.05 -> We have finally our shield.

1340.37 -> We have figured out security,

1341.99 -> haven't we, in the last 20 minutes.

1345.17 -> Time to move on to the next part of your cloud adventure.

1348.47 -> And once you figure out security,

1350.45 -> you start building things in the cloud, right?

1352.4 -> You, you go and Lambda functions, S3 buckets, permissions,

1356.93 -> and you give people correct permissions and they start

1359 -> building, cloud formation.

1361.214 -> I don't know. And, and the things start getting dark.

1365.93 -> They're dark because you start to

1368.154 -> lose visibility on, you know,

1369.62 -> you can build fast in the cloud,

1372.47 -> but it can also be quite expensive in the cloud, right?

1375.5 -> So you need to be able to manage your cost.

1377.54 -> Let's say, do people use infrastructures code?

1381.11 -> Terraform, cloud formation, CDK, Pulumi, CFEngine,

1384.53 -> backs scripts, right?

1386.27 -> So infrastructures code helps you deploy things very,

1391.49 -> very fast, but they also help you

1394.82 -> make things expensive very, very fast.

1399.05 -> Have you ever asked yourself, gee,

1401.72 -> I wonder this Terraform template I've written and I spent

1406.725 -> days writing it, how much will that cost?

1408.29 -> Right? You can go to the a AWS pricing page,

1411.32 -> look over the pricing page and look, yeah, you know,

1413.69 -> this is the cost of essence, EC2 instance.

1417.02 -> This is a cost of a file transfer.

1418.97 -> This is a cost of this NAT gateway,

1423.249 -> but it takes a lot of time.

1425.81 -> So I'm gonna show you a way you can navigate yourself around

1429.92 -> costs for your infrastructures code templates,

1432.68 -> in this case Terraform, to figure out with some magic,

1437.87 -> how can you, how much this will cost you, right?

1440.99 -> So let's go and talk about a tool called Infracost.

1446.51 -> Infracost is an open source-ish tool.

1450.95 -> So it is a tool that is open source,

1452.9 -> but there's some paid elements to it as well.

1455.36 -> I'm gonna be talking solely about the open source part here.

1457.88 -> It is a tool that helps you take a cloud formation template,

1462.23 -> not, sorry, I'm sorry, cloud formation Terraform template

1465.65 -> and, and analyze it for costs.

1468.98 -> What it does, it takes all the elements

1471.47 -> from a Terraform template and just

1474.146 -> bounces them off the pricing API.

1477.74 -> So if I show you something here, there's a main.tf file.

1482.21 -> Very simple.

1483.47 -> There's a single EC2 instance with a 100 gig volume size,

1488.87 -> just a root volume size.

1489.89 -> There's a, a one terabyte secondary volume,

1493.85 -> there's an S3 bucket, and there's a just a,

1496.73 -> a simple access control list for that bucket.

1502.31 -> How much does this cost?

1505.04 -> I do a lot of live streaming. And when, whenever I show

1508.116 -> something on the, on the cloud,

1509.06 -> there's usually one person in the comments is like,

1511.589 -> "How much does this cost?"

1513.357 -> "What if I host a website on AWS,

1514.85 -> how much is that per month?"

1516.59 -> And that is a super valid reason to, well, super val,

1520.34 -> valid question, right?

1522.98 -> So what I can do here is do Infracost,

1529.28 -> and just basically tell it, listen, check all of the files,

1534.2 -> all of the Terraform files I have here and analyze them and

1538.34 -> tell me how much will this cost me per month...if it works.

1545.45 -> It did.

1548.237 -> EC2 instance, 900 bucks a month,

1551.24 -> block storage, 10 bucks a month,

1552.92 -> bigger storage, a hundred and a lot money a month.

1556.91 -> There's an S3 bucket, but it gives me,

1559.79 -> this is how much per gigabyte.

1562.4 -> So I get a total of $1,000 a month for this

1567.8 -> oversized web server, right? In a single S3 bucket.

1572.96 -> This is amazing. It gives,

1574.88 -> like if I had a hundred EC2 instances, gateways, KMS keys,

1578.72 -> all the different elements of AWS,

1582.53 -> you would get all of that here.

1585.5 -> But wait, there's more.

1587.57 -> That S3 bucket, I don't like that.

1589.76 -> That is very vague, right?

1591.98 -> So what does this cost, is the bucket free?

1594.95 -> Is it's, how much is,

1596.9 -> how much is, how much is this thing?

1599.21 -> What you can actually do with Infracost is you can actually

1602.33 -> create something called a usage file.

1606.05 -> The usage file can individually attach to individual

1611.12 -> resources within your T file,

1613.1 -> and you can define usage patterns.

1617.516 -> So if I say my bucket will have that many gigabytes,

1620.51 -> and this is how much monthly tier one requests

1624.89 -> and tier two requests.

1627.316 -> So this is some of the things,

1628.838 -> there's plenty more I can define here

1630.14 -> and plenty different types of usage calculations

1632.54 -> you can have for different database resources.

1635.75 -> Creating this usage file helps me actually do one thing.

1640.7 -> So if I do Infracost and define a usage file,

1645.38 -> so it'll take my Terraform template,

1648.44 -> slap on a usage file on it,

1650.93 -> and hopefully it will give me the actual usage of it.

1656.15 -> Now you see how much will my bucket cost me

1659.362 -> with 10,000 gigabytes?

1660.195 -> How much will the transfer cost me?

1662.298 -> It's very much nicely broken down,

1665.21 -> so you can kind of estimate that it should not

1667.7 -> be too expensive, it should be relatively affordable or not.

1672.62 -> This is, when I first saw this,

1675.71 -> a lot of my fellow architects just (gasp) "thank you".

1680.21 -> because it helps you understand how much this costs.

1684.141 -> Because one of the biggest problems,

1685.78 -> "problems", in the cloud is like, okay, so how much does,

1687.887 -> how much does, how much will this cost me?

1689.39 -> In the old days, I don't wanna say good old days,

1691.58 -> but in the old days you would buy a server, that's $100,000.

1695.75 -> That's how much costs you.

1697.07 -> Add some electricity on it, and that's it.

1699.53 -> This helps you determine these things much, much better.

1702.65 -> But let's play around.

1704.812 -> So let's modify something,

1706.34 -> then let's do change Infracost.

1708.74 -> Let's, let's make some changes to it, right?

1711.259 -> Let's modify the instance type to, let's, let's,

1714.29 -> let's do intel instead of,

1715.46 -> instead of graviton and then change the volume type

1717.71 -> from one terabyte to eight terabytes.

1723.29 -> And there is a way I need to figure out what my command for

1725.87 -> this thing is because I don't, I don't memorize commands.

1730.34 -> You can actually output the, the, the,

1737.87 -> the run of this file of Infracost, right?

1743.24 -> So instead of putting it out on the screen, it,

1745.743 -> I will output it to a json file. If I call it J, not jsauce.

1749.088 -> - [Curtis] With an N.

1750.686 -> (Darko laughing)

1751.806 -> Jsauce, is that, that's my new favorite file format, jsauce.

1757.26 -> Okay, so I'm outputting this thing right now.

1760.76 -> So if I go back and change my main.tf and go back to a

1765.32 -> Graviton instance, M six G, and change this back to one,

1772.669 -> one terabyte, save, and then run a diff.

1779.127 -> So there's a diff in Infracost, this should be death.

1785.57 -> Beautiful.

1787.304 -> So it will actually change, tell me how much

1789.852 -> cheaper would be me changing from eight terabytes

1792.95 -> to one terabyte and how much

1794.27 -> cheaper it is changing from Intel to Graviton, right?

1797.9 -> So you can do these things, and let's be honest,

1801.8 -> you're probably not gonna do this on your laptop.

1804.2 -> You may when you're building, when you're testing stuff out.

1806.42 -> But once you advance into provisioning infrastructure

1809.72 -> through CICD, that's a pull request, right?

1814.25 -> On your GitHub actions on your code pipelines,

1817.16 -> one of the actions before actually deploying a system would

1820.07 -> be to check how much does it cost.

1822.11 -> If somebody accidentally put a 16 x large

1826.238 -> instead of a T2 micro,

1828.475 -> you wanna see that on a pull request.

1831.088 -> That's how it works. It's a wonderful tool.

1832.37 -> You can play with it today. It, it has a free, free option.

1837.62 -> You can just grab a,

1840.59 -> a free API key for it.

1842.708 -> The thing that's, it needs that, because it's actually

1843.83 -> querying the AWS pricing API. So, and that's it.

1847.61 -> It's, it's basically a brew install

1849.68 -> and you also have to get the API key

1851.42 -> and log in with that API key.

1853.16 -> That's all. Wonderful tool helps you navigate

1857.6 -> those pesky costs, especially at the beginning.

1861.71 -> So let me grab my compass for navigation.

1865.88 -> Alright, now get back into the cart, Darko.

1869.45 -> Now let's go somewhere else.

1871.67 -> You figured out security, you figured out cost.

1875.15 -> Now it's time to build. It's time to test things out.

1878.63 -> It's time to go and build your Terraform files.

1882.14 -> It's time to do CLI, unsuccessfully

1885.53 -> like your presenter here,

1887.93 -> and you don't want to potentially do that on accounts.

1892.1 -> So let's talk about playing, building,

1895.16 -> trying things out locally on your wonderful comfortable

1899.3 -> laptop here with a tool called...

1903.11 -> - [Curtis] LocalStack.

1904.006 -> - LocalStack. Yes, LocalStack.

1906.243 -> LocalStack is also an open source-ish tool.

1908.57 -> So Curtis, tell us what LocalStack is

1910.522 -> and why should we care about it?

1911.355 -> - Thanks Darko.

1912.46 -> Well, first off I can say that LocalStack

1915.538 -> with regards to what we just did with our

1917.138 -> Infracost analyzer is gonna,

1918.83 -> I don't wanna say throw it out of the window,

1920.51 -> but what LocalStack is gonna do is by way of running in a

1924.38 -> local docker container, you will give new builders,

1928.22 -> even experienced builders for that matter,

1929.84 -> an opportunity to go out by way of the LocalStack emulator

1933.8 -> and test what the creation of various resources

1937.416 -> are gonna look like.

1939.71 -> I wanna just pause so everybody can hear that.

1941.87 -> You're basically simulating how to build

1944.788 -> AWS resources in a local environment without

1948.08 -> actually running them within the account and incurring cost.

1952.43 -> I'm sure we've all run into that situation where, you know,

1955.52 -> just outta curiosity, maybe we wanted to run that,

1957.98 -> I don't know, X1 instance, you know,

1961.01 -> you generate the command line for it, it works great,

1964.67 -> but then someone's gonna pay you a visit and ask you about

1968.536 -> this $100, $200 or $300 bill

1970.852 -> as a result of not turning it off.

1973.295 -> So wanna just quickly show you what LocalStack looks like

1976.01 -> in terms of its ability to go out and again, just do that.

1979.58 -> Use an emulator and simulate the

1981.2 -> running of AWS resources without actually incurring costs.

1985.46 -> So what Darko's gonna do here is gonna ensure that the

1987.98 -> LocalStack docker is running.

1991.28 -> - So this thing's running,

1993.34 -> running local host, that's running on my laptop.

1995.225 -> It's basically a container that will accept

1997.67 -> AWS API calls.

2000.1 -> - [Curtis] Perfect.

2000.933 -> So it looks like we're up and running.

2001.766 -> If we go to a command line using the endpoint URL parameter

2006.4 -> and pointing to the LocalStack local host,

2008.71 -> why don't we just go out there and do

2010.33 -> a EC2 describe instance.

2012.82 -> - What you see here is there's an endpoint URL, that's a

2015.64 -> parameter of the database CLI, that you can actually,

2018.25 -> instead of it going default to the endpoint of the actual

2021.19 -> AWS CLI, it goes to a local endpoint.

2025.27 -> - And part of the beauty of that, by the way,

2027.56 -> is if you're looking to just kind of

2029.588 -> migrate between the two environments,

2031.46 -> simply remove that parameter and everything that you're

2033.956 -> learning and developing by way of LocalStack,

2036.192 -> you can immediately turn around and run it

2037.832 -> into the environment itself.

2039.496 -> As we see here, we have absolutely no EC2 instances

2042.25 -> running in this emulation of what's an AWS account.

2046.81 -> By way of another Terraform stack,

2048.7 -> now let's go out and see what it would look like if we were

2051.37 -> to simulate the launching of a number of EC2 instances.

2056.02 -> By the way, we're also gonna be doing this, we,

2057.77 -> we'll give you a quick look at what

2060.164 -> the main tf file looks like here.

2062.414 -> I also want to indicate that we're using

2064.366 -> what's known as a tf local tool,

2065.5 -> which is basically gonna allow a Terraform like utility to

2069.79 -> interact with the, with the LocalStack Docker image as well.

2073.99 -> Here we see in the defined Terraform,

2077.38 -> we want to go out and create 10 EC2 instances

2081.79 -> of that omni type.

2084.268 -> So using the TF local tool,

2086.68 -> we'll go out and do an init.

2092.266 -> - [Darko] Is this big enough? Should we make it bigger?

2093.46 -> - Yeah. Sorry.

2094.995 -> - [Darko] Okay.

2095.916 -> - That's good.

2096.749 -> Okay, now let's run a, an apply.

2102.04 -> - [Darko] We need to validate first. No.

2104.77 -> Never do TF Terraform apply without doing anything else.

2111.189 -> - Boom. And just like that,

2113.048 -> we now see that the tf local utility is now communicating

2116.17 -> with the LocalStack and saying, okay,

2119.2 -> let's go out and now create ourselves a number of instances.

2122.634 -> And then if we go back and run that describe instance again,

2127.63 -> we could have shortened it for you

2129.13 -> and maybe an output format.

2131.952 -> But now we've got a number of EC2 instances that are

2134.708 -> running in the emulation environment.

2136.58 -> Again, really cool way for, you know, those new builders,

2138.55 -> those experienced builders that just wanna make sure that

2141 -> they are, you know, have their Terraform built correctly.

2146.71 -> And then to the point I made a second ago,

2148.63 -> if they're ready to now deploy that into the actual account,

2151.27 -> they can just simply remove that endpoint URL parameter and

2154.36 -> be able to go and build very similar in their environment.

2157.72 -> The cool thing about LocalStack is it supports a variety of

2161.47 -> different AWS services.

2163 -> We're talking S3, we're talking EC2, RDS, DynamoDB.

2167.77 -> I could, could keep going forever, but just know,

2170.952 -> again, it's a, it's a great way for you to go

2174.952 -> out there and test the build of your environments

2176.035 -> without actually having to incur cost.

2177.67 -> Notice, also, that LocalStack has over

2180.64 -> a hundred million docker pulls, I think to, to this date,

2184.81 -> which is a testament of just how much the community has

2187.54 -> learned to accept and, and, and embrace this tool.

2191.14 -> - A big note here, we didn't launch anything, right?

2194.17 -> There's no EC2 instances is running,

2196.45 -> it is emulating that it's running.

2199.327 -> So you can test your Terraform templates

2201.4 -> that it'll actually do something.

2202.96 -> You can test your CLI commands that it actually works,

2205.66 -> instead of attempting it on real stuff.

2208.558 -> So it is not actually launching anything.

2210.67 -> You can build them in those instances.

2213.005 -> But the management process,

2213.838 -> the operations of launching instances, creating tables,

2217.87 -> S3 buckets, whatnot.

2218.92 -> You can actually test them locally in their laptop,

2220.87 -> so they will work and you just remove

2224.2 -> the endpoint URL and you run that

2226.63 -> on an actual account.

2229 -> Way faster to test things out,

2230.56 -> especially when it comes to launching a

2232 -> lot of resources with Terraform.

2234.43 -> So I think this for testing,

2236.95 -> for playing around, will also help you with the cost thing,

2239.77 -> but it'll help you do things way faster.

2242.84 -> So if you've not checked out LocalStack, definitely do,

2244.81 -> I mean it's a, it's a little tool that can, again, it's,

2247.63 -> it's free. There are some paid, paid features of it,

2249.73 -> but for the most of us, free is enough.

2253.33 -> - [Curtis] Yeah, the enterprise versions will start

2256.88 -> to allow you to integrate more of your

2258.43 -> CICD processes and methods in it for, you know,

2262.195 -> just kind of, kind of full stack testing there.

2264.28 -> - Awesome. Now let's go back to your pilot adventure.

2268.03 -> Now we have come here and we finally have a book because we

2271 -> learn things and we have a sword because we use the

2273.34 -> knowledge to go forth, right?

2275.98 -> We are ready to get off the train of, of,

2279.49 -> of open source tools and explore further, right?

2283.42 -> Now you have handled costs, security,

2288.715 -> local development, but that's not all right?

2292.63 -> As you go across the, the river of despair

2296.86 -> into your further cloud adventures,

2298.96 -> into trying new things out, new services,

2301.45 -> all the re:Invent stuff,

2303.13 -> you will actually encounter plenty of more open source tools

2308.02 -> in the forest of open source.

2310.645 -> These are not the only open source tools for yours

2312.28 -> AWS adventure. There are plenty more out there.

2315.13 -> And I'm gonna just mention a few of them.

2317.65 -> The first and foremost, Former2.

2320.62 -> Made by the same guy who made IAMLIVE.

2324.477 -> Ian did this too.

2326.467 -> If you have trouble building infrastructure as code,

2330.19 -> you build everything through click ops, which is fair.

2333.46 -> I do click ops.

2335.68 -> What if there was a way you could export your

2338.701 -> bits of your architecture into a cloud formation file?

2343.03 -> Into a Terraform file?

2345.34 -> Former2 does that.

2346.18 -> It will basically read what's running within your database

2349.75 -> account and create infrastructure code templates.

2353.86 -> Wonderful, wonderful things. I'm not gonna show it,

2356.347 -> I'm just gonna talk about it.

2359.62 -> Coming up.

2361.03 -> AWSLS. AWSLS is literally what it says on the thing.

2365.11 -> You list resources within AWS. So if I go back to here,

2370.57 -> I do believe I have it kind of ready.

2373.382 -> If you do, let me do this, AWSLS,

2377.74 -> you can actually list all of the resources.

2382.24 -> For example, AWS VPC's, in your default region,

2386.59 -> in your default account.

2389.11 -> You can go even more AWSLS, you can do

2393.1 -> multiple accounts, multiple regions.

2395.17 -> If I just add a comma here, it'll cover multiple regions.

2400.51 -> Have you ever lost a thing at AWS?

2402.726 -> Like, oh, I, I seem to have an EC2 instance rank somewhere,

2405.4 -> but Amazon has like bajillion regions now.

2408.55 -> How can I find it?

2410.05 -> You can do this.

2412.492 -> Or how many of you have more than one account?

2415.48 -> Exactly right. So this also works for multiple accounts.

2419.59 -> So if you have like multiple profiles,

2422.92 -> I only have a single one, maybe I can do default like that.

2427.99 -> It'll go through multiple, multiple CLI profiles,

2432.37 -> multiple accounts basically, and it'll run the same thing.

2435.4 -> It'll throw those things out there.

2437.14 -> You will see the default profile and my BOA202 admin and

2440.77 -> the VPCs in the US East and US West one regions.

2444.55 -> Super simple, but very useful

2447.01 -> when you're trying to find that thing

2448.99 -> that's costing you money.

2451.984 -> All right, back to the forest of open source.

2453.52 -> What else do we have?

2455.312 -> CloudMan, or cloud manager.

2457.15 -> It is a simple tool for all of the EC2 nerds out there.

2460 -> Who loves EC2?

2462.214 -> - [Curtis] There we go.

2463.786 -> - There's dozens of us.

2465.64 -> You can use CloudMan to basically manage EC2 instances.

2470.65 -> It's a command line tool, because we all have command line,

2473.65 -> but you can do a couple of things here like you can,

2476.874 -> you can start, stop, terminate,

2477.707 -> and you can also connect to an instance, right?

2479.11 -> If you have like SSM enabled.

2481.03 -> I'm not sure if I have SSM enabled on the instance,

2484.049 -> so we'll see. I don't.

2485.878 -> But if you have SSM enabled on that instance,

2487.492 -> you can actually connect it through systems manager.

2488.41 -> If you have serial connection enabled,

2490.535 -> you can connect through serial connection.

2492.473 -> So let me actually see, does that work?

2493.87 -> Does serial work on this one? I, I don't know.

2496.12 -> It works beautiful. Here we go.

2498.702 -> So, no it doesn't. Nevermind.

2499.63 -> But if, if you have those things,

2500.77 -> you can do all of those things.

2502.641 -> Connect to an instance, you can filter them,

2504.343 -> you can do a bunch of stuff with EC2.

2506.733 -> Wonderful little simple tool, but it helps.

2511.485 -> Now what else?

2513.37 -> Where am I? I am lost in the forest of open source.

2516.91 -> Alright. Oh yes, yes.

2519.201 -> My favorite tool, AWS-Nuke.

2522.55 -> Wonderful tool.

2524.327 -> If you ever build things without using

2525.4 -> infrastructure as code

2526.63 -> and you need to clean up after yourself.

2529.6 -> Yes, this thing will nuke your account,

2532.33 -> delete everything within an account.

2534.31 -> It is wonderful because it helps you not incur cost.

2538.39 -> Clean up after yourself.

2539.62 -> And we all had a long day of building testing things and

2542.89 -> just leave your laptop and and go home and you come back,

2546.4 -> you're like, oh, okay, what did I do yesterday?

2549.58 -> So AWS-Nuke helps you clean on that.

2551.5 -> Start with a fresh slate. Wonderful.

2555.76 -> And finally, AWS Lambda Powertools.

2559.24 -> If you build Lambda functions,

2561.46 -> if you use Lambda for Python and type script,

2565.06 -> Powertools are just an open source tool, utility,

2568.609 -> you can basically plug into your Lambda functions, that will

2570.79 -> help you do things that are usually complicated,

2573.123 -> rather easy.

2574.69 -> So tracing, metrics, logging,

2577.39 -> all of those things can literally be just

2578.427 -> plugged in with Lambda Power tools.

2582.04 -> If you are not using Lambda Power tools and you are

2584.063 -> a Lambda user, check them out.

2586.27 -> Trust me, you'll thank me in the morning.

2589.9 -> And finally, let's go here and let me ask you,

2596.966 -> what is your favorite open source tool?

2600.88 -> Who wants to say something? Don't be shy.

2602.23 -> Come on, scream it.

2604.69 -> Vin.

2607.421 -> - [Audience Member] Leap.

2608.383 -> - Leap. What is leap?

2610.309 -> - [Audience Member] (indistinct)

2614.98 -> - Okay.

2616.417 -> - [Audience Member] (indistinct)

2621.94 -> - Yeah. Ah, okay.

2624.961 -> - [Audience Member] (indistinct)

2631.133 -> - Amazing. Amazing.

2631.966 -> So there's a lot of open source tools out there that can

2635.23 -> help you do what you do, right?

2638.17 -> These are just a tiny little sliver made by the amazing

2641.2 -> folks out there who do this.

2643.33 -> I don't wanna say on their spare time,

2644.74 -> but some of them do it on the spare time.

2646.69 -> So make sure to support those people.

2649.21 -> Make sure to check them out. If there's a pull request,

2652 -> make it better, improve it, add your own things,

2654.64 -> and also build your own open source tools.

2657.88 -> Before I wrap up, I wanna ask,

2660.4 -> are there any questions we can answer

2662.23 -> within the next five minutes?

2664.024 -> Go ahead.

Source: https://www.youtube.com/watch?v=bEg-mIFZEmc