AWS re:Invent 2022 - Terraform providers using AWS CloudFormation custom resources (BOA316)

Aug 16, 2023

AWS re:Invent 2022 - Terraform providers using AWS CloudFormation custom resources (BOA316)

Terraform, AWS CloudFormation, and AWS CDK support a broad set of AWS services. Often, users want to automate more and even create resources that are not supported out of the box. In this session, explore how to build CloudFormation custom resources using your chosen programming language or Docker container. This method has been successfully used with CloudFormation and CDK, and it also works well with Terraform. Using Terraform modules, you only have to add a few lines of code to your projects to use a custom provider. Explore how this solution is designed to be secure, simple, future-proof, and reliable.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

0.18 -> - [Martijn] All right.

1.29 -> Welcome everyone to Terraform providers using

3.717 -> AWS CloudFormation custom resources.

7.59 -> Before I start, quick show of hands,

9.66 -> who has experience with CloudFormation?

13.6 -> Okay, almost everyone. And with Terraform?

17.562 -> Okay.

18.507 -> And who likes the one above the other?

21.9 -> Doesn't matter which one,

22.83 -> but do you have a preferation on one of those?

25.08 -> - [Audience Member] It depends.

26.493 -> - Okay, it depends. Correct answer. Yeah.

28.65 -> Hopefully after this talk you will love both

30.978 -> of the applications because we are using them quite a lot.

36.312 -> Before we start, quick introduction.

38.43 -> My name is Martijn van Dongen.

40.11 -> I come from the Netherlands, Amsterdam.

43.68 -> I'm a Cloud Evangelist working for Schuberg Philis.

47.04 -> Probably you never heard of our company.

48.93 -> We're just with 400 people in the Netherlands

51.45 -> with basically local customers,

54.12 -> but the customers we also have

55.89 -> are becoming more global customers.

58.26 -> So one of them will be tomorrow in the keynote.

60.84 -> So I encourage you to visit the keynote.

65.76 -> We are basically a mission critical company,

69.75 -> so we aim for 100% with everything.

74.52 -> So first place, 100% customer satisfaction,

78.84 -> but also 100% automation, etc.

81.48 -> So that sets the stage for this talk, as well.

86.04 -> Next to my day-to-day job at Schuberg Philis,

88.02 -> I'm also a Cloud Evangelist, sorry, a community hero,

91.77 -> and that basically because I organize a lot of meetups

95.67 -> in the Netherlands,

96.93 -> and so they are little bit smaller than this room,

101.16 -> about 80 people,

102.27 -> so I'm kind of a little bit nervous right now.

104.85 -> Although we also did a big event a couple of weeks back,

108.45 -> a community day with 500 people,

110.46 -> and Werner Vogels as the keynote speaker,

112.62 -> so we're really proud, well, that's a big event.

117.09 -> Next to me is Sohan. Can you introduce yourself?

119.37 -> - Yeah, of course.

120.203 -> Hi, my name is Sohan Maheshwar.

121.68 -> I'm a Senior Dev Advocate with AWS based in the Netherlands,

125.13 -> where I work with Martijn on community events.

128.28 -> And yeah, this is my fifth re:Invent,

130.14 -> so I'm super excited to be here.

132.72 -> Martijn, why don't you take it away?

134.16 -> - Cool.

136.41 -> So our talk is based on a true story

139.29 -> and actually based on two true stories.

142.86 -> The first one is one of our customers,

145.95 -> it's a big lease car company service,

150.03 -> and they have over a thousand engineers

152.85 -> that, of course, have to automate everything.

155.97 -> And when they were automating everything they found out,

158.07 -> they discovered,

158.903 -> that there not all the applications

160.53 -> were able to be automated.

162.48 -> So we had to use some kind of technology to also automate

166.86 -> the parts that are not native in AWS, for example.

170.64 -> And if we let them go, so just develop it on their own,

174.6 -> you get all variations of implementations,

177.78 -> and that's what we would like to avoid.

182.16 -> That's what you can see here.

183.54 -> So one team is using CloudFormation.

185.58 -> So they've built their own custom resource provider,

188.01 -> or their own tooling to provision those custom resources.

193.383 -> The Terraform implementation is also different.

195.78 -> They use different tools, different coding languages,

199.17 -> but also different implementation.

201.39 -> And for CDK and for all the other infrastructures,

203.76 -> code frameworks, same thing happens.

206.61 -> So what we were looking for was actually this,

209.1 -> where we can have a common implementation

211.41 -> that all these popular applications, cloud infrastructure,

216.041 -> configuration languages, can work with.

219.15 -> And we found it using CloudFormation,

221.481 -> that I will show you in this presentation.

225.18 -> The other example came in a little bit later.

228.24 -> So just a few weeks ago.

230.746 -> I found out that in one of our customer teams,

233.91 -> they were working with a lot of breweries around the globe

237.66 -> and they use IoT, Greengrass, the green icon.

241.89 -> And to provision your IoT Greengrass,

245.25 -> you can use Terraform, you can use CloudFormation.

250.358 -> And with Terraform, if you provision something to AWS,

253.853 -> like EC2, it works great.

256.32 -> If you try to do it with Terraform to Greengrass,

259.32 -> it's not that optimal.

262.05 -> When you use CloudFormation, it's close to optimal,

264.6 -> but there were still some things that we would like to do

267.63 -> that was not natively supported in CloudFormation.

270.57 -> So we resolved the specific use cases using an AWS Lambda.

278.404 -> - [Sohan] Right.

279.57 -> So essentially Martijn spoke about two customers.

282.42 -> Now we can't show the demo of those products, of course,

285.168 -> so we want you to imagine like a legacy application.

288.21 -> I'm a kid of the nineties,

289.649 -> so this is how most websites looked back then.

292.65 -> So just think of a legacy application.

295.8 -> Maybe it needs a little more comic sense, I dunno,

298.001 -> but, you know, so this is just a visual aid, alright?

301.59 -> And we are gonna talk to you about

303.54 -> how you can use CloudFormation,

305.34 -> specifically custom resources in CloudFormation.

309.96 -> Now, I know everyone here knows what CloudFormation is,

312.84 -> but this talk is also being recorded,

314.85 -> so there may be people who might not know.

316.53 -> So just a very, very quick refresher on CloudFormation.

320.31 -> It lets you model, provision, and manage,

322.65 -> AWS and third-party resources using infrastructure as code.

327.18 -> It's the OG infrastructures code service,

330.39 -> as I like to call it.

331.5 -> Don't quote me on that outside of this room, I guess.

334.17 -> But essentially you code in YAML or JSON directly,

338.01 -> or you can use sample templates.

340.02 -> You can upload these templates to an S3 bucket,

342.63 -> or to the CloudFormation resource itself.

346.32 -> When you deploy these templates, it creates a stack, right?

349.77 -> It creates a stack of the resources that you have deployed

352.86 -> and those stacks and resources are provisioned

355.35 -> as your running environment.

357.39 -> This way you have represented

358.8 -> your entire infrastructure as code.

360.8 -> So this is repeatable, it's completely automated,

363.162 -> you can roll back, yada, yada, yada.

366.45 -> What we want to really focus on is this one feature

370.29 -> of CloudFormation, called Custom Resources.

373.65 -> We think that this is a pretty powerful feature

376.41 -> of CloudFormation.

378.09 -> And essentially it allows you to write

380.25 -> custom provisioning logic within a CloudFormation template.

384.96 -> Now, there are two things you can specifically do.

387.12 -> One is, there are AWS services

390.03 -> that aren't supported in CloudFormation,

392.73 -> which you could reference using a custom resource,

395.779 -> or you could also reference a third-party resource,

399.21 -> using custom resources.

402.15 -> When you're writing CloudFormation code,

404.01 -> this is typically what it looks like, you know?

406.74 -> Simplest example, you have a type, which is an EC2 instance,

410.13 -> so there is a particular syntax

411.93 -> that you have to follow, right?

413.46 -> So you say type is equal to AWS EC2 instance,

417.06 -> and then the properties below,

418.53 -> sort of, describe that EC2 instance.

421.47 -> And that's how when you deploy this CloudFormation template,

424.35 -> this EC2 instance is up and running in the cloud.

428.91 -> For custom resource, the syntax is quite similar.

432.505 -> The difference is, the type is custom,

435.27 -> followed by a name that you can give,

436.98 -> and we are going to call it a unicorn tracker.

439.41 -> It's a simple application that, you know,

441.3 -> just allows you to add and delete unicorns,

443.73 -> which you will see very soon.

445.68 -> And essentially in the properties you have to mention

448.47 -> something called a service token, right?

451.2 -> So how custom resources works, is first there is something

455.1 -> called a custom resource provider, right?

457.92 -> So someone who's written a template

459.42 -> that provides for the custom resource.

462.06 -> Then, of course, there is the custom resource itself,

465.21 -> which is what we'll be doing.

466.95 -> And third, CloudFormation actually runs the two.

470.49 -> Now this provider and the custom resource, sort of,

473.25 -> talk to each other using this thing called a service token.

477.6 -> The service token could either be a lambda function

480.6 -> or it could be an SNS topic.

483.12 -> So essentially when that custom resource, that template,

485.85 -> starts this lambda function, or the SNS topic,

488.97 -> is alerted and things happen behind the scenes,

492.21 -> which I will describe to you in the next slide.

496.11 -> This is our implementation for custom resources.

499.56 -> So essentially you have an author

501.27 -> who's written a custom resource provider

504.48 -> in a CloudFormation template.

506.52 -> Again, there is a slight difference in the syntax,

509.19 -> there is a custom resource provider

512.13 -> and a custom resource itself,

514.5 -> both of them are written as CloudFormation templates.

518.73 -> The logic of this custom resource

520.92 -> is authored in that lambda function.

523.47 -> So when you deploy the CloudFormation stack,

525.96 -> it speaks to the lambda function,

527.66 -> which gets the provider and the custom resources.

531.42 -> What happens behind the scenes is that this lambda

534.24 -> actually writes to a pre-signed S3 bucket, right?

537.66 -> So there's a pre-signed URL with an S3 bucket,

540.39 -> and the results of the deployment

541.86 -> are sent to that S3 bucket.

544.012 -> CloudFormation verifies looking at that S3 bucket

547.68 -> and then goes ahead and creates all of your resources.

551.13 -> So all of that is happening behind the scenes.

553.132 -> We as builders or you as builders,

555.6 -> we don't have to worry about any of this, right?

558.06 -> You have to focus only on your custom resources

560.812 -> and the custom resource provider.

564.75 -> So what sort of code do you actually write

567.72 -> in a lambda function for a custom resource?

570.42 -> Well, it is typical CRUD operations, right?

572.94 -> Create, Read, Update, and Delete.

575.55 -> So this is sample code

577.41 -> of how your lambda function would look like.

580.32 -> Again, goes without saying, but you could write this

582.36 -> in any language supported by lambda.

584.726 -> If you want, you could also do that in a Docker container.

588.31 -> For this demo we have written it in Python.

591.21 -> So as you can see, that is an API,

593.25 -> which is the API our ugly legacy application uses

596.91 -> behind the scenes.

598.59 -> And you can see we've written a create function,

601.05 -> which just creates a unicorn,

602.79 -> and we've written a delete function,

604.2 -> which just deletes a unicorn, right?

606.03 -> Super simple.

608.94 -> A CloudFormation code looks a little something like this.

612.66 -> You can see a lambda layer there.

614.52 -> That layer, if you're not familiar with lambda layers,

617.64 -> it's something that you can use

619.17 -> within your lambda functions,

620.82 -> if your lambda function has things like dependencies.

623.7 -> So for instance, if you have some Python libraries

626.4 -> that you want to have as part of your lambda function,

629.01 -> you can put that in a lambda layer, which can be shared.

631.978 -> Then of course we describe the lambda function itself

635.307 -> and you know, you see all of that,

637.29 -> and right at the bottom is the endpoint,

639.75 -> which again I said is the API endpoint

641.85 -> for our legacy application.

644.7 -> So let's actually take a look at how all of this end-to-end

647.94 -> will look in a demo.

649.41 -> We have, of course, prerecorded it

651.12 -> because waiting for CloudFormation to, you know, think,

654.54 -> isn't the most fun demo.

655.83 -> So we'll go through it rather quickly.

657.78 -> All of this code is available on GitHub

659.277 -> and we will be sharing the link at the end, right?

661.5 -> So all of it's there, so don't worry about that.

665.622 -> Right, so essentially we haven't spoken yet about Terraform,

670.74 -> but the room seems very familiar.

672.81 -> We're gonna use just two Terraform commands

674.91 -> for this part of the demo.

676.278 -> Terraform init, initializes Terraform.

678.252 -> Terraform apply, goes ahead and creates cloud resources.

682.74 -> Right? That's pretty much it.

685.14 -> So hopefully you can see the screen at the back,

688.71 -> but we are essentially initializing Terraform,

691.05 -> and we are going to initialize our legacy application.

694.56 -> Shout out to Dennis,

695.46 -> and a couple of folks here on the front row,

696.96 -> who helped us write that application.

699.42 -> And this is going ahead and creating...

704.25 -> Yeah, I'm just gonna pause that there.

706.53 -> Yeah, this has created resources in the cloud

709.29 -> and it's created an endpoint,

711.18 -> which is the API gateway URL, right?

713.79 -> This is the endpoint for our legacy app.

717 -> We will be using this in the rest of our demo.

720.45 -> Again, this API is an open API as you will see right now,

724.5 -> but if you want to add authorization or anything to it,

727.2 -> you can do that via open API gateway,

730.56 -> for the purposes of the demo,

731.79 -> this is right now an open API, right?

734.13 -> As you can see, there's nothing in the API,

736.56 -> in the backend right now, no items.

738.18 -> So we will be adding it using our lambda function very soon.

745.17 -> Okay. So yeah.

749.25 -> So now we're gonna create the custom provider itself

752.64 -> using this particular CloudFormation template, right?

756.48 -> And as you can see in this template,

758.46 -> this is the code that I showed you earlier,

759.99 -> there is a lambda layer

761.49 -> and there is an endpoint right at the bottom.

764.04 -> So we are going to deploy

765.63 -> this CloudFormation template very quickly.

768.84 -> You can see the resources that are mentioned in this

772.05 -> particular CloudFormation template.

775.74 -> And this is written in Python 3.9, but like I said,

778.53 -> you can write it in any language that lambda supports.

786.12 -> Right?

787.2 -> So what we are going to do now

788.4 -> is also very quickly look at

790.29 -> the lambda function itself, right?

793.26 -> The one thing I want you to look at

795.51 -> is line number three, right?

797.07 -> That's the hidden hero of this entire demo,

799.26 -> which we didn't speak about in the slides,

801.51 -> but it says from CR Helper, right?

803.64 -> So that is short for Custom Resource Helper,

807.12 -> it's an open source module developed by AWS itself.

811.29 -> It's very, very helpful

812.61 -> when you want to reroute requests from custom resources

815.97 -> because it takes care of the mapping.

817.83 -> So when CloudFormation

818.82 -> sends an event to your lambda function,

821.1 -> this takes care of the mapping automatically, right?

823.56 -> Otherwise you have to do all of that manually.

825.81 -> So as you can see in the code itself,

827.64 -> you see @helper.create, @helper.read, @helper.delete,

832.62 -> this is CR Helper doing all of that work for you.

835.65 -> So if you're using custom resources,

838.41 -> I highly recommend you check out this particular module.

842.94 -> The code in itself is trivial,

845.22 -> all it does is add and delete unicorns,

848.55 -> so don't worry too much about it.

850.2 -> You can see a backend API mentioned there

853.02 -> and the rest is typical CRUD code.

855.24 -> We haven't written the update function.

857.46 -> That's an exercise for the audience

859.17 -> or we just ran out time, one of the two.

860.895 -> Yeah, you understand.

862.176 -> Yeah, and as you can see, you can delete unicorn as well.

869.73 -> So what we are going to do now is

871.26 -> we have the cloud formation template,

874.091 -> we have the lambda code.

876.6 -> So yeah, let's just deploy it.

879.15 -> So to deploy it first we need an S3 bucket

882.21 -> and I'm gonna create an S3 bucket,

884.04 -> which is imaginatively named some-random-bucket-1234.

888.06 -> Honestly, it was a miracle that this worked, right?

890.04 -> I was convinced that someone's taken that name before,

892.68 -> apparently not.

894.39 -> So yeah, this is the end point we got earlier

897.75 -> from our API gateway, right?

900.18 -> This is gonna be the API for our legacy app.

904.02 -> I created this bucket from which

906.63 -> our CloudFormation template will be read.

910.71 -> That bucket's created and now I'm just going to

913.29 -> package our CloudFormation template, right?

916.26 -> So if you do AWS CloudFormation package,

920.13 -> you specify a template file,

922.14 -> which in this case is template.yml

925.47 -> and then you specify the S3 bucket

927.96 -> to put the build.yml file into.

933 -> So I specify the S3 bucket

934.41 -> and I forget to actually put the yml file,

937.73 -> so I'm just gonna do that again right now.

940.11 -> These things happen, yeah.

943.614 -> I'll just push the build.yml file there and that's it.

947.07 -> So right now it's in the S3 bucket.

948.75 -> I can actually deploy my CloudFormation template

951.39 -> from this location, right?

953.31 -> And you can actually see that if I just pause there,

956.01 -> the name of the S3 bucket that we just created,

959.07 -> some-random-bucket,

960 -> has been replaced in that build.yml file that you see there.

966.72 -> So now that we have the build.yml file,

968.64 -> we can just go ahead and, you know,

971.58 -> deploy this and see if it works.

974.25 -> So when we do CloudFormation deploy,

976.14 -> we specify a stack name,

977.46 -> in this case unicorn tracker provider,

979.8 -> 'cause it's a custom resource provider.

982.08 -> We specify the template file and we do two things.

985.29 -> We do a parameter override, right?

988.516 -> You can see that here.

992.13 -> Yeah, I'll just pause it.

993.48 -> You see we've done a parameter override

995.19 -> so that we can override a variable in the template

998.94 -> with the variable of the API that we mentioned before.

1002.39 -> And after this

1003.35 -> we are going to be adding the IAM capability, right?

1007.07 -> So that it has access.

1008.54 -> If you don't do this,

1009.373 -> you will see the dreaded permissions error.

1011.81 -> So we have to mention that in the CloudFormation deployment,

1014.84 -> as well.

1016.61 -> Once you do that,

1017.54 -> the CloudFormation stack is ready to be deployed.

1021.71 -> If you're the type

1022.543 -> that enjoys seeing CloudFormation stacks being deployed,

1025.07 -> like I am, you can open up the browser

1027.71 -> and see all the events that are happening, of course,

1030.44 -> and you will see that the lambda function has been created.

1034.31 -> There you go.

1036.56 -> Yeah, so our custom unicorn tracker has been created

1039.89 -> all using the custom provider.

1043.7 -> So the provider has been created.

1045.83 -> So now that we have done that,

1047.09 -> we have a custom resource provider,

1049.34 -> we can create a custom resource, again using CloudFormation.

1053.69 -> So I just enter into the directory.

1058.91 -> And in this directory, this is my very simple template.yml

1063.5 -> that uses a custom resource.

1065.6 -> As you can see in line number three,

1067.55 -> the type is custom resource.

1070.1 -> I mentioned earlier that you have to have a service token

1072.65 -> that's either a lambda function or an SNS topic.

1075.89 -> We are doing lambda,

1076.723 -> so you have a lambda function mentioned there,

1078.86 -> and we are gonna create a unicorn.

1080.18 -> We've chosen a completely random name of Werner,

1082.781 -> complete coincidence, don't know where that came from.

1086.12 -> And yeah, let's just go ahead and create a custom resource

1089.48 -> using CloudFormation.

1092.893 -> Yeah, that's lambda function and that's our unicorn one.

1098.57 -> Again, deploying this CloudFormation

1101.51 -> is very straightforward.

1103.43 -> All we have to do is just say CloudFormation, deploy.

1107.45 -> We give the stack a name,

1109.55 -> and this time we can call it

1110.84 -> unicorn tracker custom resource.

1114.29 -> We point to the template file and that's it.

1120.92 -> So now the CloudFormation stack is going to deploy

1124.07 -> and, again, you can view that in the console itself.

1127.25 -> You will see that the lambda function's been created

1129.77 -> and all of the other resources as well.

1132.2 -> And now we can actually test out the API that's running.

1136.97 -> And as you can see there,

1138.08 -> you can see this unicorn named Werner has been added, right?

1142.46 -> So just as a quick recap,

1144.8 -> we basically first created a custom resource provider,

1147.92 -> using CloudFormation,

1149.21 -> and then we created a custom resource,

1151.34 -> using CloudFormation, to add a unicorn to our legacy app.

1156.83 -> So we've done that using CloudFormation.

1158.87 -> Now let's look at how we could potentially do that

1161.21 -> using Terraform.

1163.307 -> And for that I hand the mic over to Martijn.

1171.68 -> - First, a quick refresher on Terraform

1174.29 -> because it works a little bit differently

1175.76 -> than CloudFormation.

1177.189 -> CloudFormation is, of course, a service in AWS

1179.141 -> where you send your templates to

1180.637 -> and it will manage your stacks.

1183.53 -> Where in Terraform, you also write those scripts,

1187.25 -> in a little bit different language,

1189.23 -> but you use a CLI tool to deploy the resources

1192.71 -> and the CLI tool will actually talk to the APIs of AWS.

1197.21 -> The state and the stack will remain local,

1200.12 -> if you store the state file locally.

1202.28 -> You can also share it centrally

1203.69 -> with all the tools that are provided.

1205.49 -> In this case it doesn't really matter,

1207.56 -> you just save the the file locally.

1211.52 -> One of the other great things of Terraform

1213.379 -> is that you can also use it for different cloud platforms

1215.45 -> for technologies other than AWS.

1219.02 -> And, of course, also AWS.

1223.704 -> Terraform works with providers.

1226.52 -> So if you just download Terraform,

1228.26 -> it can't deploy immediately to AWS.

1231.29 -> You first need to specify

1233.6 -> which providers you would like to use.

1235.94 -> And if you select AWS,

1237.2 -> it will download additional libraries,

1238.567 -> put it in some hidden directory,

1240.92 -> and from that moment on you can define the resources of AWS.

1247.7 -> For example,

1248.533 -> this example where we want to deploy an AWS instance,

1253.22 -> so that's the type, AWS_instance,

1256.4 -> almost similar to how we define it

1258.2 -> in CloudFormation with the colons.

1260.33 -> And then the next one is web,

1262.25 -> which is a logical name for this resource.

1266.029 -> So we can reference that resource

1267.68 -> in other parts of our template or our project.

1271.76 -> I also have a list of properties

1273.77 -> or Terraform calls them, "resource attributes".

1276.62 -> This is basically the configuration of that item.

1279.62 -> So this EC2 instance

1281.24 -> will use a specific AMI instance type, etc.

1292.37 -> And it will, of course, use the AWS provider.

1297.74 -> Now we can also embed that CloudFormation template

1300.65 -> we used before within our Terraform template.

1304.67 -> So in this example we translated the YAML example

1309.74 -> towards JSON.

1311.69 -> We prefer Jason in this, this example.

1314.93 -> And you see a couple of components on the top.

1317.113 -> You see the provider AWS, where we also specify the region.

1320.51 -> We have one variable, that's the name,

1322.79 -> the name of the unicorn we are going to create.

1325.49 -> And there are also two specific resources,

1328.1 -> which is called data.

1329.87 -> This is where you can retrieve data from your AWS account.

1333.627 -> So these are in CloudFormation,

1335.93 -> you will use functions in Terraform,

1338.736 -> you will use the data options.

1342.83 -> And then the resource is a CloudFormation stack

1346.04 -> with the name unicorn with some parameters

1349.19 -> with the template in there.

1354.56 -> We also have a thing called Terraform modules.

1357.35 -> So what you, for example, have in CloudFormation

1360.05 -> is if you create your template,

1361.88 -> it suddenly grows towards 1,000 or 2,000 lines of code.

1366.62 -> Same thing can happen in Terraform,

1369.41 -> but the great thing of Terraform

1370.477 -> is you can put it in separate files,

1372.47 -> but you can also create something called a module

1374.99 -> to isolate, for example, all your servers

1378.08 -> in a particular module,

1379.82 -> or a database in a particular module,

1382.88 -> so that you can easily reuse that

1384.89 -> across multiple teams.

1387.47 -> And that's also something we want to do

1388.97 -> for our unicorn tracker, right?

1390.47 -> It's easier to share a module with our colleagues

1394.16 -> instead of a template that they have to update

1396.8 -> in their their Terraform projects.

1401.09 -> So this is how it looks when we create this module.

1405.05 -> We can share it on a lot of locations.

1408.2 -> We chose for the S3, an HTTPS endpoint.

1411.98 -> So that were basically a public endpoint

1413.57 -> where we can retrieve that module.

1416.33 -> So let's say we deployed that module to S3, to I3S,

1421.55 -> in our new Terraform project,

1423.2 -> where we want to leverage the unicorn tracker.

1426.44 -> That can be basically any application.

1429.89 -> There we define all our resources

1431.81 -> but also invoke the module and configure the module.

1436.22 -> Plan and apply, when you apply this template,

1441.29 -> it will use CloudFormation to create that custom resource

1445.13 -> together with some other resources

1447.11 -> that were defined in our Terraform project.

1451.7 -> When we want to use a module, it looks like this.

1455.48 -> So this is a complete, can be a complete project.

1458.36 -> We have again the providers, AWS,

1460.34 -> we have two data objects to retrieve some information

1464.48 -> and then the module, where we have a source.

1467.083 -> Based on the path of the source,

1469.04 -> it will either download it from the internet,

1471.44 -> or go through your data repository,

1473.48 -> or can basically be any location.

1477.26 -> And then we define the name, again completely random, Andy.

1485.12 -> The module, this is exactly the same file as we used before.

1491.63 -> It's basically because every project in Terraform

1494.51 -> is automatically a module

1496.76 -> if you put it in a separate directory.

1499.28 -> So what we will do in the demo is put a file in a directory,

1503.66 -> zip it,

1504.493 -> and upload it to S3 so that we can use that module.

1509.66 -> And if we want to automate everything,

1511.43 -> we also want to automate the creation of

1513.74 -> and sharing that module.

1514.97 -> So in this template we will archive the files.

1520.91 -> So we'll create a zip file based on the source.

1522.79 -> So the module will be zipped,

1524.72 -> create an S3 bucket,

1526.43 -> and upload of the zip we created to S3.

1530.87 -> And here in the top point you'll also see

1532.49 -> that together with AWS we also have a different provider,

1535.01 -> which is the archive provider.

1539.51 -> And the outputs at the bottom of this example

1541.88 -> is simply the URL where the module can be found.

1545.33 -> So that's something we will copy

1547.006 -> and use when we use the module.

1562.67 -> So this is where we ended the previous demo, right?

1565.37 -> We created Werner and now we want to create...

1570.44 -> We want to try out first

1572.24 -> using Terraform and CloudFormation together.

1575.09 -> The first example in the slides

1577.13 -> where we just embed CloudFormation in our template.

1588.755 -> There it is.

1589.588 -> This is the template again

1591.86 -> with the variable, the data pro objects, and the resource,

1596.66 -> which is the hard coded CloudFormation, in the template.

1601.04 -> Give the name unicorn.

1604.46 -> You have to pass the parameters and, of course,

1607.28 -> specify the template body.

1613.85 -> Now we'll type a Terraform init,

1616.37 -> and if you type Terraform init,

1618.184 -> it will download all the modules and also all the providers.

1625.7 -> Then we hit a Terraform apply,

1627.95 -> it will ask for the variable, the name of the unicorn,

1633.14 -> confirm that it's okay to deploy this.

1637.01 -> And after a couple of seconds,

1642.92 -> it's now deploying the stack in CloudFormation,

1646.707 -> CloudFormation is triggering the lambda function

1649.19 -> and it will deploy our additional unicorn.

1652.67 -> So now if you open the API,

1654.89 -> we see two unicorns and the Android.

1658.67 -> Like I said, we don't want to share a CloudFormation file

1662.3 -> that they have to embed in their Terraform project.

1666.68 -> We would like to share a module.

1668.63 -> So I copy this template and let's say

1670.567 -> I'm gonna maintain this for all my colleagues.

1674.06 -> So I will create a module for it.

1677.54 -> Here you see the module source,

1679.28 -> and that's just a copy of the file we used

1682.34 -> in the earlier demo.

1685.88 -> Same file. Same things in there.

1693.65 -> The only difference,

1694.61 -> this is what they also showed already,

1697.52 -> creating a zip file,

1703.7 -> creating an S3 bucket,

1708.65 -> and then deploy the file, make it public.

1713.106 -> So now you can use it everywhere.

1714.92 -> There are no secrets in there.

1716 -> Of course, if you use secrets

1717.317 -> and a secret API then you should not put it public,

1720.264 -> but put it in your own repository.

1724.13 -> And as the output, the URL of the module.

1733.46 -> To show you that the file is public,

1737.09 -> I will just open a browser.

1742.49 -> I'm a little bit faster than the demo.

1746.99 -> Or copy the URL to the file.

1768.23 -> There it is.

1770.947 -> And it will provide you download.

1772.49 -> So I can also download it just to show you

1775.28 -> that it's just a file on the internet right now.

1780.98 -> And then we want to use it in our project.

1783.05 -> So this is the whole project we now have.

1785.93 -> It's just again, provider, some data, and the module,

1789.8 -> where we have to specify that URL to our file.

1798.02 -> And the name, Andy.

1821.57 -> It's now downloading the module.

1824.12 -> So together with all the providers,

1827.57 -> also the module downloaded, it's stored in the project.

1831.5 -> And if we in the future want to update it,

1833.96 -> we just have to update the lambda function,

1837.14 -> we can keep the template the same.

1839.69 -> But if we want to make a change in the template,

1841.76 -> we make a new template, upload the new module,

1845.6 -> ask our developers,

1846.447 -> "Hey, you might want to use our new module."

1849.65 -> Just change the version type of the module,

1852.68 -> maybe add some additional parameters, some properties,

1855.8 -> or attributes, and then it's updated.

1864.56 -> Just so that we actually have a third unicorn now

1868.97 -> from our project.

1871.494 -> And there it is.

1881.158 -> And we can, of course, also destroy the template.

1884.33 -> Having implemented also the delete in the CRUD application.

1889.713 -> So if we destroy the project,

1893.33 -> it will also delete the stack,

1894.92 -> and also delete all resources related to that stack.

1906.65 -> Now it's gone.

1916.55 -> So the final implementation look like this.

1919.28 -> So we have a lot of teams working

1921.23 -> in their preferred way of working.

1923.09 -> That can be either CloudFormation, or Terraform, or CDK,

1926.69 -> or maybe even another framework.

1930.29 -> As long as it is capable

1931.61 -> of deploying CloudFormation stacks.

1936.86 -> Then CloudFormation will create a stack

1938.45 -> using the custom provider,

1940.04 -> and if you need other resources.

1946.28 -> So without the solution,

1948.23 -> and let's say we chose a different path,

1950.45 -> we probably have a thousand engineers all thinking,

1953.907 -> "Hey, how should I automate my deployment?

1957.17 -> Oh, I don't know any programming languages,

1960.38 -> so maybe I should do something in Bash."

1962.63 -> Somebody else says,

1963.463 -> "Okay, I'm gonna learn Go,

1964.55 -> so I can create my own provider for Terraform."

1967.67 -> And somebody else who's using CloudFormation might think,

1970.647 -> "Okay, let's use that CloudFormation custom resources."

1974.99 -> So you get all the different solutions

1977.27 -> within your organization

1978.38 -> and if you want to make some change in your unicorn tracker,

1981.86 -> you have to tell all your colleagues,

1984.027 -> "Hey, there's a change in the unicorn tracker.

1985.91 -> Everybody needs to change it

1987.337 -> and we will do it Saturday night."

1989.36 -> It will completely go wrong.

1991.31 -> Everybody has trouble with rolling back, etc.

1993.83 -> And with the solution, if you look at this diagram here,

1997.85 -> if you want to change something in the unicorn,

2000.67 -> it's basically updating the lambda function,

2004.15 -> updating the CloudFormation file.

2006.22 -> Share the CloudFormation file

2007.42 -> with all the teams that use CloudFormation,

2009.91 -> share the module with all the teams that use Terraform,

2012.67 -> and everything is automated and can be tested.

2016.63 -> And if you use your versioning correctly,

2019.12 -> it can be rolled out in separate weeks.

2026.74 -> Few next steps for Sohan.

2029.29 -> - Right. Thanks.

2031.441 -> Okay, can you hear me?

2034.182 -> Hello? Yeah, hello?

2037.09 -> Yeah, okay, so earlier Martijn took a poll

2039.37 -> of how many people use CloudFormation and Terraform.

2041.74 -> Any CDK users here? Anyone?

2044.23 -> Not as many, right?

2045.85 -> Which is, this is good.

2048.82 -> As Martijn said, this is a solution that we see

2051.91 -> that when there are large organizations,

2054.28 -> teams tend to use their own set of tools,

2056.14 -> services, etc, right?

2057.7 -> So maybe there is another team that uses

2060.37 -> a different infrastructures code provider.

2062.68 -> So how do you integrate that, as well?

2065.17 -> You could potentially also use AWS, CDK,

2068.35 -> or cloud development kit.

2070.72 -> For those of you all not familiar,

2072.01 -> and maybe I can spend a little more time on CDK,

2074.959 -> 'cause there doesn't seem to be as many users here.

2077.71 -> CDK is open source, it is infrastructure as code.

2081.49 -> And again, you can, you know,

2082.78 -> provision, and build, and deploy resources in the cloud.

2086.47 -> The difference though is you can do that

2088.39 -> using your favorite programming language.

2090.91 -> In CloudFormation and in Terraform you were using

2093.67 -> a very structured way like YAML or JSON or HCL.

2097.45 -> But here you could use type script, JavaScript, Java,

2100.9 -> Python, Go, and C#, if I'm not mistaken, right?

2105.01 -> And also you can do programming things,

2108.31 -> like doing conditioners, for loops, FL statements,

2112.24 -> and all of that.

2113.17 -> So imagine creating cloud infrastructure

2115.48 -> using your typical programming languages.

2118.78 -> How CDK works is,

2119.89 -> it's sort of like a layer above CloudFormation.

2122.83 -> So you have stacks very similar to CloudFormation stacks,

2125.977 -> but you also have these things called constructs.

2129.25 -> So in CDK there could be three levels of a construct.

2132.82 -> A level one construct is a direct mapping

2135.76 -> to a cloud formation resource.

2138.04 -> So for instance, in CloudFormation,

2139.75 -> if you have CFN bucket as an S3 resource,

2143.2 -> an L1 construct in CDK directly maps to that.

2147.7 -> An L2 construct is a curated construct by the AWS team

2152.95 -> where you have the best practices of an S3 bucket, yeah?

2156.61 -> So you don't have to mention every single variable

2160.9 -> or different things that you want to do about the S3 bucket.

2165.58 -> If you say new bucket using the best practices

2168.64 -> that we have curated for you, it creates that.

2172.33 -> And in an L3 construct, you can basically use patterns

2175.72 -> that have already been developed, right?

2177.76 -> You can create your own, you can share your own,

2179.92 -> but you could also use existing L3 constructs.

2183.52 -> So for instance,

2184.353 -> you could use a pattern for a serverless database,

2186.88 -> with a queue and a lambda function,

2189.07 -> and just reference that with one line of code and boom,

2192.04 -> that whole pattern is created for you.

2195.25 -> The commands in CDK, again,

2198.286 -> follow a very similar, sort of, syntax.

2201.16 -> When you do something like a CDK init,

2202.67 -> it initializes and builds a new project

2206.08 -> there is a CLI, of course.

2208.18 -> When you say CDK synth, it synthesizes, you know,

2211.69 -> the templates and assets that you have already defined.

2214.96 -> And when you say deploy

2216.37 -> it deploys it via CloudFormation, yeah?

2220.27 -> Very often when I talk to my friends who do a lot of front

2223.21 -> end dev work and, you know, are familiar with programming

2225.85 -> but not necessarily on the cloud side of things, I say,

2228.437 -> "Hey, CDK is the place for you. Go check it out."

2230.98 -> So, do check it out.

2233.65 -> You could do this entire thing

2235.96 -> with custom resources, Terraform providers.

2238.57 -> You could also use CDK.

2241.06 -> And just to mention,

2241.893 -> there is Terraform support in AWS CDK, as well.

2245.8 -> So there is something called CDKTF,

2248.83 -> another acronym for you,

2250.18 -> which is Cloud Development Kit for Terraform, right?

2253.54 -> And using this you can, you don't need to learn HCL.

2257.813 -> So you can actually still program

2259.93 -> in your own programming language

2261.91 -> and still access the same resources.

2265.18 -> Typically we suggest people to use CDKTF if you want,

2269.825 -> you know, a really strong abstraction layer here,

2273.16 -> or if you just prefer programming it in, you know,

2275.798 -> like a programming language of your choice.

2278.8 -> So as we've seen in this demo, you could use CDK,

2283.24 -> you could use Terraform, you could use custom resources.

2285.85 -> And with this sort of implementation, you know,

2287.74 -> you could still achieve what the customers

2290.86 -> that Martijn was speaking about, really achieved.

2294.7 -> For next steps and resources,

2296.11 -> and maybe you wanna take a screenshot of this,

2297.97 -> all of the code that you saw on the slides and in the demo,

2301.75 -> is available on github.com/tfworks.

2305.17 -> Again, shout out to Dennis,

2306.61 -> and also a couple of other people,

2307.81 -> Anton and Shrimas for helping us with this session.

2311.65 -> Do read a little more on custom resources

2314.23 -> if you haven't used it already.

2316.48 -> Again, I think that there is a lot of power

2318.76 -> in being able to do it.

2320.42 -> And especially if you're working with third-party resources

2324.04 -> that you want to automate using CloudFormation.

2327.13 -> And also do checkout Terraform Providers,

2330.4 -> and that's the link right there.

2333.46 -> Before we end, like a quick wrap up and summary,

2336.1 -> and also a bit of a disclaimer.

2338.89 -> So as a summary,

2340.96 -> we started out with what our custom resources were

2344.71 -> and how a custom resource needs a custom provider.

2347.74 -> And you could do that in CloudFormation.

2349.66 -> We spoke about Terraform and Terraform Providers

2351.97 -> and then integrating the two,

2353.74 -> and as next steps,

2354.73 -> we spoke about doing the same thing with CDK, as well.

2358.99 -> Now, this has been proven to work with both legacy

2362.23 -> and modern applications at scale,

2364.9 -> but we are not saying

2365.733 -> this is the only way for you to do this, right?

2368.11 -> This is a way to solve a problem.

2370.87 -> My favorite answer to anything in technology is,

2373.577 -> "It depends", right?

2374.83 -> It really depends on your SLAs, on your team,

2377.26 -> on your dependencies, on your tech stack.

2379.6 -> So this could be one approach, but like I said,

2382.39 -> it's worked at scale for a legacy app

2384.82 -> and for modern applications, as well.

2387.43 -> So you know, you could expect results with it.

2390.55 -> Just wanna say thank you again.

2391.66 -> Hope you have a good rest of re:Invent.

2393.73 -> Martijn and some of the team will be here for questions.

2397.39 -> As luck would have it, my next session's one hour from now,

2400.12 -> so I have to power walk to the room,

2402.67 -> but they will be there for questions.

2404.47 -> Thanks again and you soon.

2406.154 -> (audience members clapping)

Source: https://www.youtube.com/watch?v=RxTVpqOZ62c