AWS re:Invent 2020: Testable infrastructure: Integration testing on AWS
AWS re:Invent 2020: Testable infrastructure: Integration testing on AWS
Reliable integration testing gives developers the confidence needed to promote the changes from testing to production environments. But how should you think about implementing tests for your applications? In this session, learn integration testing best practices with AWS CodePipeline, AWS CodeBuild, and AWS Step Functions.
Learn more about re:Invent 2020 at http://bit.ly/3c4NSdY
Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4
#AWS #AWSEvents
Content
1.52 -> hi everyone and welcome to re invent
4.72 -> my name is natalie white and i'm an
6.799 -> enterprise solutions architect at aws
9.36 -> and this session is devops 301 testable
12.719 -> infrastructure
13.759 -> integration testing on aws now before
17.6 -> i came to aws i was a java developer for
20.48 -> 15 years
21.76 -> and i don't care what your
25.039 -> role is or how long you've been in tech
27.199 -> at some point
28.08 -> this has happened to you you make some
31.039 -> changes
31.679 -> today we're talking about infrastructure
33.28 -> configuration
35.04 -> you do a quick code review
38.079 -> on your own you maybe have a teammate do
40.16 -> a code review
41.6 -> run some unit tests and maybe you even
44.48 -> deploy your changes to
46.32 -> a non-production environment just to
48 -> make sure that the deployment is going
49.52 -> to be successful
51.52 -> everything looks good so you deploy your
54.16 -> changes to production
56 -> and maybe immediately you get reports
58.96 -> that something has broken
60.16 -> if it's related to connectivity or
62.879 -> authentication
64.4 -> or maybe weeks or months down the road
67.04 -> you find out that a change that you made
69.04 -> has affected
70.159 -> some downstream process that was
72.32 -> dependent on your infrastructure
73.92 -> configuration like a billing or
75.92 -> reporting process
77.84 -> either way when you find out there's an
79.52 -> issue there's a mad dash to try
81.439 -> and fix it as fast as possible and
83.759 -> figure out what happened
86.159 -> wait a minute if you're an application
88.64 -> developer
89.36 -> like me you had a mental record screech
93.439 -> right there between the first and second
95.52 -> images
96.799 -> what about integration testing we've
99.2 -> been using integration testing to verify
101.759 -> the interaction between different layers
104.32 -> of our applications
105.6 -> and dependencies on other services
108.96 -> and downstream processes for a long time
112.159 -> and there's no reason why integration
114.479 -> testing
115.2 -> can't also be a part of our
116.96 -> infrastructure cicd process once we're
119.6 -> doing infrastructure as code
121.6 -> we want to break this cycle of
123.759 -> unintended consequences
125.439 -> and build our confidence in the changes
128 -> that we're making on a regular basis
130.879 -> sound good while we're talking about
134.4 -> infrastructure
135.28 -> and integration testing it is good
138.08 -> always good to define
139.36 -> what that means and when we're defining
142 -> what something means sometimes it helps
143.44 -> to define
144.16 -> what it's not it's integration testing
147.44 -> is not
148.48 -> unit testing right it's not a static
150.4 -> analysis of your configuration or your
152.959 -> cloudformation template if you're using
154.84 -> cloudformation
156.08 -> you want to programmatically verify the
159.36 -> state of your infrastructure as a whole
162.16 -> after
162.879 -> it's been deployed and
166.319 -> running these sorts of tests can expose
168.879 -> problems across
170.16 -> whatever complex dependencies that your
172.64 -> infrastructure has
174.16 -> and these dependencies aren't always
176.16 -> obvious from looking at a single
177.76 -> resource or
179.36 -> a single cloud formation template it
181.28 -> could be across services across multiple
183.44 -> applications
184.319 -> roles and permissions so running these
187.2 -> sorts of tests
188.08 -> can give you visibility into those
190.159 -> dependencies and verify that you haven't
192 -> broken anything
194.319 -> these tests can also expose downstream
196.4 -> failures
197.36 -> in a way that unit tests and eyeballing
199.76 -> your changes can't
201.68 -> the changes that you're making could
204.08 -> have
204.879 -> a wider scope than just the functional
206.879 -> requirements of your
208.4 -> environment maybe multiple teams or
211.12 -> multiple business processes or using
213.04 -> what you're deploying
214.08 -> and maybe you want to verify that best
215.68 -> practices and governance requirements
217.92 -> are being met
219.2 -> and maybe these are things that are
220.64 -> application specific
222.48 -> that can't be verified by an
224.72 -> organization wide or an account wide
227.04 -> guardrail maybe some tagging
231.599 -> another thing that integration testing
233.439 -> can be used for is to expose
235.68 -> any security gaps so this can include
239.04 -> revoking access that you didn't mean to
241.12 -> invoke revoke
242.56 -> or potentially more damaging opening up
245.599 -> access that you didn't intend to
247.599 -> so verifying your changes after they've
249.76 -> been deployed
250.64 -> can help find and visualize those
254 -> security gaps without having to do any
257.519 -> complex
258.56 -> static analysis on your cloud formation
260.56 -> template
261.6 -> but the main thing that you want to take
262.88 -> away is that you want to run these
264.08 -> integration tests
265.52 -> on an environment that is
266.96 -> production-like as possible
268.639 -> because that repeated consistent
271.04 -> deployment methodology can reduce the
273.199 -> risk
273.6 -> of untested changes going into
276 -> production
278.56 -> so now that we know what integration
279.759 -> testing is
281.6 -> i'd like to talk about how you can use
284.56 -> it
285.12 -> not just on premises but also obviously
287.919 -> in aws
289.04 -> it's not integration testing isn't
290.56 -> unique to the cloud and in fact you can
292.639 -> multi
293.12 -> you can implement these same patterns
295.6 -> that we're going to go through today
297.12 -> on your on-premises infrastructure cicd
300.16 -> processes
301.199 -> but a few things are different when
302.88 -> you're deploying infrastructure in the
304.4 -> cloud and so i want to go through those
309.199 -> so the first thing that's different
310.4 -> about deploying infrastructure
311.52 -> on-premises versus aws
313.6 -> is that on-premises we get used to
315.52 -> having stale and inconsistent
317.52 -> environments
318.4 -> so if you're not already provisioning
320.32 -> your infrastructure as code you know the
322.24 -> pain that's associated with
324.32 -> having these stale environment
326.4 -> configurations and the most obvious
328.24 -> example of this one that i have from my
330.479 -> own experience
331.6 -> is say a performance environment that
333.52 -> hasn't been touched or used in
335.52 -> several release cycles you have to
337.52 -> figure out when was it last updated
339.36 -> what changes have been made since then
341.52 -> what order do i need to apply those
342.96 -> changes in multiple support tickets
345.039 -> things like that and deploying
346.639 -> infrastructure as code
348.08 -> kind of resolves all of those
351.199 -> stale environments because you can
352.8 -> deploy these changes programmatically
354.96 -> and immediately across all of your
356.639 -> different environments
358.16 -> without having to submit a bunch of
359.6 -> tickets and keep track of which ones are
361.36 -> being applied
364.08 -> the second difference is that
365.44 -> on-premises there's just no way to
368.08 -> justify
368.8 -> buying hardware or provisioning a
370.319 -> production-like environment
372.08 -> on-premises for a system that's not
374.56 -> going to be handling a production
375.759 -> workload in fact
377.039 -> sometimes a single server is running
378.8 -> every layer of your application
380.8 -> and sometimes it's running multiple
383.199 -> applications
384.8 -> right it you you can't have a
388.4 -> production-like
389.36 -> system on-premises because of the
391.919 -> financial impact
393.44 -> but when you're using infrastructure as
395.039 -> code and deploying it to the cloud you
397.36 -> can provision a test environment just to
399.28 -> run those integration tests
401.039 -> and then destroy it when you don't need
402.72 -> it anymore and you've only paid for the
404.16 -> few minutes that those tests have run
407.36 -> and the third and most important
408.56 -> difference is the culture change that
410.319 -> comes along with
412.16 -> moving into the cloud on process we
415.36 -> have gotten really used to having
416.639 -> functional silos
418.4 -> for our development and release
420.16 -> processes the it team manages
422 -> infrastructure the application developer
424.4 -> tosses artifacts over the fence to
426.08 -> deploy on that infrastructure
427.919 -> the database team only does databases
430.639 -> the security team
431.759 -> can really only look at static reports
433.759 -> from code analysis or running periodic
436.08 -> pen tests against production and no one
438.56 -> on the technical side has any idea what
440.4 -> anything costs
441.28 -> because the hardware in the data center
443.12 -> is already a sunk cost
445.44 -> but the only thing we did know is that
447.28 -> there were never enough servers to do
448.72 -> what we wanted
449.52 -> in non-prod systems and so then when
452.24 -> issues occurred
453.28 -> it's always a struggle to determine who
455.12 -> has the responsibility to determine a
457.199 -> root cause and who has the authority to
459.12 -> fix it
460.72 -> but when you're deploying things into
461.919 -> the cloud and using infrastructure as
463.68 -> code
464.4 -> those silos are flattened and teams own
467.199 -> everything that they build
468.56 -> the it team can automate the heavy
470.4 -> lifting of what they were doing before
471.919 -> and focus more on strategic initiatives
474.479 -> application developers can provision
477.44 -> infrastructure
478.479 -> on their own without having to wait for
479.919 -> another team's availability
482.4 -> and you can migrate fully managed
484.319 -> database services and focus on the
486.319 -> business value of the data itself and
488.08 -> how your application interacts with it
490.639 -> security teams can now get full
492.72 -> visibility into the entire system
495.599 -> and implement guard rails and
496.96 -> remediations to prevent breaches ahead
499.12 -> of time
500.08 -> and what each implementation costs is
502.24 -> transparent to the teams that have
503.68 -> ownership over their own cost
505.12 -> optimization
506.319 -> so it's a paradigm shift for many of our
508.319 -> customers but it creates the autonomy
510.56 -> and efficiency that allows your teams to
513.2 -> use the skills that they already have
515.039 -> to do more than they could before but
517.519 -> with that ownership comes a lot of
519.039 -> responsibility
520.32 -> to put processes like integration
522.24 -> testing in place
523.36 -> to guard against the kind of production
525.12 -> issues that we're talking about
530.08 -> all right now we've talked about what
531.6 -> integration testing is
533.12 -> why it's important and who's responsible
536.48 -> so i have three patterns that i want to
538.24 -> go through to show you
539.76 -> how you can start implementing
541.279 -> integration tests in your own
543.839 -> cicd process and they're in order of
547.279 -> complexity
548 -> so they kind of build on each other
551.36 -> pattern one is the simplest we can start
553.839 -> invoking lambdas from
555.2 -> our code pipeline this is a 300 level
558.399 -> talk
558.8 -> so i'm already assuming that you know
560.8 -> about cicd processes
563.12 -> and our code suite of developer tools
566.48 -> if you haven't go check them out they're
568.48 -> awesome and they're what i use to build
570.24 -> this
570.72 -> these demos in my presentation so we
573.68 -> have our pipeline we have a code
575.04 -> repository we have a build that happens
577.76 -> and what we want to do is deploy the
580.08 -> same cloud formation stack
582.16 -> in multiple environments and have tests
585.04 -> running in between to verify
587.12 -> and guard against breaking changes going
589.68 -> into production
592 -> so you'll notice that my test
594.16 -> environment and my production
595.36 -> environment i intentionally left blank
597.44 -> it doesn't matter
598.48 -> what you're provisioning or what you're
600.16 -> deploying it could be a single s3 bucket
602.8 -> it could be a multi-tier web app it
604.88 -> could be a fargate cluster
606.24 -> it doesn't matter all that matters is
608.64 -> that your tests are running against that
610.48 -> test environment and able to assess
612.64 -> the status of that infrastructure after
614.64 -> it's been deployed
617.04 -> you'll also notice i have three lambdas
619.36 -> here
620.16 -> and they're kind of floating out in the
621.92 -> middle of nowhere because
623.68 -> pattern one assumes that you've already
626 -> deployed these lambdas
627.6 -> in either a manual way or a separate
630.48 -> pipeline and they're just sitting out
631.92 -> there
632.24 -> waiting to be invoked when we run our
635.2 -> deployment pipeline
636.959 -> i've divided them up into subject
640.079 -> domains so i have a set of default tests
642.24 -> that could maybe
643.279 -> look at governance requirements i have a
646.32 -> networking test suite that will test a
648.32 -> networking configuration
650.399 -> i have a security test suite that will
652.56 -> look at my security configuration
654.88 -> and you can divide up your tests the way
657.44 -> that you want
658.88 -> but you do want to find a balance
660.24 -> between having a single test running
662.399 -> in a lambda because then you have a lot
664.56 -> of lambdas
665.92 -> and it's not very efficient because you
667.6 -> have to wait for those to be invoked
670.32 -> and you the other side of the spectrum
672.56 -> is you have all of your tests running in
674 -> a single lambda which isn't very
675.44 -> efficient because you have to wait for
676.8 -> them to all run sequentially
678.64 -> and because you don't have a lot of
680.56 -> visibility into
682.399 -> where you're at during the test run and
684.88 -> what happened
688.32 -> so as a part of our pipeline we're going
690.64 -> to create
691.68 -> some lambda invocation actions or you
693.839 -> could create a custom resource whatever
695.92 -> you prefer
697.279 -> to go into and execute those lambdas
700.399 -> which will run tests against the test
702 -> environment and assuming everything is
703.519 -> good
704.079 -> then you'll deploy your changes to
705.519 -> production
708.48 -> so i'm not going to demo this pattern
710.639 -> because
711.519 -> it's really just a place for you to get
713.36 -> started testing out integration testing
715.68 -> in your own
717.2 -> infrastructure we're going to iterate on
719.92 -> that for pattern two
721.36 -> because there are a couple drawbacks to
723.2 -> this framework the advantages of it is
725.44 -> as simple and the test scope is readily
727.2 -> apparent from your pipeline you can see
729.76 -> which tests are being run in what order
731.76 -> and they're all failing independently
733.92 -> but the lambdas are provisioned outside
736.079 -> of the pipeline
737.04 -> which means that our tests are hanging
739.12 -> out somewhere else not
740.399 -> in our infrastructure repository and
743.44 -> there's a separate process that you have
744.88 -> to keep track of
746.16 -> to make sure that everything is in sync
748.16 -> when you run the pipeline so you don't
750.079 -> have
750.48 -> issues being reported when they're not
751.92 -> issues and you're running the latest set
753.76 -> of tests
755.279 -> the other drawback is that potentially
757.92 -> if you continue with this pattern
759.68 -> into the future and you keep adding
761.279 -> lambdas and adding tests you could
762.88 -> eventually hit code pipeline action
764.8 -> limits
765.6 -> or if you're using custom resources and
767.68 -> custom actions
768.72 -> there are limits there as well so if you
771.839 -> continue with this pattern
773.6 -> you may run into those limits and there
775.92 -> are other ways to
777.519 -> execute your tests and orchestrate them
779.519 -> that provide you a little bit more
781.04 -> flexibility
783.76 -> like i said pattern 2 is going to build
785.36 -> on pattern one we have the same pipeline
787.36 -> and the same deployments happening
789.279 -> but this time instead of having the
790.8 -> lambdas sitting out there waiting to be
792.16 -> invoked we're going to use
793.04 -> cloudformation to deploy them
794.8 -> as a part of our pipeline
797.839 -> so those lambdas will be in our code
801.12 -> repository
802 -> right along with all of our code that's
804 -> making our infrastructure changes
806.079 -> and we'll have the same lambda actions
808 -> as a part of our pipeline to
810.48 -> deploy or to execute those lambdas which
813.6 -> will execute the tests
815.12 -> and then go to production and assuming
817.04 -> everything is good
818.32 -> i do have a demo to show you
821.839 -> how this is going to work so we'll get
824.639 -> into that now
828.24 -> now i built my demo using cdk for
830.639 -> typescript which builds my cloud
832.16 -> formation templates and you can see
833.92 -> three
834.56 -> templates three stacks that have been
836.24 -> deployed here my demo stack
838.16 -> just contains the pipeline that i just
840.32 -> showed you it's a convenience for me to
842.48 -> deploy it as a cloud formation template
844.399 -> and we're not going to dive
846.16 -> deep into that because it doesn't matter
848.24 -> how you build your pipeline really
851.44 -> you can also see that there is a test
853.44 -> environment stack and a prod environment
855.199 -> stack just like we saw in the
856.56 -> architecture diagram
858 -> and for my demo all that's being
860.32 -> deployed into these
861.6 -> is a single s3 bucket and a vpc
865.44 -> there are two instances of the same
867.519 -> template that are using a property to
870 -> to rename them depending on which
871.92 -> environment i'm deploying to
875.12 -> so you can see the resources that i'm
876.8 -> deploying here again are
878.48 -> my s3 bucket and a vpc that's
881.6 -> just using the default configuration so
883.6 -> it has all of the subnets and everything
885.6 -> that comes with a default vpc
890.079 -> and you can see that my production
892 -> environment stack is deploying the exact
894 -> same resources
895.44 -> renamed so that we can easily
896.8 -> differentiate them
904.399 -> all right so that's what's being
905.44 -> deployed i have one more cloudformation
907.76 -> stack that's being deployed
909.279 -> and that's for my lambda tests
912.48 -> so we'll go take a look at that one
921.279 -> so this is deploying all three of my
924 -> lambda tests
925.279 -> and all of the roles that they need to
928.639 -> be able to run their tests but the main
930.399 -> thing i want you to take away from this
931.759 -> is that as a part of my deployment i'm
933.44 -> publishing those lambdas deployed arn's
936.48 -> so i can pass those along in the code
938 -> pipeline for my execution
940 -> stages
948.72 -> all right now we can actually look at
950 -> those functions
952.24 -> you can see i have my three functions
953.839 -> here that are being deployed
955.36 -> and there's an additional lambda there
956.8 -> that's going to be my invocation action
958.72 -> as a part of my pipeline
960.88 -> we don't really need to look at that
961.92 -> because all it's doing is invoking my
963.36 -> lambdas
964.32 -> but my tests actually have all of the
966.48 -> same code within them so we'll
968 -> explore them just by looking at the
969.519 -> defaults stack for now
971.759 -> you can see here i'm using a python
973.68 -> testing framework
974.88 -> you can use whatever framework you're
976.48 -> familiar with if you're a powershell
978.079 -> person you can use pester if you don't
979.68 -> want to use a framework at all
980.959 -> you don't have to this was just for ease
984 -> of demo and reporting and things like
986 -> that
987.6 -> and we can look into our code and see
989.04 -> what tests are actually being run
990.72 -> as a part of each of our test suites
995.759 -> each of them is using this same
997.839 -> framework of going and getting
999.759 -> my deployed cloudformation stack and
1003.199 -> using the boto3 resource api to
1006.72 -> get the deployed resources
1010.16 -> and then we can use those resources to
1012.959 -> assert that certain things are true
1014.8 -> about them
1019.12 -> so the test that i'm running here i'm
1020.639 -> gonna go make sure that my s3 bucket has
1022.48 -> some kind of tagging going on
1024.079 -> and it will anyway because it's being
1025.679 -> deployed by cloudformation
1027.679 -> but i also want to verify that my
1030.079 -> default tag that i've put there
1032.48 -> is one of the tags because maybe i'm
1035.199 -> using that for a billing process or a
1036.959 -> reporting process downstream this could
1038.799 -> be something that's not
1039.919 -> needed for the functionality of an
1042.079 -> application running on this
1043.199 -> infrastructure but it might be needed
1044.72 -> for a downstream process
1046.959 -> then i'm testing and making sure that my
1048.559 -> s3 bucket is versioned again
1050.48 -> my application may not even need that
1052.24 -> but for auditing purposes maybe it does
1054.559 -> so your test can really be anything here
1057.039 -> this is just an example of what you can
1058.799 -> do
1059.28 -> with that resource api and verifying the
1062.16 -> deployed infrastructure
1066.32 -> we'll move on to my networking test it
1068.72 -> just goes and looks at my vpc
1070.559 -> and it makes sure that
1073.679 -> the vpc exists that i have
1076.72 -> a cider block that's actually the
1078.48 -> default cider block i didn't
1079.76 -> intentionally set that
1082 -> and then it's going to make sure that
1083.12 -> all of my vpc subnets are available
1085.6 -> again these tests can really be anything
1088 -> it's up to you
1094 -> my security test i just have one test in
1096.4 -> there and it's for the s3 bucket and i'm
1098.08 -> verifying that it doesn't have any acl
1100.24 -> access configured actually my bucket is
1103.679 -> blocking all private access but this is
1105.679 -> one of those preventative tests that you
1107.2 -> could put in
1108 -> place to make sure that no one makes
1110 -> changes in the future
1111.12 -> that might cause a security issue
1115.52 -> all right that's it for lambda code now
1118.24 -> we can go look at my pipeline
1119.84 -> it looks exactly like my architecture
1121.44 -> diagram i have a source
1123.2 -> stage a build stage a deployment stage
1127.52 -> and a test stage and then of course
1129.6 -> after that we deploy to production
1132.24 -> of course this has already been a
1133.6 -> successful run but what we want to do
1135.6 -> for the demo is make some breaking
1137.039 -> changes and see what happens
1140.08 -> so this is cdk typescript code and it's
1142.64 -> how my environment stack is being built
1145.36 -> and you can see that my multi-stack
1147.44 -> property there is determining whether
1149.12 -> this is a
1150.08 -> production environment or a test
1151.6 -> environment and it's changing the name
1153.919 -> of my bucket and my bpc depending on
1155.919 -> the value of that property i'm deploying
1158.799 -> my s3 bucket with all the configuration
1160.799 -> that goes along with it and my default
1162.84 -> vpc as a part of the cdk framework
1166.4 -> i'm going to comment out that my bucket
1168.08 -> is going to be versioned which will
1169.28 -> change the status to suspended
1172.24 -> and i'm going to remove that tag because
1174.24 -> as an application developer and
1175.6 -> infrastructure developer i don't think i
1177.039 -> really need it
1180.64 -> so we'll promote those changes push
1182.48 -> those changes into the code repository
1184.799 -> and see what happens
1195.12 -> all right
1198.4 -> so you can see that my source
1201.44 -> has now been updated my breaking changes
1204.24 -> have been recognized and my pipeline has
1206 -> started and
1210.64 -> these pipelines actually take around six
1213.44 -> or seven minutes to complete
1215.2 -> but i've sped up the video just for your
1218.48 -> own sake
1219.44 -> of watching it
1223.52 -> now we've built we're going to deploy
1226.32 -> our
1226.72 -> cloud formation templates that were
1228.159 -> built as a part of the build step using
1229.84 -> cdk synth
1232.24 -> and now we can invoke the test that we
1234.24 -> just deployed
1235.76 -> against the environment that we deployed
1237.679 -> okay so we can see that our default test
1240 -> failed
1241.12 -> and i'm returning the details of those
1243.44 -> failures
1244.799 -> back from my lambda so that you can see
1247.039 -> from the pipeline what failed
1248.88 -> my networking and security tests were
1250.96 -> fine so those succeeded
1252.48 -> but because my default test failed i
1254.799 -> didn't deploy
1255.52 -> anything to production
1266.159 -> all right
1269.6 -> so as you can see adding integration
1271.28 -> tests to an existing pipeline
1273.28 -> is pretty easy by using lambda tests
1277.36 -> and we've resolved the issue of our
1279.52 -> tests not being up to date and our tests
1281.28 -> not being close to our code because
1282.64 -> they're part of our same repository and
1284.24 -> they're being deployed as a part of the
1285.6 -> same pipeline
1287.2 -> however this approach still has some
1290.24 -> drawbacks
1291.52 -> using lambdas invoked directly from
1293.76 -> cloud or from
1295.28 -> code pipeline can be a very brittle way
1298.559 -> to run your tests
1300.159 -> if you have a bunch of test suites that
1302.64 -> are running in a combination of in
1304.159 -> parallel
1304.96 -> and in a series if one lambda breaks
1307.679 -> early on in the process the rest of them
1309.52 -> aren't going to run at all because the
1310.88 -> pipeline is going to stop right there
1313.52 -> you have very limited workflows you can
1315.28 -> only choose whether they're going to run
1316.88 -> in parallel or series you can't add any
1319.28 -> conditional statements or anything like
1321.12 -> that
1321.84 -> your pipeline is just going to run and
1323.52 -> it's going to fail at the first failure
1325.44 -> you don't have any option to retry until
1327.28 -> you make some changes
1330 -> we can iterate on this pattern for a
1331.84 -> third the time
1333.44 -> and add aws step functions
1337.52 -> so this pipeline is exactly the same as
1340.559 -> my first two patterns and instead of
1342.96 -> deploying lambdas
1344.32 -> we're going to deploy the lambdas along
1346.4 -> with a step function state machine
1348.88 -> that's going to define how those lambdas
1350.88 -> are going to be run
1352.24 -> and define a more conditional workflow
1354.96 -> that
1355.44 -> we can use to create a more dynamic test
1358.84 -> run
1360.32 -> we'll still use a code pipeline action
1363.12 -> to
1363.76 -> invoke that step functions state machine
1366.96 -> and as we do that the start state
1370.32 -> will go into in this case i've coded
1372.96 -> mine to
1373.679 -> still run in parallel like they were in
1375.36 -> the previous pipeline they'll run tests
1377.44 -> against the test environment
1379.36 -> and then they'll consolidate their
1381.36 -> output and return that
1382.64 -> back to the pipeline and then on to
1384.96 -> production if everything's okay
1389.039 -> the demo for this one is much shorter
1390.88 -> because we don't have to go through all
1392.24 -> of the build up of what's happening in
1393.84 -> the pipeline and what's being deployed
1395.52 -> i have the same framework that i had in
1397.52 -> pattern two
1398.559 -> in pattern three we have my demo stack
1400.64 -> which has my actual pipeline
1402.08 -> in it and we have the same test and
1404.559 -> production
1405.2 -> environment stocks being deployed they
1407.52 -> have the same exact resources in them
1409.679 -> nothing has changed so we won't spend
1411.12 -> any time looking into those
1413.679 -> my pipeline also looks exactly like my
1415.679 -> architecture diagram
1417.12 -> i'm deploying my step functions workflow
1420.48 -> and then i'm executing it in the next
1422.24 -> step
1424.799 -> and my lambdas also look very similar
1428.559 -> the same tests are being run but now and
1431.52 -> we still have a test proxy which is
1433.2 -> actually going to be invoking my state
1434.72 -> machine instead of my three lambdas
1437.44 -> and then we have an additional lambda
1438.96 -> here that's going to consolidate
1440.72 -> all of the test results from those three
1442.799 -> steps being run in parallel
1444.72 -> and return those back to the pipeline
1448.08 -> the nice thing about step functions is
1449.919 -> you can get a historical view
1451.76 -> of previous runs and you can view those
1454 -> results graphically
1455.44 -> so we'll go look at a previous
1457.279 -> successful run to see what's actually
1458.88 -> happening
1460.96 -> you can see i have my start state i'm
1463.2 -> going to run all three of my tests in
1465.279 -> parallel
1466.96 -> i'm going to consolidate the results and
1469.2 -> then i'm going to determine whether any
1470.72 -> tests failed
1471.6 -> from those results in this case
1473.919 -> everything passed
1474.96 -> so we're good but now that means we get
1477.76 -> to go make some breaking changes and see
1479.279 -> what happens
1481.919 -> you can look at the trace of
1485.2 -> where steps were entered and exited you
1488.159 -> get a lot of really great visibility
1489.52 -> inside of step functions
1495.52 -> so we'll make the same breaking changes
1497.2 -> as before we'll remove versioning
1499.279 -> and we'll remove our tag
1507.52 -> and we're also going to make a change to
1509.6 -> the security test so that we can see how
1511.44 -> those test results are consolidated
1513.52 -> so i have this commented out test and
1515.6 -> i'm going to uncomment it so it runs
1517.76 -> and i'm still doing my acl access test
1520.48 -> but now i'm
1521.36 -> making sure that there are no public
1523.44 -> subnets inside of my vpc
1527.76 -> so we'll uncomment that test
1536.24 -> what it's actually doing is it's going
1537.84 -> and grabbing the route table that's
1539.039 -> deployed as a part of my vpc
1541.36 -> and it's looking for an internet gateway
1543.12 -> and if that internet gateway has any
1544.559 -> subnet associations
1545.919 -> i'm going to report that there's a
1547.44 -> public subnet in this vpc
1550.799 -> so we'll push those changes to our
1552.4 -> pattern 3 git repository
1554.32 -> which will trigger our code pipeline
1564.559 -> our source changes were picked up so
1567.36 -> we'll move on to the build process
1570.24 -> the build process is running the same
1572.24 -> typescript tests
1573.36 -> and cdk synth to generate my
1575.44 -> cloudformation templates
1577.919 -> we're deploying that test environment
1579.919 -> and my step functions
1581.679 -> cloudformation template
1585.6 -> and now we're running our state machine
1593.52 -> all right it failed so because i'm
1595.6 -> consolidating all of the test results i
1597.279 -> can see them all here in my pipeline
1599.36 -> what failed and if i want additional
1601.76 -> visibility into
1603.44 -> what failed and why i can go into
1607.12 -> step functions and go look at that most
1609.679 -> recent
1610.96 -> invocation and view what happened so
1613.76 -> you'll see
1614.4 -> all of there weren't any issues actually
1616.32 -> running my tests so you can see that all
1618 -> of those were successful
1619.44 -> however the output that they're
1620.88 -> providing is showing that there were
1622.559 -> some issues with the tests that were run
1624.4 -> inside of them we consolidate those
1627.44 -> results so we're combining the results
1629.279 -> from the default test and the security
1631.12 -> test into a single list of things that
1632.88 -> went wrong
1634.559 -> and then we're using that conditional
1636 -> did any test fail to determine
1637.919 -> what state our our state machine will
1640.159 -> end in
1646 -> all right
1649.52 -> so this keeps the same advantages as
1651.6 -> pattern two our tests are always being
1653.679 -> updated they're close to our code but it
1655.76 -> adds a couple more advantages
1657.44 -> to over the lambda only integration
1660.96 -> tests
1661.679 -> the consolidated test results can be
1664.24 -> propagated to dashboards or email
1666.08 -> reports or a persistent data store
1668 -> for immediate visibility into all of the
1670.24 -> tests that failed
1671.679 -> you can use all of those service
1673.2 -> integrations with step functions
1674.88 -> to configure notifications or
1677.84 -> remediation actions depending on how
1679.76 -> severe
1680.32 -> those issues are and step functions
1682.88 -> allow you to orchestrate the tests and
1684.64 -> not just
1685.12 -> parallel or in series you can have
1687.12 -> conditional flows
1688.64 -> like say if this test fails i don't want
1690.799 -> to run all these other tests
1692.48 -> or maybe i want to run the same state
1694.799 -> machine against the development
1696.24 -> environment
1697.279 -> and my integration test environment and
1699.2 -> maybe even against production
1700.88 -> and only run a subset of tests against
1703.919 -> each environment
1705.12 -> so that i can speed up my build time
1708.96 -> you can configure error handling and
1711.279 -> retries as a part of step functions
1713.44 -> and if you create the pipeline itself
1715.6 -> within cloud formation with or without
1717.52 -> the cdk
1718.96 -> it's a pattern that can be used as a
1720.72 -> starting point for new projects and
1722.64 -> shared across
1723.44 -> teams so you can start propagating these
1726.799 -> best practices of running integration
1728.72 -> tests as a part of your infrastructure
1730.399 -> deployments
1731.36 -> into the future
1735.12 -> so we walked through three different
1737.679 -> patterns that you can use to get started
1739.919 -> adding integration tests to your cd ci
1742.399 -> cd pipelines
1743.76 -> and hopefully these gave you some ideas
1745.679 -> of how you can start to do this and
1747.52 -> build on those
1748.799 -> ideas moving into the future thanks a
1752.159 -> lot for
1752.88 -> listening to my talk and if you found it
1755.44 -> interesting and enjoyed it
1757.12 -> there is a similar talk or related talk
1759.679 -> in the builder's library
1760.96 -> suite of sessions and it's called
1764.24 -> testing software and systems at amazon
1766.64 -> from charlie roberts who's a senior
1768.24 -> software developer and test here at aws
1772.159 -> and don't forget to complete your
1773.76 -> session survey it really helps us to
1775.36 -> improve content going forward let me
1776.96 -> know how i did
1778.24 -> and thanks for watching my session
Source: https://www.youtube.com/watch?v=KJC380Juo2w