AWS re:Invent 2022 - Best practices for securing your software delivery lifecycle (DOP316)

Aug 16, 2023

AWS re:Invent 2022 - Best practices for securing your software delivery lifecycle (DOP316)

In this session, learn about ways you can secure your CI/CD pipeline on AWS. Review topics like security of the pipeline versus security in the pipeline, ways to incorporate security checkpoints across various pipeline stages, security event management, and aggregating vulnerability findings into a single pane of glass. Also discuss best practices, processes, and tools that can improve an organization’s ability to deliver applications and services in a secure manner.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

0.72 -> - My name is James Bland.

1.8 -> I am a principal solution architect

4.56 -> and I'm also the the global tech lead

7.05 -> for DevOps here in AWS and...

9.81 -> - Hi everyone I'm Curtis Rissi.

11.34 -> I'm a principal solutions architect focused

13.23 -> on our partner solutions for App Mod,

15.75 -> but in a past life I got to play

19.02 -> worldwide tech lead for for DevOps.

20.49 -> So lot of fun getting to see all of you here

22.74 -> and thank you for for giving us time I know there's a lot

25.53 -> of sessions you could have chosen

26.64 -> at this time slot so we really appreciate

28.53 -> you giving us your focus and time

29.79 -> and hopefully you'll get something useful out of this.

34.41 -> - So security challenges that we're hearing from from a lot

37.23 -> of customers that they're having

38.28 -> today right is one is there's a proliferation

41.16 -> of tools that are out there,

42.57 -> and a lot of times customers

44.557 -> don't know what to choose when, right?

47.4 -> So you know you've got like over a 100,

49.83 -> couple 100 tools that's available for security,

53.34 -> which is the best practices,

55.17 -> what are the best ones to use,

56.94 -> it's kind of hard to tell in the market right now.

59.85 -> Other problems that we're also hearing from customers is

62.43 -> it's hard to integrate like security and compliance

65.07 -> and regulatory to these DevOps

68.76 -> or these security lifecycle of software, right?

73.32 -> So it's hard to do these integrations

75.06 -> there's not really good integrating points

76.98 -> and so that's just another feedback that we're

79.17 -> hearing from customers.

80.55 -> And the last thing that we're

81.72 -> also seeing in terms of security challenges too right?

85.5 -> Is that organizations

88.59 -> are just finding it hard to... (chuckles)

95.31 -> You know to react to incidents that they're having

97.768 -> within the delivery lifecycle.

100.41 -> So when you have a security incident how do you react

102.6 -> to it right?

103.433 -> So you're moving into the cloud

106.44 -> what we're finding is

107.31 -> a lot of customers are seeing that the expansion

111.57 -> and the complexity of their environments are growing

116.43 -> and so when you have a security incident

118.02 -> how you know what to go to,

119.4 -> how do you know to react where you, you know,

122.34 -> when you get an alarm,

123.87 -> what services do you look up, how do you use logs,

125.88 -> things like that.

126.713 -> - Yeah when you think about the complexities though

128.58 -> of microservice based workloads

130.2 -> as you continue to move Monolist

131.76 -> to something like that it's very distributed.

134.28 -> And the complexity of that environment continues to grow

136.98 -> and when you're trying to manage the security footprint

139.17 -> of that it can be rather difficult

140.64 -> and so how do you do that effectively,

142.05 -> how do you automate that, how do you respond effectively?

145.02 -> And knowing what tools can actually

146.67 -> help you there is a really important challenge

148.41 -> that customers are faced with today,

150.09 -> so hopefully you'll get some of that out of this talk.

153.09 -> Yeah we'll definitely discuss that in detail.

156.06 -> - So why do we need a different approach?

157.83 -> So if historically if you look at how we used to do security

161.1 -> or how we do do security now at a lot

163.59 -> of different organizations,

164.97 -> is we look at it from the outside in.

167.97 -> So we basically spend a lot of time

169.74 -> trying to protect the edge.

171.15 -> We put more locks on the door

174.45 -> even though the windows are still open pun intended.

178.59 -> But yet at the same time it's still...

180.36 -> We're getting a lot of knowledge

182.34 -> and we're putting you know like

183.21 -> IPS IDS's and things like that

185.13 -> on the edge but it's really not fixing

186.69 -> the core problem right?

187.89 -> We're finding that attackers are becoming very creative

191.4 -> or bad actors I should say are becoming very creative

194.34 -> and they're finding different ways

195.39 -> to come in and they're no longer coming in from the edge

197.52 -> they're finding different ways

198.72 -> to come in through the windows

199.86 -> or the back door or things like that.

201.54 -> So no matter how many locks we put

203.16 -> on the door we're never gonna actually

204.66 -> really solve the problem

205.74 -> because that's not where bad actors are coming from anymore.

210.03 -> The other reason we need a different approach is

212.73 -> because I mean I don't know if like all of you,

214.71 -> like some of you have different roles,

216.18 -> but like as an engineer when I was doing

218.52 -> development work I would always find

220.41 -> that it was somewhat quite annoying

223.05 -> when I would build a package build an application

225.66 -> and then then send it down

227.1 -> the QA and then finally QA would send it

229.2 -> to operations team and then eventually like a year down

231.75 -> the road security would step in do a scan send

234.87 -> me a thousand different problems that I have.

238.47 -> How do I react to that?

239.64 -> I mean I got a thousand different things

241.38 -> that I'm trying to move forward with my

243.06 -> application trying to develop new features

245.07 -> and functionality I can't react to that so in a lot

247.74 -> of cases what I usually did was ignore it, right?

250.23 -> And so and that doesn't really

251.7 -> help me and that didn't really help

253.08 -> the organization as a whole.

254.64 -> So we needed a different approach.

259.11 -> So other external drivers that we're seeing

261.24 -> that's driving this changes between 2020

264 -> and 2021 we saw 650 percent increase in the number

268.62 -> of supply chain attacks that happened right I mean...

273.09 -> And of that so 98 percent we also saw a 98 percent increase

277.41 -> in the number of dependencies

279.45 -> that were used that were open source

281.7 -> into application package like previously

285.06 -> this number was around like 75 to 80 percent

287.55 -> that we would see would be open source

288.99 -> packages that people would include

291.03 -> in their application and then eventually like

293.58 -> within the last couple years this is actually

295.35 -> ballooned up to 98 percent.

297.18 -> So that means that only 2% of the packages that are

301.08 -> out there now are proprietary, right?

304.14 -> Where most of them are,

305.46 -> 98 percent are made up of open source software.

309.33 -> And the other interesting statistic

311.19 -> that we found right is that 81 percent

313.89 -> of the issues that people are finding with

317.19 -> security are with the open source software packages.

320.94 -> So that's why

324.03 -> we need to key in on that the supply chain

326.49 -> and try to fix the problem at the root instead

328.95 -> of trying to fix it at the edge you know with

330.96 -> like firewalls and trying to block it after the fact.

336.27 -> All right I wanna level set

337.62 -> real quick before like how we think

340.02 -> of it so the approach that we want

341.58 -> everybody to start using is DevSecOps, right?

344.94 -> And what is really DevSecOps and I need

346.68 -> to kind of explain what DevOps is,

348.51 -> and so DevOps is not necessarily a team

353.01 -> that lives in your organization,

354.72 -> nor is it a tool set that you can actually use.

357.99 -> So ultimately DevOps is a philosophy

360.3 -> and it came about because what we needed to do is break down

362.73 -> the barriers between the dev and the operations team right?

367.53 -> And to do that it wasn't a tool that helped do that

370.59 -> what helped do that was basically

372.39 -> breaking down those communication barriers

374.28 -> so then teams would talk to each other right?

376.14 -> So you have the operations team that then is working with

379.17 -> the development team while

380.52 -> they're in ideation phase to help them build products

384.15 -> that can actually make it to production run at scale,

387.63 -> and be, you know, run at scale be highly resilient.

392.07 -> And then you have the developers team

393.87 -> that then needed

395.19 -> to know how to actually write their applications

397.71 -> in a way that then the operations team can consume

401.88 -> that then was you know proficient that was highly scalable

405.12 -> and that didn't cause a lot

407.01 -> of problems and a lot of pages

408.361 -> that they would have to deal with at night.

410.94 -> And so DevOps becomes the merging of those two

413.67 -> and is basically those two teams working

415.92 -> together and it's an overarching philosophy that,

420.235 -> of DevOps that,

422.37 -> that as a team what you're trying

424.35 -> to do is you're trying to produce features

426.33 -> and functionalities at a faster pace,

428.7 -> so then that way your organization wins overall, right?

431.97 -> And so you combine these two teams they talk together.

434.88 -> Why I like the term DevSecOps is

436.44 -> because DevSecOps it's kind of it's sandwich in the middle

439.5 -> and it's kind of a mashup right between developer security

443.61 -> and operations and that mashup basically means that we need

446.82 -> to think of security holistically

448.35 -> like security needs to be part of the equation,

451.17 -> not at the end but also at the beginning like

453.81 -> as you're ideating about your product

455.7 -> thinking about it doing threat models with

458.353 -> the development and the operations team,

461.46 -> so that security shouldn't be a silo.

463.77 -> So that's why I kind of like the-

464.88 -> I love the term DevSecOps but remember it's not another team

468.57 -> that you have in your organization

469.92 -> it's a philosophy that all of these different teams should

472.74 -> be working together breaking down

474.06 -> the silos to produce you know better products

478.05 -> and releasing features and functionalities at a faster pace.

483.39 -> Other things that we're seeing

484.29 -> and we all know this it's somewhat common knowledge right?

487.56 -> Is that the cost of repair

491.88 -> is greater the further on down the

495.81 -> software delivery lifecycle that happens, right?

497.97 -> So like if a developer fixes a problem

501.09 -> on their desktop you know if they're getting

503.37 -> the signal then they could fix the problem while it's still

505.26 -> on their desktop it's a lot cheaper than trying to fix

507.72 -> that problem after it's been

508.77 -> in production for like the last year, right?

511.11 -> I mean it's pretty common knowledge

513 -> and so that's also a key driver in why we need DevSecOps

517.467 -> and we need to start thinking about shifting left,

521.01 -> like the different functionalities

522.24 -> that we have in terms of scanning

524.4 -> and security and things like that.

526.2 -> And we also need to shift right as well, right?

529.17 -> Because not all the problems can

530.79 -> be solved on the developers desktop

532.56 -> or you know like in the development lifecycle some

534.81 -> of these problems need to be also solved

536.97 -> in the operations team or as we think

539.31 -> about it you know like shifting right.

542.68 -> 'Cause we don't know what we

543.513 -> don't know at the end of the day.

544.8 -> - [Curtis] Exactly.

547.59 -> - So here I'm going to talk a little bit about

549.18 -> like some of our developer tools here at AWS

551.73 -> and this is only to give you context so as I talk about some

555.42 -> of these developer tools

556.41 -> that we have note that if you're

558.9 -> using different tools within your

561.3 -> environment that's okay like if you're using-

563.22 -> like if you're not using a CodeCommit

564.597 -> and you're using GitHub

566.28 -> or GitLab kind of service that's fine what

569.52 -> we're talking about today is it can be generalized

572.01 -> and it applies to like whatever

573.87 -> tool and lifecycle that you happen to be using.

577.23 -> So here's kind of like our developer portfolio,

580.05 -> we have CICD tools we have ML tools we have IDE tool kits

586.53 -> and a whole array of SDK's,

588.96 -> our SDK's are pretty important to us.

591.57 -> They they allow you to leverage

593.19 -> the AWS services but also using our best practices

596.07 -> so they have things like exponential back off

598.86 -> they also include jitter they also include retries

601.56 -> and things like that

602.393 -> that make them very important in terms

606.18 -> of the ecosystem of the developer tools that we've created.

608.55 -> And you can see from this list here

610.5 -> it's a pretty a pretty broad offering

613.86 -> that we have and it's also a pretty deep offering.

617.79 -> - Yeah the thing that I'd like

618.93 -> to add here is is going back

620.34 -> to that conversation of you don't have

621.72 -> to use these specific tools,

623.49 -> James and I actually work with the third party

626.28 -> integrators the ISP solutions and we help build a lot

629.37 -> of these integrations

630.203 -> so if you're using a third party solution

631.83 -> in place of something like CodeCommit

632.67 -> or CodeBuild we want to make that

635.25 -> as a seamless as a process as possible,

637.26 -> we wanna make sure that you can continue leveraging those

639.24 -> and see success of running

640.23 -> and securing your workloads on top of AWS.

642.15 -> So by no means think

643.23 -> that you have to be bought into this

644.55 -> particular scenario or our specific set of tools,

647.58 -> but just understand that these are built to be able

649.8 -> to communicate well with each other simplify

651.81 -> the integration story

652.77 -> and many of them mean you don't

653.91 -> have to manage certain solutions yourself,

655.77 -> so there's some in intrinsic value of using some

659.31 -> of the AWS services so you don't have to spin-up

662.07 -> an instance of something have AC2 instance

664.14 -> that you have to try and manage maintain secure

665.91 -> in addition to the workloads themselves.

669.27 -> - Absolutely.

671.01 -> Our oops-

672.39 -> CodeCommit sorry, CodeCommit.

674.07 -> So CodeCommit is basically our Git repository

677.01 -> so it's a secure Git based repositories

680.31 -> that live with inside of an AWS account, so developers,

683.25 -> this kind of goes in terms of the

685.08 -> developer self enablement where developers can

687.45 -> spin this up in account

688.89 -> and these are private repos so these

692.52 -> integrate with well with all

694.26 -> the other different AWS services

695.76 -> so you can wrap around like IAM policies

697.95 -> to do different things plus they

699.42 -> also send out like when you do different

701.82 -> transactions with inside CodeCommit

703.86 -> like you know like Git you do check ins

706.23 -> or you do merges and things like that,

707.82 -> all of those are emitted into events into CloudWatch events

711.48 -> or event bridge where you can actually key off

713.79 -> of different rules so you can do

715.89 -> different things based off of CodeCommit.

720.63 -> Oop. Push the wrong button. (chuckles)

724.23 -> CodeBuild.

725.1 -> Now CodeBuild is

726.45 -> our code are generalized like CodeBuilding tool.

731.07 -> Basically you can use your pre-packaged image

736.8 -> or you can use a custom built

738.72 -> image to actually run your build jobs.

740.85 -> So when I say it's a generalized like build tool

743.61 -> 'cause it also does building

744.69 -> and testing so if you look on-

746.31 -> if you can see on the right hand side like

747.93 -> the buildSpec.yaml file it has an example

750.69 -> of like different build stages that you can run

753.24 -> where you can do like...

755.16 -> Like in this particular example you do pre-build, build,

757.32 -> and then it has a post-build section.

760.14 -> So it's basically you just run like arbitrary Linux commands

765.78 -> of what you're trying to do with inside each

767.49 -> of those different sections for a CodeBuild.

772.53 -> And CodeArtifact.

774.528 -> CodeArtifact is our artifact

775.98 -> repository so think of this is like the a pip maven

780.87 -> or a npm kind of service so basically

785.34 -> it lives with inside your AWS account

788.16 -> and when you request these

790.59 -> different dependencies it goes if

792.45 -> the dependency isn't located on CodeArtifact

795.06 -> it will actually go out into you know like npm

798.03 -> or pip or maven and it will actually pull that artifact down

801.39 -> and store it locally with inside your CodeArtifact.

804.12 -> Some of the advantages of being able to do

806.01 -> that is that now you can wrap around policies

808.23 -> and you can wrap around different things

809.85 -> because now you actually have a local copy

812.13 -> of that CodeArtifact, right?

813.48 -> So you can pin things you can actually do scans against

817.44 -> the CodeArtifact to make sure

819.48 -> that your developers are actually

820.95 -> consuming secure dependencies into their

826.44 -> build jobs.

830.131 -> And then we have CodeDeploy

831.03 -> and CodeDeploy is an actual like a

832.74 -> true deployment system so I see a lot

835.05 -> of customers that are using tools like like terraform

838.44 -> or using cloud formation,

840.45 -> while that's good

841.62 -> and it's great for infrastructure it is

843.63 -> not a true application deployment tool.

846.81 -> So you need a true application deployment

849.27 -> tool then that's what CodeDeploy actually

851.16 -> is where it can handle the different states

854.1 -> of your application as it's moving

855.51 -> the cycles but more importantly

857.31 -> it can truly roll back an application

859.38 -> and not leave it in a terrible state.

861.15 -> So like if you're using a terraform as an example

863.73 -> the problem with using a tool like that for deployment

866.52 -> of the application is that it doesn't have a context

869.43 -> around your what your application actually does

871.41 -> so when it does rollbacks it can actually

873.27 -> leave your application as a really poor state

875.64 -> and so CodeDeploy actually does a lot

877.74 -> of that clean up for you when it's configured right?

880.56 -> So it is a true deployment engine,

882.63 -> so it has a context about you know

884.52 -> the status of the of the application

886.672 -> that it's actually deploying.

889.65 -> All right and then finally we have

891.15 -> CodePipeline so CodePipeline kind

893.1 -> of like stitches all this together.

896.07 -> It's the orchestration tool that puts together

898.44 -> like CodeCommit CodeBuild CodeDeploy.

902.25 -> And CodeArtifact in fact.

905.73 -> And it's an orchestrator so basically

907.41 -> it has build stages in here so think

909.78 -> of these stages different,

911.55 -> they're a little bit different

912.54 -> from the CodeBuild that I showed you earlier.

915.12 -> So CodeBuild those were stages with inside

917.22 -> the actual build job this

918.42 -> is where this is one where you can

920.04 -> actually have different stages where you have

921.5 -> to like check out

922.53 -> and then you can actually have like an actual

924.96 -> like build stage in a testing stage in separate,

927.57 -> and actually have multiple

929.43 -> of those CodeBuild environments

931.5 -> going on and even simultaneously, right?

933.6 -> So it'll run in parallel as well.

939.135 -> And then finally this is how things are kind

940.5 -> of like stitch together so like when we look

942.39 -> at like the source build test deploy

944.4 -> and monitor stages for CICD

947.49 -> this is kind of how they kind of fit like

949.26 -> so you have CodeCommit on the far left hand side

951.54 -> then you do CodeBuild

952.86 -> for your build testing you also use CodeBuild

955.053 -> 'cause like I like I mentioned earlier it's more

957.54 -> of your your general purpose like build environments

960.21 -> where you can run arbitrary like Linux commands

962.85 -> and then you have your deployment engine

964.41 -> and then all of that uses CodeArtifact

967.47 -> and then then you can monitor this with

970.537 -> AWS X-Ray which is a tracing type utility for APM

976.86 -> and Amazon CloudWatch.

981.84 -> All right so let's talk about pipeline security so we think

984.51 -> of pipeline security here AWS in 2 different ways,

987.87 -> one is we think of pipeline is security

989.85 -> in the pipeline.

991.17 -> So this is what most people are commonly familiar with

993.69 -> where security in the pipeline is-

996.78 -> security as your application is traveling through

1000.08 -> the pipeline itself like how you're

1001.64 -> securing it are you doing static code analysis

1003.95 -> testing how you're doing different testing

1005.84 -> within the pipeline itself right?

1008.21 -> So going through the source, build, test, deploy,

1010.4 -> and monitor phases.

1012.02 -> And then we have another way

1013.31 -> of thinking of it was just security

1015.35 -> of the pipeline and security of the pipeline

1018.2 -> is where you're thinking of your pipeline as

1020.09 -> an actual application where

1023.87 -> we wanna make sure that you're actually securing

1025.73 -> the pipeline itself making sure

1027.38 -> that the people that can make changes

1029.6 -> to the pipeline make changes to the

1031.1 -> builds make changes to like testing things like

1033.77 -> that are that that you're leveraging

1036.62 -> best practices for security of the pipeline.

1039.26 -> And this is where

1040.13 -> we want you to think about like the pipeline

1042.2 -> as a workload in itself it's

1043.58 -> an application that needs to be patched

1045.53 -> it has its own configuration things like that.

1049.55 -> So let's talk about a little bit

1050.87 -> about security of the pipeline.

1054.56 -> So here I want to share with you today the SLSA framework.

1057.68 -> So the SLSA framework is not necessarily

1061.43 -> open source it is a creative commons framework

1064.97 -> that provides best practice guidance

1066.8 -> on how to secure the actual

1068.51 -> pipeline itself so like the different steps

1072.05 -> that I have outlined here like ABCDEFGH.

1077.21 -> Are different points of the of the pipeline

1081.02 -> that you really need to think about and the

1082.49 -> SLSA framework provides guidance for each one

1084.92 -> of those different stages on best practices

1088.4 -> and guidance on how you should secure that things

1090.437 -> and the different considerations

1091.79 -> that you need to take for each of those different steps.

1096.83 -> So the first thing I wanna

1097.79 -> talk about is like source integrity

1099.23 -> so this goes back to the SLSA framework.

1101.57 -> So source integrity

1104.39 -> is related to those threats that are like around source.

1107.45 -> So basically you know like how do you

1108.83 -> submit changes without review, how do you...

1112.85 -> Preventing evasion of code review process

1116.54 -> and also code review bypasses.

1118.94 -> And also you know like how do you compromise like...

1122.33 -> How people can compromise source repos.

1124.7 -> And some of the mitigations

1125.93 -> and you know and I'm not including all of them here

1128.12 -> on the slide but some of the mitigations

1129.95 -> that you can do is like require like two

1131.9 -> persons to do like a peer review before you can

1137.18 -> merge a change into like a mainline branch.

1140.03 -> Other things is right is like using a role

1142.07 -> to do your merges instead

1143.75 -> of actually depending on an actual person

1145.79 -> because I don't know about you Curtis

1147.2 -> but like I've been guilty of it myself where I

1150.23 -> had like you know like super user access

1152.54 -> and I didn't have anybody to do a code review

1155.81 -> so I went in there

1156.65 -> and I just click you know merge myself very

1159.26 -> poor habit I would I don't recommend doing

1161.03 -> that for anybody but I've been guilty of it myself.

1164.45 -> - It's important to have those checks and balances

1165.98 -> because it even the lead developer like

1167.72 -> we find those scenarios

1168.68 -> where somebody submitted something late on Thursday

1170.9 -> or Friday and so you just

1172.13 -> go in and make that change yourself.

1173.72 -> and the reality is even as a lead

1174.95 -> developer there should be some kind

1176.06 -> of check checker balance put in place

1177.74 -> to prevent you from making those kinds of changes

1179.42 -> that are risky.

1180.29 -> - Yeah absolutely.

1183.47 -> And so here's a solution that I want to kind

1185.24 -> of like talk about a little bit

1186.35 -> of like how you would actually solve like some of-

1188.33 -> like a specific use case for source integrity

1191.03 -> and this one solves the use case for

1193.82 -> having at least 2 people or having

1195.89 -> one person at least do a peer review

1197.75 -> and not allowing a specific person to be able to do

1201.44 -> the merge into a pipeline.

1203.87 -> So in this particular use case.

1206.06 -> What we do is you would set up

1207.59 -> an IAM policy that don't allow it actually

1210.2 -> any individual user to do a merge

1212.93 -> within CodeCommit, right?

1215.12 -> And then CodeCommit when it actually,

1217.13 -> when you have a merge request

1218.51 -> or when you have a pull request basically it would actually

1223.94 -> send an event to Amazon event bridge

1227.99 -> and then once it sends an event to Amazon

1229.91 -> event bridge it kicks off a lambda job that then looks

1233.33 -> at the looks at the repo and sees that somebody is actually

1238.19 -> peer reviewed it and it's not

1239.45 -> the same person that actually submitted the pull request

1243.47 -> and then once somebody does that it actually

1245.48 -> uses a role to make that merge itself

1248.69 -> and then deletes the branch that it's actually merged from.

1254.42 -> And then once that all happens

1255.8 -> basically it sends off an alert using SNS

1258.92 -> and then you can fan out this alert to slack to email

1262.07 -> or to whatever alerting system that

1263.54 -> you have in place to notify

1266.06 -> the developer that you're that you've

1267.68 -> been successfully merged.

1268.79 -> And this solution allows you

1270.83 -> to kind of like decouple

1274.598 -> the different you know like areas

1276.14 -> of responsibility so now you don't have a single

1278.15 -> person actually trying that has to do the merge

1280.55 -> or you know like some-

1282.71 -> that you're applying to a developer

1285.68 -> to allow them to do all the merges.

1290.57 -> And then the second thing

1292.16 -> of of the SLSA framework is build integrity

1295.28 -> so like some of the some of the general threats

1297.23 -> that you see in kind of like build

1298.43 -> integrity is being able to build

1300.65 -> from like an unofficial fork

1303.35 -> or a branch build from you know like

1306.29 -> different build-

1307.19 -> unofficial build stages,

1308.75 -> and also you know like comprised build environments

1314.48 -> of you know like you using different build environments

1317.843 -> that that you're not basically supposed to, right?

1321.02 -> And then also how do you know tamper

1323.87 -> artifact tampering with inside build environment

1326.75 -> so build integrity that that framework it

1329.54 -> provides guidance on how to actually mitigate a lot

1332.39 -> of those different threats

1333.223 -> and some of the mitigations that it

1334.61 -> actually has is basically

1335.69 -> you can target a specific branch for builds

1338.75 -> and what I mean by that

1340.16 -> is that a lot of customers I see what they do

1342.53 -> is they basically target like hey anytime

1344.96 -> anything is like any branch they just wild card it

1348.17 -> and they just build off

1349.07 -> of that branch just while that's great

1351.95 -> in some respects when you're

1353.09 -> you know doing development work you kind

1354.59 -> of want that feedback.

1357.11 -> What you wanna do is you wanna try to limit that

1359.27 -> so you have to-

1360.95 -> A using a naming convention but also that all that you

1366.11 -> wanna make sure that you're not...

1368.36 -> That you're targeting specific branches

1371.24 -> so then people can't arbitrarily like create branches

1374.99 -> and then introduce like different type

1376.79 -> of code that is going outside

1378.17 -> of your like main

1380.96 -> development software delivery lifecycle.

1383.81 -> - Yeah and this goes back to that door analogy kind

1386.69 -> of at the beginning so as attackers

1388.61 -> know that you're hardening the door

1390.17 -> or hardening your production

1391.19 -> environments are hardening certain aspects

1392.69 -> of your application they're finding

1394.7 -> the alternatives they're going further left

1396.35 -> into the stack further left in the software

1398.09 -> development life cycle finding ways to try

1400.07 -> and get in early into those processes

1402.38 -> and if you don't have those checks

1403.49 -> and balances in place to identify

1405.83 -> the opportunities for those those attackers

1408.02 -> to get code in if you're not scanning for

1410.09 -> that you put yourself at risk

1411.56 -> of basically using some of these compromised environments,

1415.1 -> and often times these are...

1416.42 -> Because these are at the developer level

1418.19 -> or the earlier in that process the teams

1420.98 -> behind them are a little bit more lax with them,

1423.02 -> because it's like oh it's it's still in the house,

1425.93 -> but you're not realizing that

1427.1 -> the attacker is trying

1428.39 -> to get into your house that's

1429.92 -> where they're they know you're least secure

1431.96 -> you're not locking your bedroom door you're

1433.52 -> locking your front door so they wanna try

1435.5 -> and get away in your house so they can get

1437.21 -> in and make those changes as early on as possible

1439.7 -> because if you don't have that in,

1441.32 -> it's easier for them to get into the code base

1443.78 -> and start affecting things later on in the process.

1448.7 -> And so here's a solution that

1449.84 -> we put together for the use case

1451.61 -> of build integrity so making sure

1454.19 -> that that you're building off

1455.96 -> of dependencies that that have been approved

1459.59 -> by your security your security departments

1461.87 -> or your scanning tools and things like that.

1465.26 -> So here in this particular solution we

1467.21 -> start with CodePipeline so CodePipeline is

1469.22 -> the orchestrator that I talked about

1470.57 -> earlier it uses CodeCommit which

1474.34 -> is a private repository that your organization has.

1478.67 -> Uses CodeBuild and CodeBuild reaches

1480.86 -> in and uses CodeArtifact.

1483.38 -> So CodeArtifact has a very interesting feature

1485.72 -> in it called the...

1488.96 -> What is it called?

1491.66 -> It's called package origin controls so within

1495.17 -> package origin controls that allows you to target

1498.35 -> and specify like hey this is a specific

1502.25 -> source is actually my my master source or my...

1507.95 -> The provenance of where this package should

1509.66 -> actually be that has been tested and what...

1513.86 -> This package origin controls is very important

1516.26 -> because what we're seeing is

1517.49 -> we're seeing attackers actually...

1520.64 -> When they can figure out what naming convention

1525.05 -> that you're using for internal packages what they'll do

1527.99 -> is they'll come up with something

1529.228 -> like let's say you're using my package

1532.25 -> version one what they'll do is they'll

1534.14 -> come out with like a my package version 1.1,

1538.25 -> and then they'll publish

1539.21 -> that 1.1 into like a public repository.

1542.03 -> and so then what ends up happening is

1543.8 -> your package manager is actually pretty smart

1546.11 -> and it knows that it's like hey I'm going out to like npm

1550.01 -> or I'm going out to pip or I'm going out to one

1551.78 -> of these other package managers.

1553.04 -> And it has an updated version. It has 1.1.

1556.4 -> And I see that our local environment we have 1.0

1559.73 -> so let me get that updated version pull

1561.68 -> it down and now you're going to build off

1563.42 -> of a a compromised source for that dependency.

1568.88 -> So what package origin

1570.35 -> controls allows you to do is that it allows you to

1573.98 -> dictate which actually one is

1575.9 -> the master so basically you're going to say

1578.27 -> hey my internal one is actually the master source

1581.42 -> or it's actually you know

1582.29 -> the origin for that particular

1583.46 -> package so even if you recognize that

1585.92 -> an update has been done for this particular package name,

1588.327 -> "No I'm still the major source", right?

1590.99 -> So it helps with that particular type

1593.84 -> of supply chain attack.

1596.96 -> And so in this solution it uses

1598.49 -> the AWS CodeArtifact once it uses

1601.58 -> AWS CodeArtifact it builds that we deploy

1605.15 -> ephemeral environments

1606.38 -> and for those who you don't know what

1607.91 -> ephemeral environment is it means it's just temporary

1610.16 -> and means that you

1611.6 -> shouldn't go out into that environment

1613.13 -> and make changes locally what you

1614.45 -> should always be making these changes with

1616.22 -> is using like infrastructure's code

1617.96 -> or using your code to actually change these environments.

1620.3 -> So you change these environments in Git.

1623.12 -> So one of the benefits that

1624.14 -> you get by treating these environments

1625.73 -> as ephemeral is that now that anything that happens

1628.82 -> within that environment you know is is a compromise

1632.12 -> of security like so if let's say if you don't allow like

1634.58 -> SSH right so like if somebody logs in SSH.

1637.61 -> Now you know that that is actually a violation

1639.89 -> of security policy so you can kind of quarantine

1642.53 -> that particular container that instance

1644.84 -> that's running in that environment.

1646.46 -> And so by treating like it isn't

1647.8 -> in ephemeral environment you kind

1649.34 -> of gain some of these positive side effects if you will.

1654.29 -> And so in part of the solution too we

1656.18 -> have like AWS config so it'll notice

1658.37 -> that anything that has changed within one

1660.23 -> of those environments it'll actually spit out

1663.2 -> an event AWS code config

1667.302 -> with a config rule will detect

1669.56 -> that event kick off a lambda or kick off

1672.41 -> an alarm and then you could send that alarm

1674.66 -> to SNS to notify you know like a slack message

1677.87 -> or you can send it to other services like

1681.56 -> security hub or even you know email and things like that.

1688.97 -> All right so availability and other threats.

1691.19 -> So other threats that we see like in terms

1693.71 -> of availability and I kind of mentioned this earlier

1695.93 -> or actually I didn't mention this earlier.

1699.44 -> One of the availability threats that

1700.79 -> I've actually even had personally is that

1702.71 -> so when you're going to build a job in...

1704.93 -> Or you're doing a build

1706.01 -> and you have different dependencies

1707.18 -> that live with inside your code.

1710.3 -> If that dependency is unavailable from the external source

1716.18 -> that you're getting it from the your build job

1718.16 -> is basically stopped you can't move for for further right?

1721.91 -> And you don't know when this availability is actually

1724.13 -> gonna back up right like I've actually had...

1726.56 -> Personally I've had a public repo or specific

1729.74 -> package on a public repo go down for three days

1732.68 -> and so we can do anything for three days

1734.72 -> 'cause we're having to wait for that

1736.34 -> and so what it made me think about is

1738.05 -> that hey we need to actually store and have our

1740.69 -> own package resource that we can actually pull from.

1743.99 -> So then we can mitigate that particular you know

1746.54 -> that downside that if this public package

1748.67 -> isn't available then now we can pull it directly

1751.01 -> from a local source.

1754.01 -> Other mitigate-

1754.843 -> So other ways that we can mitigate

1756.38 -> too is basically you know retain version controls

1759.62 -> and the history of it

1762.05 -> and I'll talk about that in a second with like

1765.74 -> some of the solutions that we have in place.

1768.29 -> - [Curtis] Yeah this is really important when you're looking

1770.48 -> at having an artifact repository something like a Jfrog

1773.69 -> or CodeArtifact or others instead

1775.97 -> of pulling things live from

1777.2 -> the Internet live from Docker Hub or wherever like

1781.67 -> the idea is you're always pulling from a

1783.2 -> local source that you know is available at all times.

1786.59 -> I've had this experience

1787.64 -> as well with with my previous company where

1790.58 -> there was a random number generator for node.js

1793.31 -> and that that one library was taken

1795.47 -> offline and it broke so many other things

1797.63 -> because it wasn't a direct dependency

1799.1 -> of our application but as a dependency.

1801.5 -> Downstream downstream in the supply chain where one

1804.65 -> of the libraries we needed

1805.85 -> needed it for one of its libraries

1807.5 -> and eventually that actually broke us

1809.66 -> and so it emphasizes that importance

1812.54 -> of having something local that you're

1813.95 -> constantly pulling from and then

1815.45 -> from the security side of it now that's

1817.01 -> something you can actually scan

1818.33 -> and we'll talk about that a little bit more in a moment.

1820.76 -> - [James] Yeah.

1822.44 -> And so here's a solution that we built for like source

1825.29 -> and artifact availability

1827.3 -> so what we want you to think about is like not

1830.03 -> only like having this within AWS so we do have a lot

1834.14 -> of redundancy like when you're using a lot of

1837.2 -> these different services but also think about copying

1839.45 -> the services off to a to like an adjacent AWS account,

1843.41 -> and when I say copying this off,

1845.54 -> think about it as like read only

1846.98 -> because I've actually had,

1848.171 -> in my personal experience,

1849.92 -> I've had where somebody was doing some kind

1851.96 -> of like pruning job they went through they pruned a bunch

1854.81 -> of like a bunch of data like 500 gigabytes

1857.81 -> of data from from a repository we had

1860.75 -> and our replication worked

1862.01 -> really well and it replicated to all

1863.72 -> the other repositories that we had

1865.58 -> and so we lost that data where we had

1867.71 -> to restore it from backup but then restoring

1870.23 -> from backups it took like 6 months to get

1872.33 -> because we had to go through a bunch

1873.41 -> of different tapes and things like that.

1875.42 -> So think about this, think about like when

1877.46 -> you're actually copying this data

1879.05 -> to a different account like for

1880.19 -> CodeCommit your artifact repository

1882.53 -> and also anything that you're storing

1883.94 -> in like containers think about like append only.

1886.94 -> So we so then that way you don't have any problems

1889.61 -> like that come along as somebody is a pruning job

1892.55 -> and then it also replicates to these other accounts

1894.77 -> because if you're doing append

1895.7 -> only then you're basically just storing

1897.56 -> the changes so you have a history

1900.05 -> of the transactions that's happening.

1901.76 -> - [Curtis] Yeah and even even worse if from a security

1903.83 -> perspective say somebody does happen

1905.3 -> to get into one of these you don't want

1906.71 -> that then replicating

1907.73 -> to other sources you always want to make

1909.17 -> sure that there's some kind of separation

1911.24 -> so that even if something is compromised

1913.07 -> you can actually pull from a clean source

1914.96 -> to replace it once you've identified that so this is

1917.21 -> an important process to help you on that way.

1919.31 -> - [James] Yep, yeah because the reality is

1921.65 -> that bad actors know about logs right they know how

1923.72 -> to go clean up logs they know how

1925.13 -> to go go clean up all this different stuff

1927.11 -> and so that's why you want to have a kind

1928.67 -> of like an append only where you have this transaction

1931.01 -> of things where people can't tamper with right

1933.377 -> and it lives inside of a different account that

1937.07 -> you know very limited people have access to.

1942.71 -> All right I want to talk a little bit

1943.82 -> about like verifying dependencies

1945.53 -> so Curtis is going to talk about later about

1947.81 -> SBOMs so I won't I won't steal his thunder on that,

1950.69 -> but what you really need

1951.68 -> to do is when you're checking your dependencies

1953.39 -> you need to require from all your dependencies is

1956.78 -> an SBOM that you can actually check against

1958.85 -> and what this is is you're verifying the

1960.65 -> provenance and you're verifying the signature of this.

1965.3 -> Curtis will talk about what's actually included

1967.176 -> in a SBOM what you need

1969.689 -> to validate but I thought it was

1971.6 -> an important topic to talk about now where you

1974.69 -> basically just need to make sure that you have this

1976.58 -> in kind of like your build

1977.537 -> and your testing steps where you

1978.89 -> are validating these hashes.

1980.54 -> And I know that like just from even from my

1983.69 -> past like I've I've always been kind

1985.31 -> of lazy sometimes a lot of packages actually come with

1988.1 -> hashes and you know, "I don't need to check it",

1990.74 -> you actually should check

1991.7 -> that right you should actually verify that

1993.74 -> the hash that you've actually downloaded

1995.54 -> matches what the what the vendor

1997.97 -> or what the open source project actually has for that hash.

2001.06 -> - [Curtis] And often times these are very

2002.26 -> simple processes to automate

2003.76 -> and so if you're not automating that...

2005.38 -> This is something you see a lot with OPA

2007.51 -> or open policy agent

2008.65 -> in containers like where you actually look at the hash

2011.32 -> of the actual container from the image repository

2014.5 -> so that as you're trying

2015.55 -> to schedule that particular container

2017.38 -> onto your cluster you verify that hash

2020.14 -> so you know for a fact that what you're actually deploying

2023.26 -> is what you actually expect.

2025.09 -> And so verifying this dependency

2026.89 -> and like you said I'll go through SBOM in a little bit

2028.72 -> but the idea is we wanna make sure

2030.31 -> that you're actually taking the steps to validate that what

2033.07 -> you're deploying is what you expect

2035.59 -> and you're not just kind

2036.423 -> of trusting the source and trusting it

2040.48 -> because somebody might be nice looking but they might try

2042.82 -> and steal your wallet you don't want to base it off

2044.71 -> of just trust alone.

2048.73 -> - [James] All right in the guardrails so guardrails

2050.71 -> is something that we use here to Amazon

2053.26 -> and so what it does is it reduces the occurrence

2056.08 -> of events and it also reduces

2057.79 -> the blast radius and guardrails are more...

2060.82 -> And I'll give you a practical example

2062.38 -> so like I manage a production account here

2066.19 -> at Amazon and one of the things

2068.08 -> that we have is at guardrails are S3 buckets.

2071.68 -> So we're not allowed to create S3 buckets

2073.57 -> or I shouldn't say we're not allowed to create S3 buckets.

2076.06 -> If you create an S3 bucket a

2077.71 -> public S3 bucket you will get flagged

2080.41 -> because you know a public S3

2082.33 -> bucket is is in some ways bad.

2084.61 -> But as a developer if I still wanna

2086.41 -> create a public S3 bucket I can I can do

2089.44 -> that intentionally I can go

2090.55 -> in there there'll be several warnings for me that says,

2093.227 -> "Hey do you know that you are creating a public S3 bucket?"

2096.4 -> I can say "yes I know I want to create a public S3 bucket",

2099.49 -> and then later I will get a ticket flagged against me

2102.22 -> that says "hey you created a public S3 bucket.

2105.1 -> Is this what you really wanted to do?"

2107.62 -> And so I have to intentionally

2109.66 -> choose to create that public S3 bucket

2111.64 -> and that's kind of how we want to think of guardrails right?

2114.04 -> It's not something that is

2115.39 -> encumbering developers but it's something

2117.46 -> that makes them think about what they're doing

2119.53 -> and make sure that they're very intentional

2122.2 -> about what they're trying to do like with

2123.85 -> the public S3 bucket

2125.23 -> and so that's why I like to think of guardrails

2127.33 -> - [Curtis] Yeah it's that intentionality that that's

2129.16 -> the important thing it's...

2131.65 -> We don't want security to be seen

2133.15 -> as a draconian process it's kind of forced on everybody.

2136.06 -> But we do wanna make sure

2137.17 -> that it's a conscious process,

2139.09 -> so in the event like the example there

2141.7 -> like you want to make sure that, "Hey."

2144.557 -> "You sure this is what you're doing what you really want?

2147.28 -> 'cause this is the risk that you're actually creating."

2150.13 -> and it's fine but we need to track those and keep track

2152.68 -> of them so when he creates a bucket I'll

2154.81 -> get a flag like I've been on stage before

2157.03 -> and had my phone start ringing there's

2158.62 -> an alarm that basically like fires off

2161.02 -> so if somebody on my team creates

2162.7 -> that I know that they've done that.

2165.28 -> But it's important to keep track of those types

2167.08 -> of things and automate them wherever possible.

2170.23 -> - [James] So here's an example of guardrail so this

2172.27 -> uses OPA so OPA stands for the open policy agent

2176.59 -> and OPA is kind of like a...

2178.693 -> Is a technology and a tool that allows you to define

2183.1 -> policy and it separates the policy versus the implementation

2186.28 -> or the the control and the enforcement

2188.86 -> of it this is an example for like a EKS a

2192.58 -> constraint.yaml that you can deploy.

2194.62 -> And basically what it says is that hey if as a developer

2198.94 -> or somebody who's trying

2199.84 -> to deploy something I can deploy-

2201.61 -> if I'm trying to deploy from my private repo a container

2205.09 -> from my private repo it allows me

2206.77 -> to but if I try to try to deploy

2209.14 -> a container from my public repo,

2210.91 -> it denies it or it stops me.

2214.3 -> As a developer I know that hey I don't get...

2218.045 -> When I use this kind

2219.46 -> of policy it doesn't actually prevent me

2221.47 -> from deploying my container it just tells me

2224.44 -> that hey you know if I really want to deploy

2226.45 -> this container I need to move it into my-

2228.85 -> or my private

2231.04 -> container repo and then from there I can actually

2233.56 -> wrap policies around it's a scan it to make sure

2236.02 -> that that repo doesn't...

2238.6 -> Or that container doesn't have

2240.283 -> like you know like malicious intent.

2245.83 -> And so some final security considerations for security

2249.16 -> of the pipeline so like I said before I want

2252.16 -> you to think about like security of the pipeline

2254.689 -> as the pipeline itself as

2256.36 -> an application so you need to think

2257.98 -> about like how you're storing secrets.

2260.44 -> How you're storing the configuration making sure

2262.45 -> that people can't have access

2264.34 -> to it to change change that configuration

2266.59 -> because bad actors if they can attack

2269.38 -> your actual pipeline itself they can do a lot of damage.

2271.93 -> Because now they control how your

2273.73 -> builds are done they can insert

2274.99 -> different things within your

2275.98 -> build process they can actually change

2278.2 -> the dependencies so you really need to think

2279.88 -> of your application, the pipeline itself, as an application.

2285.55 -> All right I will get over to Chris.

2287.74 -> - All right now that we've talked about

2289.99 -> how do we secure the pipeline itself

2291.49 -> and secure it as an entity let's talk about how we actually

2293.86 -> move things through it,

2294.73 -> and how do we secure things within it.

2297.34 -> At a high level there's a lot of different...

2300.549 -> There's multiple stages that exist within your pipeline

2302.62 -> and so they, kind of, you've seen them on screen

2304.57 -> previously but just to get a high level here's

2306.55 -> the code stage there's the build stage

2308.5 -> there's the test and deploy in the monitor

2310.39 -> or observability stages.

2312.22 -> Within each of these stages there is

2314.41 -> an opportunity for you to perform

2316.63 -> a specific set of security tasks

2319.69 -> that relate to that particular stage.

2321.82 -> It's like we said earlier it's

2323.26 -> like you don't just put security

2325.18 -> at the end when you're running something

2326.74 -> in production every step along

2328.78 -> the way for how you get from code

2330.85 -> on the developers machine out to that production stage.

2334.78 -> Each of those has an opportunity there

2336.61 -> and when we started from the and I'm going

2337.95 -> to explain each of these in depth in a moment,

2340.15 -> but when we started the code level this is where we can do

2342.85 -> things against the source code

2344.14 -> so as developers are actually writing

2345.97 -> the code we have opportunities

2347.29 -> to run scans against those codes or that code

2349.96 -> and make sure that they're not writing vulnerabilities

2352.6 -> into the code and then storing that in the

2354.34 -> source code repository now it's part

2355.9 -> of your version history.

2358.06 -> I'm moving on to the next one instantly. (chuckles)

2360.7 -> But when we move into the build phase that

2362.11 -> something that now that we're building that that code

2365.38 -> and we're actually running that code it lets us

2367.27 -> start looking at like what are the components of it?

2369.4 -> So as we're building

2370.233 -> the artifacts are we including libraries,

2372.64 -> are we including downstream things,

2374.23 -> are we including things

2375.5 -> from public open source repositories?

2377.89 -> Now we have to start considering like

2379.57 -> how does our security footprint look

2383.95 -> with the fact that we're

2384.783 -> actually pulling these disparate sources

2386.17 -> and we're pulling them together into our code base

2388.72 -> and so we have to accept how-

2390.16 -> like understand how we're looking

2391.54 -> at the risk for that and what

2392.53 -> are we doing to mitigate the risk.

2394.06 -> When we actually take those built artifacts

2396.58 -> and start to execute them

2397.54 -> in the test phase now we have running applications

2400.54 -> so this gives us the opportunity

2401.86 -> to now start running tests against those

2404.53 -> running applications so this

2406.09 -> is where things like pen testing load testing things like

2408.67 -> that come into play where we're past the code phase

2411.79 -> of the pipeline and now we're into the actual running state.

2415.03 -> So do we have ports open? Do we have SQL injection risks?

2420.01 -> this is the opportunity

2420.94 -> for you to start looking at that

2422.197 -> and then as we start to go through

2423.61 -> deployments as we mentioned earlier this

2425.89 -> is your opportunity to say we've built an artifact

2428.23 -> and we built it we did all these security checks

2431.02 -> and we tested those are validated

2433.66 -> and it's safe we created a hash of that artifact.

2436.87 -> So we know what that artifact

2439 -> is in its signature essentially

2440.41 -> so then we go to actually deploy that our

2443.05 -> deployment system can actually take a look at that

2445.06 -> and say play we're looking for this hash here's

2448.21 -> the artifact we downloaded let's run a

2450.25 -> check on that does the hash match, if it doesn't,

2453.13 -> you stop the pipeline,

2454.6 -> alert security, alert your operations team,

2456.76 -> because something happened in there

2457.93 -> so these are the opportunities that you have there

2459.76 -> and as you go and you actually run this

2461.26 -> in production it's not like that's the end.

2464.44 -> Yes you've written your code you built your artifact

2467.02 -> and you've actually pushed it

2468.04 -> out but it doesn't stop there.

2470.02 -> Like now that it's running in production this is now

2472.69 -> your opportunity where people are still going to try

2474.91 -> and find their way in

2475.743 -> so you need to continuously monitoring,

2477.19 -> so this is where runtime scanning things like

2479.383 -> that can actually come into place.

2481.72 -> So let's look at these and these

2483.37 -> opportunities in a little bit more depth.

2486.19 -> So early in that process when your

2487.87 -> developers are actually writing their code this is

2490.6 -> an opportunity for you to start doing things

2492.76 -> like scanning for sensitive information.

2495.34 -> How many of you are developers in this room?

2497.74 -> Okay, a few of you good, good chunk of you actually.

2500.86 -> How many of you used a local username

2502.93 -> and password to a local database before?

2506.11 -> Quite a few of you yeah usually you're gonna

2508.3 -> use some kind of local environment environmental variable

2510.58 -> and you're going to use that to

2511.51 -> access it so that that username

2513.55 -> and password isn't necessarily in your code you're

2515.62 -> gonna be referencing it essentially dynamically

2518.11 -> and then later on in the pipelines maybe you're

2520.15 -> referencing it from some kind of security store

2522.46 -> or something like that.

2524.29 -> But every now and then it's Friday night you're

2526.84 -> on a Mountain Dew craze development kick

2529.6 -> and you're on a roll

2531.22 -> so you just kind of added in in the code itself

2534.82 -> well this is your opportunity

2536.2 -> that when you go to save that code your ID can have a plug

2539.77 -> in that can actually scan that immediately

2542.38 -> and alert you "hey you slipped up there's

2546.28 -> an actual secret in here get it out."

2548.95 -> The same as in the same can be done it with the sense

2552.19 -> of like Git Hooks.

2553.33 -> So at AWS we have a tool called code defender

2555.91 -> that we use I don't know if it's

2557.29 -> something that we built internally but

2559.39 -> there's specter cloud offers one too

2562.12 -> that basically plugged into get locally

2564.37 -> so when you actually go to create

2566.08 -> a Git commit it will trigger a scan

2569.23 -> and let you know immediately whether

2570.73 -> there's something sensitive in there it'll tell you

2572.38 -> the line that the code is

2574.03 -> at and you can actually fix that before it

2576.19 -> gets into your history in your Git repository.

2579.13 -> And then is something that has

2580.6 -> to go through a cleansing process

2581.83 -> and all kinds of other stuff,

2582.73 -> and so the fun thing about that

2584.35 -> is it starts to notify people

2586.36 -> so how many repositories do you have how many risks have-

2591.13 -> are you trending up, are you trending down,

2592.75 -> based on these risks and these are things that

2594.91 -> don't have to be done by

2596.35 -> the developer that just kinda

2597.55 -> there to help support the developer

2598.96 -> so it's not additional steps

2600.31 -> and chunks for them but it's something that

2601.99 -> can actually be be very helpful and then you can do

2604.63 -> that on the repositories themselves,

2606.79 -> so at regular intervals

2608.8 -> you can be scanning those repositories

2610.18 -> and making sure that something

2611.11 -> maybe didn't slip through in a pull request

2613.6 -> or something maybe a review process didn't catch it

2616.03 -> or the tools have been updated with

2618.58 -> new data around the vulnerabilities

2620.95 -> and now you need to re scan and fix it if you find it.

2625.33 -> When we look at something like software

2626.8 -> composition analysis this is moving down that

2629.41 -> the stack so after we've written the code this

2632.29 -> is the dependencies so this is the opportunity for you

2635.11 -> to go through and actually start scanning the dependencies

2637.78 -> of your applications.

2639.4 -> Do we have any Node developers in here?

2642.13 -> Yeah how many of you have

2643.18 -> written like a 4 line node application

2645.25 -> and then ran npm build in a 4000 libraries later

2650.08 -> you have this massive application this is

2652.87 -> an opportunity for you to scan those 4000,

2654.937 -> and I joke it's more like 400 but you can

2657.88 -> scan those applications in those dependencies

2660.22 -> so that you know you're not including

2661.54 -> something that has a vulnerability downstream

2663.4 -> and things like OWASP dependency tracker will let

2666.1 -> you scan those open source solutions and keep track of them.

2668.86 -> Or better yet, pull them down like we said into a local

2672.34 -> repository that you have automatic scanning

2675.28 -> and you've basically validated and embedded it.

2678.76 -> One point that I want to get across

2680.26 -> throughout this entire process is it's like eating

2682.12 -> an elephant like you're not going to eat it all

2684.01 -> at one chunk but we want to find that

2685.57 -> there's these opportunities that as you grow

2687.61 -> and mature in these processes

2689.53 -> you can introduce new things one step

2691.78 -> at a time and grow these over a period

2693.61 -> of time and as you go through this these tools that are

2696.55 -> on the screen here through Snyk

2697.9 -> and Mend and Anchore can help you ease that

2701.77 -> adoption and ease that maturity as you as you grow.

2707.53 -> Moving one layer down from that we've got

2709.772 -> static application security testing or SAST.

2713.65 -> And secret scanning is an example

2715.96 -> of this but the idea is this happens early

2719.41 -> on in the development process this is what

2721.54 -> we would consider things like white box testing

2724.24 -> so you have access to

2725.47 -> the code base you know it's dependencies

2728.41 -> you're actually scanning against

2730.12 -> that code base for known vulnerabilities

2732.55 -> known issues so that you can catch them

2734.86 -> before you go into the build processes before you

2736.9 -> go in the pipelines the reason this

2739.06 -> is important is imagine you're running a

2740.98 -> sprint your sprints are time bound you don't

2743.83 -> wanna get to the end of that sprint where you're trying

2746.05 -> to do a release to then find out you

2748.18 -> have more development work to do.

2750.37 -> But these tools let you at least identify those things

2753.25 -> and act against them so even if it is later in the

2756.52 -> development cycle for that that process you can

2758.68 -> at least know that hey do we need

2760.51 -> to do a quick hot fix do we need

2762.25 -> to write a patch really quickly get it in there?

2764.32 -> Or can we assess the risk of that particular vulnerability,

2768.76 -> and make a decision was whether

2769.9 -> or not we want to come back next sprint

2772.03 -> and address it is it a low risk is it something

2774.01 -> that we don't think is going to actually be

2775.69 -> an issue can we can make adjustments against it.

2778.33 -> But the idea here is you have your code base,

2781.12 -> you should be going through at regular intervals

2783.64 -> and scanning that code base for these vulnerabilities

2786.55 -> because new vulnerabilities

2787.57 -> appear every day and our knowledge

2789.07 -> of those vulnerabilities adjust and adapts every day.

2791.83 -> We've got solutions like Snyk,

2793.42 -> SonarQube is probably one that most

2794.92 -> of you are familiar with but also we've

2796.78 -> got Amazon CodeGuru which has a lot

2799.54 -> of value across multiple stages of your build

2801.85 -> but it also lets you know it will scan your application

2805 -> at regular intervals

2806.05 -> or pull requests for known vulnerabilities as well.

2811.39 -> Moving further from that and down the pipeline we have

2814.69 -> dynamic application security testing.

2816.94 -> So this happens after you've built your application,

2820.72 -> so this is something that's done that's considered black

2823.48 -> box testing so you don't necessarily have access

2826.39 -> to the code base itself but you do

2828.43 -> have access to the running application

2830.41 -> and so think of dynamic application testing

2833.38 -> is something that the actual attackers in the real world

2836.26 -> are gonna be using,

2837.25 -> they're going to be trying to find

2839.05 -> the ports that you left open

2840.52 -> the SQL attacks the the gaps in the

2842.74 -> running application that they can kind

2844.12 -> of weasel their their way into so this is

2846.49 -> them trying to get through your window

2847.96 -> or the back door essentially of the

2849.52 -> running application or your built house

2851.95 -> and the idea here is that there's there's

2853.66 -> tools like OWASP zap that will go through

2856.03 -> and actually scan for these things.

2858.13 -> The challenge with this is it's very much focused

2861.64 -> on the stuff that's already made it in your code.

2864.7 -> So the gaps are already there it's just trying

2866.95 -> to discover them like you've already opened that port

2870.46 -> so it's not that a new port is

2872.11 -> opened up it's just that it's there

2873.58 -> and you may not have known about it

2874.577 -> and so it these tools are really defined.

2877.6 -> Once you've got the running code,

2879.13 -> what have you left left open essentially?

2882.13 -> And these are very helpful

2883.48 -> because most of the time we're staring at the code

2886.69 -> or we're staring at observability

2887.95 -> or monitoring solutions later on the story.

2889.93 -> This is in that process of before

2891.79 -> it gets to production we can try

2893.53 -> and find any any additional vulnerabilities

2895.72 -> that that might have made their way in before

2897.76 -> they can be exploited, but once it's in production,

2902.29 -> there's the runtime application self protection.

2906.22 -> And this is different from dynamic applications testing

2909.94 -> in the sense that this is the running application

2912.97 -> and an active threat.

2915.61 -> So this is an attacker trying

2917.38 -> to do something in real time trying to add

2920.05 -> a sidecar to your container that's running

2922.57 -> or your deployment that's running,

2923.92 -> trying to run a binary application on one of your systems

2928.51 -> and there's these are typically

2930.01 -> implemented through some kind

2931 -> of agent that is either monitoring

2932.5 -> the container monitoring the server monitoring

2934.81 -> the workload that's actually running in real time

2937.12 -> and it's taking all

2938.92 -> of the data that's coming in and they can either

2941.2 -> do this through machine learning and trying

2942.73 -> to figure out like what is the normal baseline

2944.53 -> or it can be monitoring the network traffic

2946.39 -> and everything else that's going on in real time

2948.31 -> and it's making a decision

2949.57 -> that what looks normal what looks

2951.52 -> good versus what is a potential threat.

2954.46 -> And the benefit of doing this and using something like Falco

2957.347 -> or Sysdig or even Aqua is that

2960.13 -> they can take action in real time.

2962.32 -> So if they start to see traffic coming in from a location

2965.71 -> that it shouldn't be coming in from it

2967.57 -> can actually in real time block that traffic

2970.69 -> and take an action to protect the application.

2973.54 -> And so you now have multiple shades

2976.78 -> of security from the code itself early

2979.48 -> on in the development process to once you're compiling

2982.9 -> that code into built artifacts to the

2985.48 -> built artifact in the running application

2987.58 -> and beyond and so we treat this is an important part

2991.48 -> of the shades of the lifecycle

2993.16 -> of your application and when we look

2995.44 -> at the software development lifecycle here.

2998.23 -> It's not a start and finish,

3001.05 -> and I'm not going to call out all the things

3002.64 -> 'cause this is an I-chart like in all reality

3005.55 -> but the idea here is that security

3007.71 -> is a horizontal, observability is a horizontal,

3011.25 -> it doesn't just happen once you deploy

3012.87 -> the application it happens consistently

3015.57 -> and continuously throughout the entire development cycle.

3018.57 -> So it's going to start when

3019.5 -> you're planning your applications

3020.76 -> and you're gonna, going back the DevSecOps conversation,

3023.64 -> you're going to talk to the networking team

3025.11 -> and explain why you need specific ports you're

3027.27 -> gonna get agreement to that put that in your code

3029.31 -> or build it in your pipeline,

3030.78 -> or the infrastructure's code,

3032.13 -> then you're going to talk about

3033.03 -> the integrations that you're gonna make

3034.26 -> between different services

3035.28 -> and make sure you have the IAM policies in place

3037.44 -> to allow access and controls or the secrets

3039.99 -> that you need stored in the proper process,

3042.6 -> and you're gonna continue this

3043.8 -> throughout the entire development lifecycle

3045.51 -> so from build to test.

3048 -> To deploy operate and you're gonna put

3050.67 -> things in place that can help at every phase

3052.68 -> of this whether it be the static code analysis

3055.56 -> or the black box testing or the runtime scanning,

3058.14 -> the idea is you wanna make sure

3059.55 -> that you're covering all of these

3061.35 -> and what this leads to is just making sure

3063.84 -> that you're implementing proper tests and checks throughout

3066.51 -> the process and so I'll use some of these examples here.

3070.29 -> When we look at AWS CodeCommit or CodeBuild

3073.17 -> as actions are performed in that part

3075.54 -> of the lifecycle you have the

3077.13 -> options to trigger off of those events.

3079.32 -> So whether it's a commit they can now trigger a scan

3083.01 -> or it's a build job that's been fired off

3085.11 -> and it's triggering this.

3086.73 -> The idea is that as you build your

3089.01 -> application start to store the artifacts

3091.62 -> and store those by areas you're taking

3093.72 -> the right action triggering through automation

3096.63 -> rather than a manual person having

3098.58 -> to go in and like validate things by hand you're

3101.22 -> automating this process throughout

3102.6 -> the entire thing and it doesn't

3104.22 -> matter whether using these solutions

3106.65 -> or you're using complementary

3107.94 -> solutions or alternative solution

3110.13 -> so GitHub will work with this

3112.56 -> Jenkins will work with these processes

3114.51 -> the importance is that you're actually

3115.98 -> doing this at every stage of it

3117.63 -> and you're taking the opportunity in context

3119.7 -> of the component and doing the right

3122.79 -> scanning and doing the right alerting.

3125.46 -> Now one of the things

3126.78 -> that we talked about earlier that James

3128.31 -> mentioned was the software bill of materials,

3130.62 -> and this is a little bit

3131.82 -> of an abstract concept for some people it's somewhat new,

3134.85 -> but the idea here is as you're building these applications

3139.02 -> and you have these downstream dependencies.

3142.02 -> If a vulnerability comes up say there's

3144.18 -> a new vulnerability with the Java version,

3146.91 -> or a specific binary library,

3149.01 -> and you have 100 applications

3150.57 -> that are all java applications.

3152.91 -> How do you know which applications are vulnerable?

3156.63 -> You gotta scan, you gotta go through

3158.67 -> and scan every single one of those applications.

3161.61 -> Or you can use something like a software bill

3164.37 -> of materials where you actually define,

3167.25 -> and there's tools that are open source

3168.87 -> there's things that you can use that will actually

3170.88 -> scan your application and all

3173.43 -> of its dependencies and create using

3175.95 -> a kind of a specification a document

3178.92 -> that basically defines all of the components

3181.83 -> through that entire software supply chain that are a part

3184.98 -> of your application.

3186.57 -> Seems a little excessive but it's incredibly useful

3190.38 -> and you can automate this process

3191.7 -> so it's not a manual person going through and like trying

3193.89 -> to find all these dependencies

3195.36 -> they can automatically do this

3196.56 -> on your behalf but it lets you

3198.27 -> know who you're pulling these from.

3200.58 -> What their component names are what versions are there.

3204.6 -> You know what data and dependency

3206.79 -> relationships are there so if you look at like one

3208.89 -> of the file formats it reads a lot like XML

3211.47 -> and it's nested so you can see

3213.39 -> the dependencies you you can visualize this

3215.7 -> you can actually scan against it so the event

3217.77 -> that there's an actual attacker an actual risk,

3221.34 -> you can know very quickly without

3223.38 -> having to go through all of your applications

3225.06 -> and scan against them which applications may be vulnerable,

3228.93 -> and so if it's a specific Java

3230.31 -> version you know that these 27 out

3232.26 -> of these 400 applications have that particular version

3235.5 -> or use that particular library and if you've done all

3238.47 -> of this properly now you know exactly which which binaries

3244.2 -> which downstream dependencies you should be preloading

3246.93 -> in the artifact repositories

3248.94 -> because these are the things that are actually needed

3251.04 -> by your applications,

3252.3 -> and if you're building a new application these are

3255.21 -> the opportunities for you to take those

3257.07 -> and load them in do the proper scanning and make sure

3258.783 -> that they're there it isn't a very

3261.09 -> powerful process but it's a process

3263.937 -> and so this is something that

3265.2 -> again goes back goes back to the conversations

3268.02 -> that you need to be having across

3269.19 -> these teams to convey that value like,

3271.927 -> "Why is it important to do this work?"

3273.907 -> "Well it's important

3275.34 -> because you hate scanning 400 applications

3277.89 -> every time there's an issue",

3279.39 -> like do a little bit of this work up front,

3281.4 -> and we can simplify that story,

3283.05 -> I as a developer don't wanna

3284.61 -> have to go through 400 applications

3286.44 -> trying to find which one might be vulnerable.

3290.22 -> I want to be able to get to those

3291.45 -> very quickly fix them get it off my plate

3293.88 -> and get back to doing fun stuff.

3297.21 -> And this starts to lead into to some of the

3299.52 -> observability so as we start to build out our pipelines

3303.45 -> and build out these processes there's a lot

3306.06 -> of components that come along

3307.05 -> for the ride and we need to be observing

3310.2 -> every aspect of this is not that we just observe

3312.66 -> the running application that's something you

3314.34 -> should absolutely be doing but the pipeline itself

3317.31 -> is something that should be observed in its entirely

3319.74 -> and there's a lot of components

3321.54 -> and a lot of tools I know James had

3322.77 -> that slide up earlier that have

3324.3 -> all of the various components.

3326.25 -> The idea is each of those can provide

3327.99 -> data that you can observe against

3329.55 -> and react against we have something like AWS

3332.85 -> distro for open telemetry that helps standardized that

3335.55 -> so instead of having different file formats a different

3338.55 -> structures you then have to pull in manually consume

3341.55 -> and try and make sense of to load in some kind of data lake

3344.1 -> that you can then scan against,

3345.57 -> and how many of you have managed an ELK stack manually?

3348.48 -> You know I have it's a pain in the butt to load all

3350.79 -> of the different log sources into that that that system

3354.48 -> and make sense of it so ADOT as we call it,

3358.74 -> it simplifies and standardizes that so you can pull in all

3361.74 -> the data from AWS services you

3363.75 -> can pull in from third party services

3365.94 -> and you can start to feed those into CloudWatch,

3369.117 -> you can feed it into Data Dog,

3372.81 -> you can feed it into NewRelic,

3374.31 -> and you can start to make sense

3375.45 -> of it all with a little bit more simplicity.

3379.02 -> Now.

3380.43 -> As we start to build this out everything that's in here is

3383.55 -> an event so every code repository commit,

3387.45 -> every build process it's all events,

3391.74 -> and it's going to lead to some kind

3393.99 -> of dashboard that you're gonna have to make.

3395.85 -> So every every part of that process

3399.81 -> or every team that's involved in those

3401.19 -> processes is another persona that you have to kind

3404.1 -> of feed so your developers are one persona,

3406.32 -> your security team is another persona,

3408.24 -> your ops teams are another persona.

3410.13 -> Each of them needs to have contextualized

3411.84 -> information that's available to them

3414.09 -> and all of these can be pulled together

3416.13 -> through something like security hub

3417.81 -> so as these events are occurring

3419.76 -> and all of these these partner

3421.11 -> solutions are these third party solutions

3422.64 -> that you're trying to pull data from,

3424.2 -> they can feed up into a centralized dashboard

3426.54 -> or a single pane of glass that

3428.49 -> then can take action against it.

3430.14 -> So if there's a runtime vulnerability that's

3432.51 -> being exploited in one of your applications

3434.46 -> and Falco triggers on that it can notify

3437.16 -> security hub through these processes in real time.

3440.37 -> Alert the security team,

3441.57 -> security team now

3442.41 -> can automate the alert to you as the developer

3445.38 -> so you can go in and figure out what's going

3446.73 -> on and the security team can

3447.84 -> go in and quarantine that and this will actually feed

3450.09 -> into event bridge so you can actually automate some

3452.61 -> of the response so you can go in and hit

3456.087 -> the system quarantine that particular container

3458.97 -> so that you can go through and do a scan against it.

3461.76 -> The idea is observability

3463.08 -> and I love this graphic is just pretend there's

3465.45 -> an actual dashboard on there

3466.41 -> unfortunately we couldn't find a picture

3467.85 -> that we're allowed to use

3468.683 -> that had an observability dashboard,

3470.4 -> it makes it look cooler than it actually is. (chuckles)

3473.22 -> But the idea is observability is key,

3476.64 -> if you have a dashboard that's green nobody's going to look

3479.97 -> at that dashboard if it's green all the time

3482.16 -> because who cares it's just it's just noise

3484.74 -> so the idea is you want to provide

3486.69 -> contextualized information

3488.46 -> to the teams that need that information

3490.17 -> need to take action against it.

3491.82 -> So we want to basically make sure

3493.29 -> that you can provide full stack

3494.61 -> observability not just for your security,

3496.77 -> but the runtime of the running

3498.03 -> and execution of your applications,

3499.65 -> and one easy way to do this is using something

3501.93 -> like CloudWatch which can be installed through an agent.

3505.17 -> So you can install this agent on your systems

3506.88 -> or you can put it in your task definition if you're

3508.68 -> using ECS you can put it in your deployment file for.

3512.04 -> For EKS

3514.53 -> but the idea is you want to start pulling this data

3517.08 -> in and making sense of it.

3519.51 -> So like I mentioned we've got ADOT, you've got X-Ray,

3522.36 -> that can do tracing within your applications

3525.69 -> and the idea is you want to take advantage

3527.1 -> of all this data to pull it together

3528.75 -> so you can make sense of it and take action against it.

3532.137 -> And so trying to wrap this up really quickly

3534.99 -> because I only got seven seconds before

3536.16 -> they're going to kick me off stage. (laughs)

3538.02 -> We've learned about how you secure your applications

3540.72 -> from the pipeline through the build to the runtime scenarios

3544.74 -> and beyond and the idea is that

3546.84 -> you should be taking advantage of this wherever you can,

3549.21 -> start somewhere,

3550.14 -> grow it over time you don't have to do it all at once.

3552.87 -> But the idea is you should be taking actions

3554.55 -> to do that if you want to learn more there's a couple

3556.56 -> of resources here these slides will

3558.78 -> be available outside of this shortly after

3561.12 -> and there will be a recorded session as well,

3562.95 -> and so with that said,

3564.87 -> I wanna say thank you.

3566.638 -> (round of applause)

Source: https://www.youtube.com/watch?v=PBxXcdchtjo