Who Owns What? Security Ownership and Responsibility at AWS | Amazon Web Services

Aug 16, 2023

Who Owns What? Security Ownership and Responsibility at AWS | Amazon Web Services

How clear is security ownership in your organization? If patching needs to happen fast, will your workforce be able to act quickly or will they question who owns what? In this interview with Laura Grit, a Distinguished Engineer at AWS, we’re discussing the importance of shared security responsibility.

Join Laura and Clarke Rodgers, Director of AWS Enterprise Strategy, as they discuss the structure of the AWS security and engineering departments and how responsibilities are shared between both. Watch now or read their conversation in detail below to hear a real-life mitigation example of how AWS handled Log4J.

Learn more at: https://go.aws/3OPNqTC

Subscribe:
More AWS videos: https://go.aws/3m5yEMW
More AWS events videos: https://go.aws/3ZHq4BK

Do you have technical AWS questions?
Ask the community of experts on AWS re:Post: https://go.aws/3lPaoPb

ABOUT AWS
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers — including the fastest-growing startups, largest enterprises, and leading government agencies — are using AWS to lower costs, become more agile, and innovate faster.

#SecurityEngineering #SecurityOwnership #CloudSecurity #AWS #AmazonWebServices #CloudComputing

Content

8.19 -> - Customers today are trying to figure out

9.99 -> the best way to balance speed and security.

12.51 -> Sharing security ownership

13.71 -> across the business may be the answer.

16.05 -> I'm Clarke Rodgers, director of Enterprise Strategy at AWS

19.71 -> and your guide for a series of conversations

21.84 -> with AWS Security Leaders here on Executive Insights.

25.35 -> Today, we're talking with Laura Grit,

27.45 -> Vice President and Distinguished Engineer at AWS.

30.93 -> As an AWS leader with an engineering background

33.446 -> Laura understands how excellence in engineering contributes

36.81 -> to better security and optimal business outcomes.

39.99 -> We hope you enjoy.

56.37 -> Laura, thanks so much for joining us today.

58.05 -> - Yeah, thank you for having me.

59.37 -> - If you'd be so kind,

60.33 -> could you tell us a little bit

61.44 -> about your background and what brought you to AWS?

64.29 -> - Sure.

65.34 -> I joined Amazon almost 15 years ago

67.98 -> after getting my PhD in computer science.

70.47 -> And I actually joined on the store side of Amazon.

74.07 -> One of the early things that I did

75.6 -> was I ran Amazon's adoption of AWS-

78.266 -> - Oh wow. - The early days of AWS.

82.23 -> And so got to experience what our customers experience

86.28 -> when we look at cloud migrations

88.5 -> from the Amazon perspective.

90.54 -> From there, I joined our infrastructure organization.

93.45 -> I worked on network scaling and availability

96.9 -> and then several years ago, I came over

98.7 -> to the services part of AWS.

100.77 -> I am a distinguished engineer

102.3 -> which is an individual contributor

104.37 -> at the vice president level.

106.38 -> This is the senior most engineering role

109.05 -> that we have at Amazon,

110.64 -> and there's actually only a handful of us.

112.89 -> And as a result, all of us are very unique

116.13 -> in the different skills and what we bring to our role.

120.78 -> Something that we all have in common is we are all looking

124.14 -> across the company to see how we can influence

127.17 -> and drive the technical roadmap based on what we observe

130.41 -> from working with different teams across the board.

133.95 -> and ultimately, we're looking at how we can best deliver

137.028 -> across the company for our customers.

139.41 -> - Got it.

140.243 -> So interestingly, I didn't hear security

143.19 -> anywhere in your title

144.93 -> and this is the AWS Security Leaders Video Series.

148.77 -> - Sure.

149.603 -> - And the reason you're here today is to really talk

152.67 -> about how we spread security responsibility out

158.13 -> throughout the organization.

159.69 -> So you're not a security badge person, yet

162.09 -> you do have quite a bit of security responsibility.

164.85 -> Could you talk a little bit about that

166.38 -> and how that works at AWS?

168.205 -> - Happy to.

169.68 -> Security is our number one priority,

172.38 -> and so as a distinguished engineer,

175.38 -> I am working with security

177.09 -> and ensuring that we are secure for our customers.

180.93 -> I do a lot in collaboration with our security organization

185.31 -> especially as we look at the types of programs

188.97 -> and other work that we do that cross our engineering teams.

193.05 -> One of the things that we do at Amazon

194.734 -> is our engineering teams act very distributed.

199.23 -> They prioritize their own roadmaps

201.39 -> based on the needs of their customers.

203.94 -> And so there's work that we do that really spreads

207 -> across all of our service teams.

209.31 -> And so I partner with security

211.56 -> and the different services that we have

215.04 -> for how do we optimize that work?

217.14 -> How do we automate where possible?

219.36 -> And so kind of work collaboratively

222 -> between the different organizations.

224.16 -> - Got it.

224.993 -> So like if we take a service for example

228.6 -> how would you influence their adoption

233.038 -> of a certain security principle

234.81 -> or maybe an engineering best practice that you've seen

238.44 -> in another service for example?

240 -> - One of the things that we do is security

243.72 -> is part of all aspects of our development.

247.26 -> And so while we have a separate security organization

251.97 -> for AWS, the security engineers are partnering

255.42 -> with our software development engineers from the design

258.78 -> and architecture to put those security best practices

262.23 -> into the design.

264.42 -> So as we learn about best practices from another team

267.6 -> that then is helped communicated

270.21 -> by our security organization

272.07 -> and the other documentation they have for how they build

275.34 -> and because they're partners as we're developing,

278.1 -> as we're reviewing designs, that then is able to get

281.28 -> from the get-go as opposed to having to do it

284.1 -> at the very end when the feature is ready to go.

286.86 -> All of our features and services

288.84 -> go through a security review as process of part of launch.

293.25 -> And then as we're operating our services as well,

295.92 -> we are partnering with security for that.

298.77 -> Ultimately, it is our services teams

301.5 -> that own the security of their service.

304.11 -> We don't hand that responsibility

305.91 -> off to the security organization.

308.64 -> Instead, they're partners with us.

310.14 -> - That's fantastic. That's fantastic.

312.15 -> So, culturally I speak to a lot of different types

315.9 -> of customers in different industries.

317.64 -> And there's often a conflict

320.94 -> or almost headbutting between the security team

324.66 -> and the engineering or the development teams.

327.489 -> It's relatively easy for a security person

331.152 -> to understand how important security is

334.05 -> for the overall health of the business, so to speak.

337.35 -> But oftentimes, I run into customers where the developer,

341.1 -> the development staff is focused on,

343.177 -> "Well, I need to get this feature out the door.

345.21 -> That's why you hired me."

348.6 -> What is done at AWS and perhaps

351.57 -> the larger Amazon to make sure that there's a certain level

355.02 -> of ownership for that developer

358.26 -> around the securitization of that particular service

361.53 -> or feature that they're working on?

363.57 -> - Yeah, great question.

364.53 -> This really starts with our CEO.

368.19 -> And talking about security as our top priority

372.15 -> in all hands, in other communication that they're doing.

375.66 -> This also feeds then into our vice presidents

378.81 -> how they're talking to their organization.

380.82 -> Part of our onboarding training talks about the importance

383.55 -> of security, why it's such a high priority for us.

386.64 -> And this crosses all roles,

388.83 -> which I think is another important thing.

390.63 -> So it's not just our engineers

393.27 -> that are getting that message, it's our product managers

395.79 -> it's our managers as well that this is all of our job.

401.1 -> And it's not just the job of security, but it's the job

404.52 -> of all of us to ensure that our services are secure.

408.7 -> - Got it.

409.533 -> So again, if I'm a developer

412.23 -> and I'm putting something

414.42 -> through the CI/CD pipeline to get to that next stage

417.57 -> and I get something kicked back

419.07 -> because it didn't meet a certain security standard,

421.95 -> I imagine it's viewed more as a learning opportunity

424.86 -> and a opportunity to make the software better

428.25 -> versus, "Clarke, you're a terrible developer.

431.34 -> and can't secure."

432.63 -> - Correct.

433.5 -> And this is really where the partnership comes in.

436.02 -> Having security be part of our processes early on

440.01 -> in development and not just late helps prevent

444.06 -> some of those last minute surprises.

446.01 -> But also as we're going through when we are scoping

450.06 -> or planning the work of how long it's going to take,

453.09 -> we put in that time to be able to remediate anything

457.47 -> that security finds as they're going through

460.381 -> the review before we launch.

461.94 -> And so part of that is also just upfront planning

465.33 -> to be able to put that in so that reduces

469.29 -> some of that tension that I think others can have

472.05 -> of really pushing something to get to launch

473.82 -> when you have that as part of an expectation

476.88 -> of what it means to release a feature.

478.86 -> - So how, in that model,

481.14 -> how are conflicts managed

482.7 -> when a feature supposed to go out tomorrow

485.76 -> and then a security vulnerability has been found

488.01 -> that maybe takes two weeks to fix.

490.65 -> How does that conversation happen within the service teams?

494.34 -> - For us, that isn't a conflict.

496.77 -> That is a conversation that is figuring out

500.1 -> what we need to do to do the right thing for our customers.

502.59 -> And if we find a security vulnerability

504.99 -> that needs to be addressed, it needs to be addressed.

507.6 -> And we then have that conversation with the executives

511.56 -> whoever needs to be involved to explain what is happening,

514.35 -> what is the risk and why that's being pushed out.

517.26 -> So part of security being that top priority

521.37 -> is being able to have these conversations at all levels

525.48 -> as we get surprised or as things need to be prioritized.

529.35 -> - And then the executives get it, the product owners get it.

531.87 -> And again it's not a fist fight at the last minute.

534.307 -> - Correct, we are engaged,

536.64 -> we know that this is an important thing

539.25 -> for our products to be secure.

541.68 -> And so it really becomes part of that definition

544.56 -> of what it means to release a feature or service.

547.23 -> - So Laura, as you know, I meet with a lot of customers

550.68 -> throughout their digital transformation journey.

553.08 -> Some are starting off and they're just pointing

555.99 -> and clicking in the AWS console.

558.03 -> Some have fully built out CI/CD pipelines

561 -> and don't even know how to spell console

562.98 -> because they're hardly ever in it.

564.36 -> Integrating security and a strong engineering culture

568.8 -> can be a challenge depending on where you are

571.981 -> in those different stages.

572.814 -> If you look at the sort of crawl, walk

574.56 -> run and sprint of cloud adoption.

577.38 -> Do you have any sort of tips or tricks

579.24 -> from an organizational and management perspective

582.45 -> to drive these behaviors that you're looking for?

586.41 -> - One of the things that I did early on

588.63 -> in my time at Amazon was work on Amazon's adoption of AWS.

592.44 -> And I've continued to be part

595.26 -> of many migrations that we've done internally.

598.98 -> And something that we do that's been really helpful

603.06 -> is when we are looking at a migration or transformation,

606.913 -> we take a few different types of use cases.

610.44 -> A couple are ones that we think are going to be easier.

613.59 -> Ones that we can get early wins to show success.

617.28 -> And then another one we'll choose

619.08 -> is one of the hardest ones that we think

621.93 -> we need to accomplish.

623.91 -> And so we were able then to go

625.56 -> to the other teams that felt they had a hard use case

628.41 -> and say, "But we got them over the line."

630.33 -> And then that showed them,

632.49 -> oh, well we can't just say that we're too hard

634.77 -> because you actually got one

636.06 -> of the hardest use cases over the line.

638.46 -> But then we also worked with some of those ones

640.62 -> that were more straightforward

641.7 -> so that we could get early wins and publicize

644.19 -> what we were doing into the company of,

646.777 -> "Hey, check this out.

647.91 -> This is really cool.

648.743 -> Anyone want to partner with us?"

650.25 -> And really build that momentum.

652.984 -> This helped us figure out what toolings, documentation,

656.37 -> those types of things we needed to do

657.9 -> while we were also working on the harder transformation.

662.55 -> Another thing that I really focus on

665.19 -> is how I can get engineering engagement

668.73 -> and how I can kind of rally our software developers

671.79 -> to get excited about the work that we're doing.

674.58 -> And part of this is doing talks or having good white papers

679.467 -> or other conversations, whatever your company uses

682.44 -> for that type of communication.

684.06 -> I've really found a lot of great folks across the company

688.32 -> that want to partner with what I'm doing,

690.78 -> either because they're really excited

692.4 -> about it or because they know

694.38 -> that they could really help shape what we're doing

697.5 -> and they want to be part of that shaping.

699.69 -> And so finding ways to find those people

702.6 -> throughout the company can also really help you

705.21 -> because that gets the word out

708.03 -> and they can really help spread what's happening

710.34 -> better than like a single person or a team

712.53 -> of people might be able to do that are more centralized.

715.17 -> - So almost having guilds of people

717.72 -> in particular areas of expertise

720.096 -> and they just help sort of coach each other,

721.86 -> it sounds like.

722.7 -> - Yeah, because they are working

724.23 -> on their particular service,

725.7 -> which spread across the company,

728.25 -> but they'll provide feedback

730.32 -> or, "Hey we're thinking about doing this,"

731.94 -> or, "This is a challenge that we're running into."

734.22 -> And having those folks that are sounding boards

737.52 -> that provide their insights of what that would mean

740.1 -> for their service or what that would mean

741.75 -> for their space can really also help accelerate

745.26 -> and ensure that you are building the right thing

747.99 -> for a larger set of services.

749.52 -> Then you might be like the couple early adopters

752.22 -> that you're working closely with.

753.78 -> - So in that sort of transition phase,

757.05 -> for lack of a better word,

758.88 -> where do you start introducing that security early

762.03 -> in the adoption or early in the development cycle?

765.3 -> - What I have found with the work that I do

767.28 -> is it is incredibly important to engage security early

771.78 -> and to work with them as partners on that transformation.

775.38 -> Making sure that you have senior security engineers as part

778.739 -> of that core working group is really important.

782.13 -> And because they can then look at things from that angle

784.77 -> which might be different than your particular perspective

787.32 -> and help you not have to make changes after the fact.

791.28 -> And instead build those security best practices in

794.43 -> from day one can really then help you accelerate

798.18 -> and also not have to run another migration

800.76 -> or something later because you didn't engage them

803.7 -> early in the process.

804.87 -> - So as customers are trying to build out

808.17 -> their security organization

809.7 -> and a strong engineering culture as well,

812.37 -> how can strong engineering actually support

815.19 -> the security team or the overall security mission?

818.79 -> How does being a strong engineer, knowing everything

821.73 -> about the libraries I'm using and things like that,

824.64 -> how does that benefit the overall security mission

827.67 -> of the company?

829.41 -> - Having engineers that are thinking security from the start

834.3 -> and how we build our tools and how we build our processes

837.93 -> encompassing security is really important for us.

841.74 -> Part of it is bringing

843.81 -> in security engineers themselves into our conversations

848.16 -> as we're building our services, as we're architecting.

850.95 -> And also as we're building tooling.

853.5 -> One of the things that we look for is ways

856.74 -> that we can do secure practices

859.98 -> such as OS patching, software patching

863.137 -> automatically where possible.

865.5 -> And so we've built a couple systems internally

868.95 -> that are able to do that automatic patching.

872.16 -> We also have tooling that allows us to know

875.371 -> what software is being used in different locations.

878.97 -> This then helps our engineers know what they need to patch

883.44 -> what they have on their box, what that looks like,

886.08 -> and helps them have all of the information that they need

890.25 -> to do the security work for their service.

893.7 -> Despite having this decentralized structure,

896.73 -> we do use a common code base, common deployment tools,

900.81 -> common ticketing system, common dashboards.

903.84 -> And that then enables us to be able to communicate

907.95 -> in a common way to the service teams.

910.29 -> And that's a big part of our partnership

912.48 -> that we have with the security organization.

914.52 -> So they are looking to provide us those dashboards,

918 -> that visibility, so that our engineering teams can then go

921.6 -> to those dashboards and see what they need to do.

923.94 -> For example, understand their patching status.

926.31 -> This is the work that is required

929.1 -> in order to maintain the security of our services.

932.46 -> - So then it sounds like that discipline

935.07 -> around what we're building, how we're building it,

937.53 -> what we're using to build it helps

940.2 -> when a new vulnerability comes out.

942.33 -> You're not wasting a lot of time,

943.89 -> do we even use this piece of software?

945.533 -> - Exactly.

946.47 -> - It can just be boom.

947.55 -> Of course, we do, we use this version and this rev

950.67 -> and we know whether we need to patch it or not.

952.29 -> - And what host it's on and therefore what service

955.05 -> is using it and we know how to contact that service to get

957.891 -> to the engineers who will then take action on that.

961.694 -> - It's the discipline as aspect

963.811 -> of the engineering that helps the overall security mission.

967.05 -> - Exactly.

967.883 -> And really looking for those opportunities

969.81 -> to continue to provide good insights and quality data

974.58 -> to our service teams, and then also ultimately,

977.46 -> tooling to help it be easy to do this work.

980.28 -> - That's awesome. That's awesome.

981.87 -> So let's sort of shift to a real world example.

986.07 -> So the world got to experience Log4j and Log4Shell,

990.15 -> and all the quote unquote "fun" that went along with that.

993.09 -> Can you sort of walk us through how that was handled

997.11 -> at AWS from a responsibility perspective

1000.281 -> since we've touched on that a lot today.

1002.57 -> What did, let's just say AWS security do

1006.5 -> and then what did the service teams do

1009.411 -> to address that issue?

1010.79 -> - Log4j, due to the nature

1013.79 -> of the vulnerability that it didn't require sophistication

1017.42 -> in order to exploit was a very, very high priority

1022.52 -> for us to remediate as quickly as possible.

1026.39 -> And so this, we put additional mechanisms

1029.99 -> in place to get the right prioritization on this.

1034.22 -> What happened is the initial engagement involved

1037.97 -> some of our most senior engineers, most senior folks

1042.02 -> in security talking to each other

1044.63 -> and then did engagement with our senior vice presidents

1048.8 -> and CEO around what this was education.

1052.37 -> - So this is not just a run of the mill vulnerability.

1054.373 -> - Correct, this is different.

1056.69 -> We need prioritization, this is what it means

1059.15 -> and that created that top most alignment

1062.06 -> at the highest levels in Amazon for what needed to happen.

1066.53 -> At the same time, they also engaged our senior engineers

1071.3 -> because those are the folks that are going to be working

1073.73 -> with our engineers and our service teams

1075.86 -> to really make sure the right prioritization was happening

1079.76 -> while at the same time we're talking

1081.05 -> to the executives around alignment.

1082.85 -> So that we found to be really important

1085.52 -> to not just talk to management, but also to engage

1088.88 -> our senior engineers in the conversation as well.

1092.21 -> The other thing we did was engage

1094.13 -> the central tooling we have.

1095.66 -> So we have host scans that help us know

1098.18 -> what packages are deployed to what host

1100.22 -> and also we know how to contact the team

1103.4 -> that owns and operates that service.

1106.94 -> And so we were able to leverage

1108.35 -> that central tooling that we use

1110.36 -> for other patching mechanisms

1113.15 -> in order to cut high severity tickets

1116.087 -> through this common ticketing system to our engineers.

1120.23 -> One of the other things that we have found

1121.73 -> over time that really is important

1123.86 -> when we have something that requires engagement

1126.56 -> from this many teams is we first have some engineers

1131.3 -> run through the documentation

1132.71 -> and instructions that we're going to be sending out.

1134.907 -> Because you want to get those tickets out

1136.498 -> as soon as you can.

1138.015 -> But taking that extra half hour,

1140.03 -> that extra hour to really review and ensure

1143.54 -> that it's clear, can help save some of the questions,

1147.08 -> confusion and ultimately get things

1149.27 -> out faster even though it feels upfront

1152.06 -> like you're going a little bit slower.

1153.71 -> - So the work doesn't start until that engineer

1156.32 -> receives the ticket.

1157.22 -> So you have to make sure that the ticket itself

1159.38 -> is as clear as possible so there's no back and forth.

1161.75 -> - Exactly.

1162.583 -> You want to make sure

1163.49 -> that the engineer that receives the ticket

1165.08 -> has all of the information that they need.

1167.36 -> The other thing we did is we set up chat rooms

1169.67 -> as well as calls that our engineers could join 24/7

1173.9 -> to be able to get the information they needed

1177.23 -> to remain unblocked.

1178.46 -> We're a global company and so it's important

1181.01 -> that we had these opportunities for conversation,

1184.46 -> for engagement, for clarification available

1186.89 -> to engineers all of the time that they're working.

1189.89 -> We also continue to engage with our vice presidents,

1193.4 -> with our general managers

1195.29 -> around how their teams were progressing.

1198.74 -> And so we had regular calls with them throughout the day

1202.391 -> being able to update them

1204.41 -> as to where we were so that they could then enable

1207.29 -> that driving of the patching within their service teams.

1210.86 -> So really helping both our managers

1213.62 -> and our engineers get access to as much information

1216.65 -> as we could and being very clear and crisp.

1219.59 -> We also then had status reporting

1221.48 -> up to the senior vice president and CEO level

1223.88 -> so that we kept them inform

1225.56 -> as to what our status was as well.

1227.99 -> - So that's pretty incredible.

1229.49 -> So it sounds like there was a strong partnership

1234.11 -> between AWS security and all the service teams

1237.74 -> and the engineering teams to actually get this done

1240.14 -> and it kind of sounds like the actual security team was more

1244.67 -> of a observe and report the status of the Log4j fixes

1250.22 -> while the engineering and development teams

1253.79 -> are actually doing the quote unquote "work"

1256.01 -> of patching, etc.

1258.2 -> - Exactly.

1259.337 -> Log4j was an example of the great partnership

1262.07 -> that we have between the services part

1264.38 -> of AWS and the security organization.

1267.14 -> We really were in all of this together.

1269.9 -> So the conversations and the calls that we had

1272.63 -> with our executives were being driven

1274.94 -> by both the security organization and the services team

1278.09 -> so that we were reporting where we were

1280.28 -> and addressing anything that we needed their help

1282.8 -> with prioritization or things like that.

1285.17 -> We were getting our status reports,

1286.85 -> the data from the security organization,

1289.37 -> partnering with them, on the tickets that needed to get cut,

1292.7 -> the documentation. And then really

1294.77 -> on the services side, doing the deployment

1297.32 -> doing the actual patching and making sure

1299.48 -> that our software development engineers

1300.98 -> had what they needed.

1301.82 -> So a great example of us coming together

1304.55 -> in order to really make this change very rapidly.

1307.721 -> - Yeah, so that's great to hear

1309.905 -> because a lot of the customers I speak to they think,

1312.35 -> well in order to have a strong security organization

1314.63 -> we need to 10x the population on the security team

1317.54 -> when in fact, you have your extended security team

1320.24 -> in your engineering organization.

1321.74 -> It's just making sure that the partnership

1323.99 -> and the culture supports all that stuff.

1325.73 -> - Exactly.

1326.563 -> Really creating that partnership,

1328.28 -> really helping engineers, managers, product managers

1331.82 -> understand how critical security is for your business,

1335.69 -> really helps create that joint work

1338.75 -> that we're able to do together.

1340.01 -> - Well Laura, thanks so much

1341 -> for spending some time with me today.

1342.83 -> - Thank you so much for inviting me.

1344 -> It was great chatting.

Source: https://www.youtube.com/watch?v=iLr8roESv5Y