AWS re:Invent 2022 - Powering Amazon EC2: Deep dive on the AWS Nitro System (CMP301)

Aug 16, 2023

AWS re:Invent 2022 - Powering Amazon EC2: Deep dive on the AWS Nitro System (CMP301)

The AWS Nitro System is a rich collection of building block technologies—including AWS-built hardware offload and security components—that is powering the recent and future generations of Amazon EC2 instances with an ever-broadening selection of compute, storage, memory, and networking options. In this session, dive deep into the Nitro System, review its design and architecture, explore new innovations to the Nitro platform, and see how it has made the seemingly impossible possible.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

0.27 -> - My name is Ravi,

1.35 -> I'm a senior principal engineer in EC2.

6.03 -> I'm so excited that all of you are here,

8.19 -> 'cause I'm gonna be talking about hardware,

11.91 -> low level firmware, BIOSes.

13.65 -> If that's your kind of thing,

14.7 -> you're at the right place.

17.58 -> So just show of hands, how many of you've been,

21.57 -> or how many of you've heard of Nitro before?

25.62 -> Awesome.

27 -> How many of you've heard of Bare Metal instances before?

31.65 -> You're gonna love this.

36.24 -> So in terms of an agenda, here's what I'm gonna do.

38.64 -> I'm gonna spend a little bit of time talking about Nitro.

42.42 -> There is a 2019 video on YouTube that I did on Nitro,

47.7 -> and so the first 10 or so slides

49.59 -> is gonna sound roughly the same, don't panic.

52.56 -> I do that because a lot of you are new to Nitro

54.66 -> and I wanna set the foundation.

57.6 -> Then I'm gonna talk a little bit about

59.1 -> kind of the continuous innovations.

61.11 -> If you've been listening to all the keynotes, Dave Brown,

65.1 -> Peter DeSantis, there's a lot of talk about Nitro

68.04 -> and the new instances and the new platforms

70.17 -> and Trainium and Inferentia.

72.38 -> So I'm gonna touch upon those, but very, very briefly

75.6 -> because this is a technical deep dive.

77.7 -> And then I'm gonna deep dive on two main topics

80.01 -> I think they're interesting.

81.45 -> First is security

83.07 -> because it comes up in so many conversations.

86.43 -> And then I'm gonna spend a little bit of time talking about

88.32 -> Amazon EC2 Bare Metal instances.

90.03 -> And the reason I wanna do this

91.89 -> is because I'm often in customer conversations where

95.31 -> we talk about Bare Metal, but customers confuse Bare Metal

98.04 -> with things like dedicated hosts and I wanna clear up that.

102.42 -> So let's get started.

105.42 -> So let me take you back a little bit.

108.81 -> In roughly 2013,

111.15 -> we started doing things like offloading I/O.

114.03 -> So the first thing we offloaded was,

116.37 -> now remember these are Zen Hypervisor based

118.68 -> instances in 2013.

121.2 -> We decided that we wanted to offload our network I/O.

124.08 -> So the way we did things was

125.34 -> we had a Intel A259 NIC on the platform.

130.23 -> And we had a custom Nitro Card built

132.39 -> that was the first Nitro Card built

134.34 -> that basically let us do our VPC traffic.

137.31 -> So customer instances talked to the NIC, the Intel NIC,

141 -> and then when the packets actually went out,

143.16 -> they got encapsulated by the Nitro Card

145.95 -> and actually went out in the wire,

147 -> so it was just a bump in the wire.

149.67 -> And this obviously saved,

152.057 -> a bunch of CPU cycles on the host CPU.

155.007 -> And just so for clarity, when I say host,

157.62 -> I mean the actual CPU where customer instances run.

161.43 -> When I talk about servers,

162.81 -> I'm talking about the entire platform,

164.04 -> including the Nitro Cards, just so you're clear.

168.18 -> And then as time went on in 2015,

170.07 -> we decided we were gonna offload storage.

171.96 -> So the first thing we offloaded was EBS,

174.36 -> the Elastic Block Storage.

176.82 -> And in order to do that,

178.29 -> we started working with a company called Annapurna Labs.

181.26 -> This was a fabless semiconductor company,

184.23 -> and they were the first company

186.33 -> that were actually seriously thinking about NVMe,

189.21 -> the Non-volatile Memory Express interface.

192.06 -> NVMe is awesome,

193.14 -> it's an interface that scales really well

195.48 -> for high performance storage,

198.24 -> and so the next thing we did was we offload EBS

201.81 -> to this new card that we built,

203.82 -> the card exposed NVMe to instances.

206.79 -> And so now what was happening was,

208.8 -> instances were actually getting more CPU cycles

211.23 -> because they were not spending CPU cycles

213.24 -> processing network traffic and storage traffic

215.94 -> to do things like,

217.59 -> completions and submissions and so on.

219.72 -> 'cause all of that was offloaded to the cards.

222.36 -> And then in 2016, we offloaded,

225.99 -> so after we offloaded EBS storage,

227.85 -> we decided the next thing we wanted to offload

229.53 -> was Local Instance Storage.

231.21 -> And as I go through the presentation,

232.56 -> I'll kind of describe the differences between

234.3 -> EBS storage and local instance storage.

238.08 -> And that led to our I3 instances

239.82 -> that were pretty much used by VMware as a first customer.

243.84 -> But as you can see,

244.673 -> the story has been that over time,

246.87 -> we started to offload things to Nitro Cards.

249.48 -> And in 2017, we actually decided to offload our hypervisor.

254.1 -> So things that were traditionally done in hypervisors were,

259.26 -> take a physical box,

260.28 -> carve it up into multiple instances or guest VMs, right?

264.33 -> And do stuff for those guest VMs, emulate I/O,

268.62 -> emulate the virtual machine, and so on and so forth.

271.98 -> We decided to offload them to the Nitro Cards.

275.37 -> And so that was kind of what led to Nitro.

277.92 -> So Nitro is is the underlying platform

280.38 -> that actually powers all of EC2.

283.26 -> The first Nitro instance that we launched was C5 in 2017.

286.71 -> So if you've used a C5 2X large,

289.08 -> you're running on a Nitro system based hardware.

293.16 -> So put simply, Nitro is basically purpose built hardware,

297.57 -> purpose built software, purpose built firmware

300.3 -> to basically let us improve on our performance

304.2 -> and security posture.

306.54 -> It also includes a purpose built hypervisor.

313.2 -> But when we talk about Nitro, we talk about the,

315.23 -> we talk about things in terms of a Nitro system,

318.426 -> it's really three different things.

320.34 -> The Nitro system consists of a series of Nitro Cards.

323.88 -> I kind of touched upon them briefly,

326.764 -> a Nitro hypervisor that's purpose built,

328.68 -> it's actually based on Linux and KVM,

330.54 -> but we've like modified it pretty heavily for a reason.

335.73 -> And the Nitro Security Chip

336.81 -> in the middle of everything else.

337.89 -> The Nitro Security Chip

338.82 -> was built for a very specific reason,

341.16 -> and you'll follow along as I go through the slide.

346.5 -> So let's talk about the Nitro Cards first.

349.14 -> There are actually four different cards.

352.2 -> And each of these cards is built for a specific purpose.

355.56 -> Now each of these cards

357.42 -> have fundamentally an SOC, an ARM SOC,

361.26 -> they're actually running some software on it.

363.57 -> And for the most part,

365.16 -> a lot of software is shared between these cards,

367.29 -> it's common software,

368.64 -> but each card specializes in one thing.

371.25 -> There is a Nitro Card for VPC,

372.9 -> it does our virtual private cloud networking.

375.87 -> There's a Nitro Card for EBS,

377.49 -> it does our NVMe based storage,

380.55 -> there's a Nitro Card for instance storage,

383.88 -> it does, Nitro Card for,

386.291 -> it basically exports Local Instance Storage to the instance.

389.82 -> And then there is a card called the system controller,

392.22 -> the Nitro Controller,

393.053 -> it's really the brains behind the whole Nitro system.

399.39 -> So let's dive into each one of these cards now.

402.51 -> So the Nitro card for VPC is a network card.

406.95 -> If I physically walk into a data center

409.56 -> and actually pull one of the servers out of the rack

412.47 -> and take one of these cards in my hand,

413.94 -> it would look just like any other PCIe card,

416.76 -> I mean it physically looks like a PCIe card.

418.977 -> And if I hold it, it actually looks like a NIC.

421.77 -> It's got a PCI interface on one end,

423.75 -> it's got ports connecting to the network

425.52 -> on the other end, right?

426.66 -> Because that's what it does, it does networking.

431.46 -> When it's plugged into a server

433.56 -> and when the server boots up,

435.27 -> what the host hypervisor sees is a bunch of

439.38 -> elastic network adapter virtual functions, right?

442.8 -> So it uses SRIOV,

445.74 -> but what the host sees is an ENA NIC,

448.41 -> just like any other NIC.

449.94 -> And so there's an ENA driver in the instance

452.37 -> that actually binds to that NIC and basically says,

454.92 -> I'm gonna drive this physical, this virtual function.

459.36 -> But the whole idea here is that

460.62 -> we implemented this ENA interface because,

463.83 -> think about the problem where you have a data center

466.65 -> and you're actually managing all these servers.

469.02 -> And if you decide you want to upgrade your NICs

470.61 -> from one gig NIC to 10 gig NICs,

472.38 -> well here's what you have to do.

473.213 -> You have to take the one gig NIC out,

474.88 -> put a new 10 gig NIC in.

477.66 -> More than likely,

478.493 -> you'll have to go change the drivers in the operating system

480.54 -> 'cause that'll come with new drivers,

482.04 -> and that's a huge pain to do at scale.

484.32 -> Well the ENA interface that we implemented

486.69 -> is actually this common interface,

488.1 -> it actually looks very similar to NVMe.

490.29 -> There's completion queues, there is submission queues,

493.05 -> there is MSIX interrupts going both ways.

496.14 -> And so what essentially what happens

498.93 -> is the ENA driver loads,

500.43 -> and it doesn't matter what generation of NIC we have,

503.34 -> you could have a, C5 instances, for example,

505.35 -> support 25 gig, C5N instances support hundred gig,

508.83 -> it's a brand new card that does a hundred gig,

510.99 -> but the instance driver is the same.

513.21 -> So you could take a army built for C5

516.6 -> and actually run a C5N instance

518.37 -> and get 800 gig with exact same driver.

520.68 -> That was the whole point of the ENA NIC.

524.07 -> But besides doing just net standard networking,

526.98 -> it also does all of the things

528.06 -> that we expect from VPC, right?

529.233 -> The virtual private cloud.

531.21 -> The virtual private cloud as you would expect,

533.19 -> is a software defined network.

536.01 -> So it does things like encapsulation and decapsulation.

538.29 -> So when your instance sends a packet

540.99 -> in the transmit direction out,

542.97 -> the packet the Nitro Controller,

545.58 -> the hits the Nitro Card for VPC,

548.1 -> the Nitro Card then encapsulates it into an overlay network

550.86 -> and sends it out on the physical network.

553.95 -> The other thing that the card does

555.39 -> is security groups, right?

557.19 -> This is firewalling.

558.27 -> So you as a customer can launch an instance,

560.747 -> use an API to say,

562.29 -> I only want to allow traffic in the inbound direction,

565.08 -> or I only want to allow traffic in the outbound direction.

567.84 -> I only want to allow these ports

569.16 -> to be able to talk to my instance,

570.3 -> so it's a way to firewall your traffic.

572.34 -> But all of that is actually implemented

574.17 -> on the Nitro Card for VPC as opposed to in the instance

578.01 -> that's running on the CPU.

580.59 -> Limiters is the other thing we do on the Nitro Card.

584.04 -> One of the goals we had is if you launch a C5 instance,

587.85 -> it doesn't matter which you are in or which,

590.64 -> where on the rack you're in

591.87 -> or what part of the data center you are in.

593.67 -> We wanted to give customers consistent performance,

596.67 -> bandwidth and latency, right?

598.62 -> Limiters is our way of basically saying

601.38 -> we can actually provide that consistent latency

603.57 -> no matter where you're physically in the network.

606.33 -> And then last thing,

607.23 -> the Nitro Card for VPC does is routing,

610.461 -> which is what you would expect when you have

612.72 -> a bunch of instances in your VPC

614.88 -> and they need to talk to each other.

617.1 -> But the key point here is that the Nitro Card for VPC

620.19 -> offloads all of this stuff

622.11 -> and it doesn't consume cycles in the instance.

627.78 -> The other thing that the Nitro Card for VPC does

630.12 -> is this thing called Elastic Fabric Adapter.

631.86 -> In fact, we actually built this

634.32 -> for applications like HPC and machine learning.

638.22 -> I have a background in HPC,

639.42 -> so I'll go into this a bit,

641.19 -> but on the, what is it?

646.43 -> It is your left side,

648.24 -> is a classic HPC application

650.73 -> running on a standard TCP/IP network, right?

653.52 -> So the NPI application, in this case called HPC application,

657.57 -> when it wants to send a packet to the network,

659.52 -> that packet has to traverse the entire TCP/IP stack

662.07 -> down to the ENA driver and then out the wire.

665.49 -> What happens with EFA is, EFA is a fabric,

668.16 -> it's a high performance fabric.

670.65 -> So HPC applications and machine learning applications

674.46 -> sit on top of libraries like NPI and NCCL,

676.86 -> Nvidia's NCCL.

678.18 -> But they use this, this library called Libfabric.

680.937 -> And Libfabric lets you provide a,

682.71 -> we implemented a provider for EFA that basically says

686.88 -> when you want to initialize the adapter,

688.8 -> you go through that part that says,

691.41 -> transition from Libfabric to user to kernel

693.42 -> back into the EFA driver.

694.47 -> But once the setup is complete and you have cues,

697.86 -> the application talks directly to the hardware,

700.11 -> there is no ring transition from user mode to kernel mode.

702.9 -> So you get this really high performance, high bandwidth,

705.45 -> low latency, and it's perfect for HPC.

708.48 -> The reason this is interesting is,

710.293 -> typically HPC applications

711.66 -> are trying to solve large problems, right?

713.52 -> They're trying to solve protein folding

716.25 -> or molecular dynamics or some fluid simulation.

721.29 -> And in order to solve a large problem quickly,

723.69 -> they distribute the problem

724.83 -> across many, many, many, many nodes.

727.26 -> While the nodes do two things,

728.73 -> they compute stuff, right?

730.48 -> So you use ABX instructions

731.94 -> and you use cache prefetching and so on.

734.37 -> But at every stage, they have to exchange

736.62 -> a bunch of partial results with each other.

739.32 -> And in order to do that efficiently,

740.91 -> they need a high bandwidth, low latency network.

744.18 -> And in order to do that, when you use something like EFA,

746.91 -> you get really, really consistent high performance.

753.48 -> EFA is actually based on an underlying protocol

756.27 -> called Scalable Reliable Datagrams,

757.89 -> and I pulled up this paper and I thought I should just

760.77 -> link it in my slide deck here.

763.14 -> This is the paper that actually talks about

764.79 -> the implementation of SRD that EFA uses.

768.66 -> SRD was meant, or was designed for data center environments

772.95 -> and it uses things like multi-pathing.

775.05 -> So if there are hotspots in the network,

777.69 -> we actually work around them.

781.62 -> And before I move on from the slide,

783.03 -> one of the things I'd like to mention is,

784.59 -> I think yesterday there was announcements of

788.73 -> this thing we call EFA Version 2 or EFA v2.

793.02 -> It's something we've been working on.

794.61 -> So EFA was launched in 2018 and since then,

797.43 -> we've been improving the performance of EFA

799.65 -> and customers have been enjoying

801.69 -> all the improvements we made

802.74 -> because we are able to actually update our machines

805.38 -> dynamically when customers are running.

807.99 -> EFA v2 specifically brings two main benefits.

813.27 -> The first is it enables full RDMA semantics,

819.63 -> so if you're doing HPC applications with RDMA.

823.56 -> And the second it brings

824.64 -> it actually goes from hundred gigabit

826.2 -> to two hundred gigabit,

827.16 -> which is what EFA v2 does.

831.45 -> But in general, if you look at ENA and EFA,

834.39 -> this is what's been happening in EC2, right?

836.94 -> In 2013, I mentioned the A259 NIC with the bump in the wire.

842.04 -> We were at about 10 gigabits per second

843.48 -> with the Intel A259 NIC.

845.85 -> In 2017, 2018 when we launched,

848.61 -> when we launched C5 with Nitro,

850.71 -> we were at 25 gigabits per second.

852.27 -> So there were instances

853.71 -> that were actually C25 gigabit per second.

856.53 -> Then we launched our C6 instances recently,

860.61 -> they offer 50 gigabit per second.

863.04 -> We also have network optimized instances

864.81 -> that do like 100 and 200.

866.1 -> So like C5N was a hundred gigabit per second NIC,

869.34 -> and we just announced a bunch of NICs yesterday

871.83 -> with C6IN, M6IN, R6IN,

874.62 -> our Ice Lake based instances

876.87 -> that do 200 gigabit per second bandwidth.

881.55 -> And then if you obviously combine multiple cards

883.65 -> in a platform and you use multiple NICs,

885.84 -> you can actually get 400, 1800, 400, 800

888.58 -> and I believe even 1600 gigabits per second.

894.6 -> The next card in my series is the Nitro Card for EBS.

899.28 -> The Nitro Card for EBS is, again,

901.32 -> it looks like a pretty standard PCIe card,

903.78 -> but what it does is when the host boots up,

905.97 -> it actually exposes NVMe to the host.

910.17 -> It actually, so from an instance perspective,

912.6 -> the instance has an NVMe driver,

914.033 -> NVMe is a pretty standard protocol.

917.01 -> The instance driver talks to the card over NVMe,

920.25 -> it submits requests, gets completions and so on.

923.64 -> The card itself actually implements

926.55 -> block storage over a network.

928.23 -> So when you, when the customer instance is running

931.26 -> on the host talking NVMe,

933.48 -> we actually take those requests

934.98 -> and we ship them to our EBS servers

936.75 -> on the far end of the network.

939.12 -> What this gives you is durability.

941.37 -> You can actually stop your instance

943.47 -> or terminate your instance,

944.52 -> but your disc that's sitting over your EBS volume

946.86 -> on the back end is completely persistent.

950.34 -> So you can come back the next day and say,

952.47 -> I want to start a new instance and attach this volume to it.

955.71 -> And so the mental model is,

958.38 -> you have your laptop,

959.52 -> you've decided you wanna go home for the day,

961.26 -> you shut your laptop,

962.58 -> doesn't mean your disc is destroyed, right?

963.96 -> Your disc is persistent.

966.87 -> The Nitro Card for EBC does a few other things.

970.65 -> It supports encryption,

971.76 -> so if you actually use encrypted volumes

973.95 -> when you launch an instance,

975.81 -> all of the encryption happens on the card,

977.49 -> it's actually offloaded to the card.

979.65 -> And so we've been doing NVMe based remote storage,

983.46 -> basically NVM fabric for a very long time,

985.47 -> at least in C5 launched.

990.06 -> The Nitro Card for Instance Storage,

992.16 -> the mental model that I want to leave you with is,

994.26 -> imagine a server with a Nitro Card

996.87 -> and a bunch of discs hanging off of that Nitro Card,

999.66 -> so a bunch of SSDs, right?

1002.06 -> This is a card if I were to look at it,

1003.89 -> it would have a PCIe interface on the top

1006.17 -> and a bunch of NVMe interfaces on the bottom.

1008.9 -> From an instance perspective, again,

1010.61 -> you see NVMe just like EBS, but in this case,

1013.675 -> when you actually write NVMe commands,

1016.1 -> you're writing to the local discs.

1017.69 -> So there is no network attached storage, it's just local.

1021.29 -> You get great performance.

1023.18 -> However, the lifetime of the disc is tied to your,

1026.133 -> is tied to the lifetime of your instance.

1028.07 -> When you terminate your instance, we wipe the discs, right?

1031.22 -> And the way we wipe the discs

1032.21 -> is we obviously encrypt everything

1033.44 -> and we basically throw the keys away.

1035.24 -> It's that simple and we trim the drive.

1038.75 -> A lot of our customers,

1039.74 -> especially database customers use this particular storage,

1043.64 -> they build their own replication strategies and whatnot.

1047.18 -> But local instance storage actually

1049.04 -> is extremely high performance.

1052.04 -> And like I mentioned,

1052.873 -> we also do all of the encryption on the card,

1055.1 -> we do limiters.

1056.36 -> One important thing that this

1057.89 -> Nitro Card for Local Instance Storage does

1060.32 -> is related to NVMe and NAND storage.

1063.71 -> The way NAND storage works is it's, you can always write,

1067.31 -> you can always go from a one to a Z

1068.87 -> or you can go from a zero to a one on NAND,

1070.61 -> but you cannot go backwards.

1072.05 -> So, the more you write to NAND, the more it wears out.

1076.58 -> So there's this thing called

1077.413 -> wear leveling that needs to happen where

1081.447 -> the FTL, it's a piece of firmware

1084.32 -> that lives in the NVMe controller

1086.87 -> that's constantly looking for new blocks to put your data.

1089.15 -> So every time you write data to your drive,

1091.55 -> it's looking for new blocks.

1092.87 -> And over time, because of use, these blocks wear out.

1096.77 -> The Nitro Controller for Instance Storage

1099.98 -> does one additional thing, which is,

1102.47 -> it looks at, it keeps monitoring wear levels

1104.42 -> of the SSDs that are attached to the card

1107 -> to make sure that you don't have

1108.2 -> this performance cliff, right?

1109.94 -> What happens in the FTL is,

1111.71 -> as you're writing to these blocks

1113.15 -> and the blocks are starting to wear out,

1114.53 -> it's looking for a new block.

1115.73 -> And sometimes over time, that looking for a new block

1118.937 -> can take a very long time.

1120.56 -> And so from your application's perspective,

1122.18 -> when you open a file and you write to the block,

1124.4 -> it takes a very long time

1125.233 -> 'cause it needs to look for a new block.

1127.25 -> And so as the drive gets worn out,

1129.56 -> you start seeing performance degradation

1131.87 -> to the point where you actually fall off a cliff.

1133.91 -> And so before that happens,

1136.1 -> we actually monitor these drives

1138.2 -> because we have monitoring software running on these cards

1140.96 -> and we basically take it out out of production.

1146.84 -> The last, well, not last,

1148.34 -> but one of the cards is the Nitro Controller.

1150.717 -> The Nitro Controller is the brains of the Nitro system.

1154.64 -> And I will tie all of these cards together

1157.31 -> very, very quickly in a picture 'cause it's not,

1159.5 -> if it's not super clear to you, it'll be in a minute.

1162.65 -> The Nitro Controller is the card

1165.29 -> that actually interfaces with the EC2 control plane.

1168.29 -> So, and I'll show you an example

1170.45 -> of what happens when you launch an instance as a customer,

1173.69 -> how it goes through the Nitro Controller,

1175.16 -> and what happens with the Nitro Controller.

1177.65 -> It's the card that basically coordinates

1179.63 -> with all the other cards in the platform,

1181.31 -> so oftentimes we'll have a server with the Nitro Controller

1185.36 -> and multiple Nitro I/O cards for networking and storage.

1188.96 -> The Nitro Controller card

1189.83 -> basically talks to the other cards,

1191.12 -> so there's, think of it as a cluster of cards in the server.

1195.601 -> The Nitro Controller

1196.434 -> also coordinates with the Nitro Hypervisor,

1198.53 -> it's that thin layer of software that's running on the host,

1201.35 -> the X86 or Graviton CPU.

1203.96 -> And the Nitro Controller

1205.01 -> also coordinates with the Nitro Security Chip,

1206.87 -> which I'll get to in a minute.

1211.82 -> So the Nitro Security Chip,

1213.26 -> there was a very specific reason why we built this thing.

1217.01 -> If you think about a server today, any modern server,

1220.61 -> the server has many besides the whole CPUs,

1224.87 -> it has memory DIMs,

1226.4 -> it has a baseboard management controller or BMC,

1229.43 -> it has multiple microcontrollers that are doing things like,

1233.3 -> keeping track of temperatures,

1235.01 -> keeping track of fan speeds and so on and so forth,

1238.19 -> each of these controllers

1239.78 -> is typically running some form of firmware, right?

1243.14 -> And when you power on the machine,

1244.88 -> that firmware has to load from somewhere.

1246.65 -> That somewhere is usually a flash chip

1248.99 -> sitting on the motherboard.

1251.15 -> Well, traditionally,

1252.32 -> if you're running a server in a data center,

1254.51 -> that firmware has to be updated.

1255.77 -> There could be bugs, there could be security bugs,

1258.65 -> there could be just crashes and so on.

1262.34 -> So typically the way these updates are done is you,

1265.16 -> you log onto your host and you run some tool

1267.56 -> and the tool basically says,

1268.7 -> I'm gonna flash that new firmware

1270.5 -> over an I square C bus or over a spy bus.

1275.18 -> Well, after we offloaded everything,

1277.34 -> we realized very quickly

1278.18 -> that we don't actually have to do that

1279.47 -> because we don't have to be on the hypervisor to do it,

1281.69 -> we have a Nitro Controller.

1283.22 -> So the Nitro Controller now is in charge

1285.41 -> of actually updating these firmware flashes.

1289.94 -> And one of the things we decided was,

1291.32 -> since we don't have to support the legacy requirement

1293.6 -> of updating the firmware from the host,

1297.14 -> we can actually just implement a chip

1298.97 -> that sits in front of every I square C bus and every spy bus

1302.9 -> and basically takes all the right transactions

1304.67 -> and throws them away.

1306.38 -> So if you are a customer running on the host,

1308.3 -> there is no way

1309.35 -> you can actually get to one of these spy flashes,

1312.29 -> it's just a security feature, right?

1314.21 -> Nobody can muck with the spy flashes on the platform,

1317.15 -> and that basically guarantees that the platform is secure

1319.76 -> every time it reboots.

1322.07 -> The Nitro Security Chip does another thing,

1324.41 -> which is every time the Nitro security,

1326.6 -> every time the platform reboots,

1328.31 -> the Nitro Security Chip holds the CPU,

1331.25 -> the X86 or the Graviton in reset,

1333.74 -> it holds it reset and then basically says,

1336.23 -> let me verify every single spy,

1338.93 -> make sure it matches what I expect it to match,

1341.51 -> and if it doesn't,

1342.71 -> I'm gonna overwrite it with what I think it should be.

1345.77 -> So even if we've had a kind of a vulnerability

1348.92 -> where one of these writes from the host

1352.76 -> is actually able to get to a spy,

1354.41 -> it will get overwritten on the next reboot.

1358.28 -> So pictorially, this is what it looks like,

1360.71 -> so there's, imagine you're a customer,

1362.24 -> you're running on the instance,

1363.62 -> you're running inside an instance on the CPU.

1367.1 -> The Nitro Security Chip, like I said,

1368.48 -> is sitting on all of the I square C buses

1371.15 -> and SM buses, and it basically says,

1373.73 -> I'm not going to let any instance running on this box

1376.94 -> get to any of the spies.

1382.31 -> The last component before I tie everything together

1385.13 -> is the Nitro Hypervisor.

1387.5 -> The Nitro Hypervisor is based on Linux and KVM,

1390.95 -> but we made a very conscious choice to simplify it.

1395.42 -> We don't need things like SSHD,

1397.67 -> we don't need things like system D,

1400.76 -> we don't have a network stack in the hypervisor at all.

1403.55 -> There's no need for the hypervisor to talk to the network

1406.43 -> because the hypervisor

1407.72 -> is only allowed to talk to the Nitro Controller.

1410.9 -> And the reason we also made the simplification

1413.51 -> is it's very easy reason about what the hypervisor is doing.

1415.88 -> It does one thing,

1417.26 -> which is program hardware for virtualization,

1420.08 -> carve up memory and CPUs and get out of the way.

1423.5 -> We also call it a quiescent hypervisor,

1425.18 -> that's the exact word I used.

1426.44 -> And the reason we call it quiescent hypervisor,

1428.15 -> if your Linux, if you guys know how Linux system calls work,

1432.38 -> is if you open an application,

1433.67 -> you make a system call,

1434.66 -> a Linux system called like open to open a file.

1438.2 -> You make a transition from user space to kernel space,

1441.02 -> And that's the only time the kernel's invoked.

1442.94 -> If your application doesn't make any system calls

1445.13 -> and is simply executing application code,

1447.41 -> you don't actually make the transition

1448.61 -> from user space to kernel space,

1450.44 -> That is the case with he Nitro hypervisor.

1452.24 -> when an instance is running and it doesn't actually need

1455.57 -> any support from the hypervisor, it just keeps running.

1457.61 -> The Nitro hypervisor just stays really, really quiet.

1461.33 -> And the reason we did this is we didn't want

1463.033 -> threads running around stealing CPU cycles and so on.

1467.15 -> The goal was to get the Nitro Hypervisor's performance

1470.24 -> to be practically Bare Metal.

1472.79 -> We've actually had benchmarks where

1475.73 -> customers have run benchmarks and said,

1477.59 -> I'm going to run this benchmark on a Bare Metal machine

1480.17 -> and run it on your C5 instance,

1481.82 -> which is running on top of the hypervisor

1483.53 -> and the performance is virtually the same

1486.41 -> to a very long tail latency.

1492.02 -> So let me tie this all together for you, right?

1493.85 -> So imagine you've got a server

1495.44 -> that looks like the picture here, it's a Nitro host.

1498.62 -> And imagine you just launched an r6gd instance.

1501.71 -> And the reason I picked r6gd is it's a,

1504.47 -> it's a memory optimized Graviton based instance

1507.904 -> that uses Local Instance Storage, the d

1511.55 -> and it's the 16xlarge, so it uses 64 cores.

1516.2 -> The first thing that happens when you as a customer says,

1518.93 -> AWS space EC2 space run instances,

1522.77 -> dash instance type r6gd.16xlarge,

1527.871 -> is the EC2 control pane gets your request

1530.21 -> and says, "Okay, let me find you a host where

1533.18 -> your instance can start running."

1535.13 -> So it looks at the capacity in EC2,

1537.35 -> in your easy of choosing, let's say it's,

1540.26 -> US West two, it says,

1541.79 -> I'm gonna find capacity in US West two

1545.357 -> and find the right host.

1548.15 -> It then sends a command

1549.68 -> to the Nitro Controller on that host.

1552.32 -> And when I say sends a command,

1554.18 -> this is actually a API call

1556.43 -> that the control pane makes.

1558.44 -> And I'll describe this in a lot of detail

1559.823 -> when I get to the security section.

1562.46 -> The Nitro Controller then says I'm going to,

1565.01 -> I'm gonna carve up bunch of memory

1567.05 -> and a bunch of CPUs for this instance type,

1570.23 -> and then I'm gonna talk to the Nitro Hypervisor

1571.88 -> and tell it to actually launch this instance.

1574.7 -> The Nitro Hypervisor receives the command

1576.65 -> and basically says,

1577.527 -> "Okay, I've programmed the hardware bits, VTX, VTD,

1581.21 -> all the virtualization features in the CPU

1583.85 -> and I'm gonna get out of the way."

1588.89 -> Step four, the Nitro Controller sends another command

1591.8 -> and says, by the way, in order for that thing to boot,

1594.65 -> it needs an EBS volume, it needs an AMI, right?

1597.47 -> For those of you who know what an AMI is,

1598.7 -> an Amazon Machine Image.

1600.92 -> So I'm gonna take an image that the customer has chosen,

1604.52 -> Amazon, Linux, Ubuntu, Windows, even MacOS.

1609.14 -> I'm gonna take an image,

1610.61 -> I'm gonna put it on EBS volume

1612.23 -> and I'm gonna attach it to the instance on this machine.

1615.5 -> And that's what the Nitro Controller does,

1616.7 -> it exposes NVMe to BIOS on the machine.

1622.58 -> The Nitro Controller then sends a command

1624.14 -> to the hypervisor and says,

1625.52 -> hey, you know that instance that EBS volume

1627.77 -> that this got attached?

1628.76 -> Expose it to the instance using SRIOV, right?

1632.54 -> So that the instance can talk directly to the hardware

1635.03 -> and bypass a hypervisor, that's what step five is.

1639.17 -> Step six, the control plane says,

1641.12 -> great, I've set up all I/O,

1643.25 -> you've programmed the hardware to do,

1644.93 -> all the VTD functions and all the VTX functions,

1648.08 -> make the instance runable.

1649.85 -> And as soon as it says that, the instance starts running

1652.25 -> and eventually you get to SSH into it.

1658.67 -> But through the last few minutes of my talking,

1662.15 -> you could probably see how

1663.86 -> we've had multiple generations of Nitro Cards

1666.86 -> for networking, for storage,

1669.29 -> multiple Nitro Controllers actually.

1671.27 -> And that gives us this broad selection of instances

1675.5 -> for various workloads, right?

1677.09 -> Network bound workloads, I/O bound workloads,

1679.37 -> compute bound workloads across Intel, AMD, ARM,

1685.58 -> our Graviton CPUs, and now MacOS and Apple.

1691.43 -> We have instances that go from like half a gigabyte

1695.36 -> all the way up to 24 terabytes in size

1697.25 -> for the high memory SAP instances.

1699.77 -> We have instances that support,

1701.81 -> 25 gigabytes of networking

1703.43 -> all the way out to like 800 gigabits of,

1704.833 -> gigabit per second of networking.

1707.42 -> In fact, if you looked at the slide

1708.77 -> from like a couple of years ago,

1710.51 -> that 575 number was like 200, right?

1713.72 -> When we first launched Nitro.

1715.64 -> In fact, this slide did have things like Apple on there,

1718.28 -> it didn't have things like networking up to 1600 gigabits.

1722.21 -> It just shows you how quickly we're able to innovate

1724.73 -> because we have these building blocks.

1729.08 -> I won't go over the slide because like I said,

1731.24 -> all of our execs have talked about this in depth,

1733.7 -> there's probably awesome recordings of it.

1736.91 -> But I want to quickly touch upon the fact that we have,

1740.36 -> our sixth generation CMR instances,

1744.32 -> which support 80 gigabits per second of EBS bandwidth,

1747.14 -> 200 gigabit per second of networking.

1749.9 -> C7gn, we just announced it yesterday,

1752.72 -> it's our Graviton based 200 gigabit per second

1757.49 -> networking instance.

1760.73 -> R7iz is our first Intel Sapphire Rapids based instance

1766.67 -> that has, that is this high frequency 3.9 gigahertz part.

1770.99 -> It's 15% better performance

1773.24 -> compared to the previous generation and inf2 and Mac2.

1776.21 -> And I'm gonna dive into Mac2 a little bit

1778.97 -> 'cause I was the lead engineer for Mac2,

1780.37 -> so I'm super excited about that one.

1784.76 -> So now that I've given you kind of the building blocks,

1786.83 -> let's talk about security.

1790.97 -> Keeping our customers' workload secure and confidential

1795.02 -> is like our top priority in EC2.

1798.77 -> And we often get into these conversations with customers

1801.56 -> about how we keep things secure and so on.

1806.274 -> AWS is responsible,

1807.95 -> and security is like a shared responsibility.

1810.09 -> AWS is responsible for security of the infrastructure,

1812.69 -> everything I just talked about.

1814.43 -> The customer, you, are obviously responsible

1816.83 -> to make sure that the instance and it's OS is patched,

1820.1 -> make sure that you don't expose your instance to the world,

1822.23 -> you do all the security groups and everything else, right?

1824.27 -> That's something we can do.

1830.03 -> And before I go further, I should mention that,

1832.58 -> this talk is about Nitro.

1834.92 -> We take security very seriously,

1836.66 -> in fact, all of our Zen based instances,

1838.61 -> the C5, anything before C5,

1840.935 -> the C4s and the C3s are also very, very secure,

1844.49 -> they just use a different set of policies, right?

1848.12 -> In the past year,

1850.13 -> this phrase confidential computing has been coming up a lot

1853.37 -> in the industry and in customer conversations we have.

1857.51 -> We've observed that a lot of people also talk about

1859.52 -> different technologies that solve different problems,

1862.73 -> and this has led to a lot of confusion.

1864.98 -> But when we take a step back

1866.21 -> and we actually get into customer conversations

1868.13 -> about confidential computing,

1869.69 -> the first question we ask actually is,

1873.05 -> who are you protecting your data from, right?

1875.06 -> That is the key question.

1878.27 -> The most important dimension that customers tell us often

1882.5 -> every single time is we wanna protect our data and our code

1887.21 -> from in two scenarios; scenario number one,

1889.7 -> from other instances running on the same physical box.

1892.34 -> It's a multi-tenant box and from you,

1895.67 -> the cloud service provider, us, right?

1899.45 -> And we really think of that problem as dimension one.

1903.26 -> We wanna make sure that we don't ever see your data,

1906.02 -> we don't ever see your code,

1908.6 -> we just think of it as radioactive.

1910.19 -> We don't wanna touch it, we don't wanna see it.

1912.2 -> That's the whole point of the Nitro system.

1916.43 -> There's also a set of customers to talk to about,

1918.71 -> talk to us about a second dimension of security,

1922.37 -> and that is around making sure that their users

1925.94 -> who are actually on the instance

1928.49 -> can't get to highly sensitive data.

1930.38 -> So they want to divide their data into two parts,

1932.75 -> most trusted and less trusted.

1935.03 -> And they want to make sure that

1935.93 -> people who are able to SSH into the instance

1938.15 -> can't get to this highly sensitive data.

1939.83 -> They wanna partition their data into two separate classes.

1943.4 -> And this is where I'll talk about Nitro Enclaves

1945.44 -> and what Nitro Enclaves does and how it solves that problem.

1950.87 -> So I'll go over kind of dimension one and dimension two,

1954.38 -> so let's start with dimension one.

1957.89 -> From everything you've seen so far about the Nitro Cards

1960.5 -> and how we've had this secure hypervisor

1962.57 -> that's minimal and so on,

1964.46 -> the first thing that you notice is that

1966.41 -> there is physical separation

1968.18 -> between where you customer instances run on the host CPU

1971.63 -> and where our software runs on the Nitro Cards,

1974.36 -> it's physically separated.

1976.85 -> The second most important thing is,

1978.2 -> when the servers actually boot up,

1980.33 -> they go through a measured boot process.

1982.22 -> So the Nitro Controller, the brain of the machine comes up,

1986.27 -> starts from a secure place, from a secure ROM,

1989.54 -> and then starts measuring

1990.92 -> every single stage of the boot process,

1993.17 -> it's like a pretty standard measured boot.

1995.12 -> But the key thing is that all of our software

1997.67 -> lives on an SSD that's attached to the Nitro Controller,

2000.1 -> I've shown it that little picture

2001.24 -> next to the Nitro Controller.

2003.55 -> And only if the measurements match,

2006.73 -> the measurements are actually,

2007.84 -> so each stage measures the next stage

2010.45 -> and extends those measurements in a TPM,

2012.79 -> a Trusted Platform Module

2014.02 -> that's on the Nitro Controller.

2015.82 -> And at the end of the boot process,

2017.23 -> if the measurements match what we expect,

2020.17 -> the encryption key that encrypts the drive

2023.35 -> that has all of our software is provided to us

2026.707 -> and we can decrypt the drive

2028.12 -> and actually get to the software.

2029.86 -> If the measurements fail,

2030.91 -> which means somebody's tampered with the firmware,

2032.98 -> you can't unencrypt the drive.

2034.21 -> And if you can't decrypt the drive,

2035.77 -> there is no way you can launch an instance on this machine,

2038.35 -> it's just a dead brick at that point.

2041.62 -> So by ensuring that the software on,

2044.32 -> the firmware on the machine is exactly what we expect

2047.77 -> and that the drive is encrypted

2049.39 -> and all our software is careful,

2050.92 -> we can ensure that the machine is in exactly in the state

2053.59 -> before it is able to launch any customer instances.

2057.16 -> I talked about the Nitro Security Chip,

2060.25 -> I mentioned that the Nitro security chip sits on every bus

2064.45 -> and makes sure that the customer instance

2066.19 -> can never ever ride to any spy flash on the platform.

2070.72 -> But the second thing it does

2072.25 -> is that when we actually reboot the machine,

2075.55 -> it comes up and holds the CPU reset, measures everything,

2078.61 -> and then says if everything is okay,

2079.84 -> I'm gonna release the CPU from reset.

2081.73 -> If things fail, it's just a brick.

2087.82 -> Secure boot is so important to us

2089.8 -> that we actually worked with some folks in academia,

2094.24 -> some of my colleagues worked with folks in academia

2097.09 -> to come up with a paper that actually formally verifies

2100.18 -> that the secure boot process

2101.35 -> that we use on these servers is correct.

2104.38 -> And that's this paper I wanted to link

2105.79 -> in case you wanna read offline.

2111.43 -> Continuing down dimension one,

2113.65 -> now you must be wondering, I talked about software,

2115.75 -> I talked about this drive and I talked about

2117.91 -> software being encrypted on this drive and so on.

2120.55 -> Well that software

2121.66 -> is software that folks like me write, right?

2123.55 -> So we write software, it's obviously reviewed extensively,

2127.24 -> we go through multi-party reviews every single change.

2130.42 -> The software also goes through our build pipeline

2132.4 -> and our build pipeline signs that software with a key

2135.61 -> that is private to the build pipeline.

2137.05 -> Like I as an individual, no matter how senior you are,

2142 -> I cannot for example, make a change

2144.82 -> and say I'm gonna target a production server,

2147.58 -> I'm gonna build that software

2148.9 -> and push it to that production server and deploy it.

2150.73 -> It just cannot happen because when that software

2153.61 -> is deployed to that production server,

2155.35 -> its signature is verified

2156.79 -> and I just don't have the private key

2158.26 -> to sign it with the production key.

2161.08 -> So when that software is actually deployed to a machine

2163.9 -> and when that software package is extracted,

2165.7 -> the signature verification fails

2167.02 -> and it'll reject my package.

2168.64 -> So the only way to get software deployed to machines

2171.37 -> is if it's built by our deployment pipeline,

2173.44 -> and that deployment pipeline holds a key that is private

2176.38 -> to deployment pipeline.

2179.26 -> The other thing that we do almost on a weekly basis

2183.22 -> is we do something called live update.

2185.98 -> So let's say we find that we want to,

2187.93 -> push a new feature to our customers

2190 -> or we want to push a security fix or a bug fix,

2194.29 -> we can actually update

2195.73 -> most of the software components I mentioned,

2197.2 -> networking, storage, hypervisor, et cetera,

2200.47 -> and push it and update our Nitro Controllers

2203.8 -> or our Nitro Cards while customer instances are running.

2206.98 -> So it's like we're basically updating software

2208.66 -> from underneath you

2210.04 -> and it does not impact customer instances.

2211.72 -> You keep running, you keep doing storage,

2213.64 -> you keep doing networking.

2216.04 -> This gives us a really cool way of updating the software

2218.5 -> and making sure everything is okay.

2222.64 -> I also mentioned on a lot of the Nitro Cards

2225.85 -> that I talked about, I mentioned we do encryption,

2228.01 -> we do encryption of storage at rest,

2229.78 -> we do encryption or networking and so on.

2232.12 -> All the keys for that encryption is stored in the TPM

2235.96 -> on those Nitro Cards so it's secure storage

2238.18 -> and it cannot leave,

2239.35 -> it's not like we write it to the drive

2240.61 -> and it just sits there.

2242.68 -> Finally, as a defense in depth, a lot of our instances,

2246.37 -> especially our more recent instances,

2248.83 -> support things like total memory encryption

2253.989 -> or Graviton has encryption on by default.

2257.65 -> This prevents, this protects you from physical attacks

2260.44 -> like a DCO attack or a technician walking up to the server

2263.56 -> and pulling out a DIM that has your memory contents, right?

2266.5 -> So the memory actually is encrypted at rest in the DIM.

2273.94 -> This is very important to us.

2277.48 -> The Nitro architecture is built such that I,

2281.14 -> it doesn't matter who you are,

2282.31 -> it doesn't matter who your seniority is,

2284.02 -> even if you're a VP, you cannot SSH to a physical box,

2288.61 -> it just doesn't exist.

2290.14 -> This was a huge security bar raiser for us

2292.9 -> compared to our Zen instances.

2294.07 -> On Zen instances, there is something called a DOM zero,

2296.74 -> which is a special DOM.

2298.24 -> You can actually SSH into DOM zero and do things.

2301.57 -> You look at logs,

2302.47 -> you can look at debug messages and so on.

2305.32 -> That just doesn't exist on Nitro.

2307.39 -> The Nitro Controller exposes a hardened API.

2311.83 -> It's REST API that the EC2 control plane calls

2315.55 -> and sort of passes in parameters.

2318.28 -> The Nitro controller validates the parameters

2319.9 -> and returns the response.

2321.76 -> And the API is actually encrypted,

2324.37 -> so there's actually a TLS session

2325.69 -> between the EC2 control plane and the Nitro Controller.

2327.72 -> So it's an encrypted transaction, it's authenticated,

2331.54 -> there's actually tokens in that API that says yes,

2334.09 -> you are allowed to talk to the Nitro Controller.

2336.94 -> And even I can't walk up to a production box and say,

2340.72 -> oh, do something funky with over the API just to,

2343.06 -> I don't have the tokens.

2345.22 -> It's authorized and everything's logged on the,

2347.77 -> so all the interactions that happen with the,

2350.11 -> between the EC2 control plane

2351.7 -> and the Nitro Controller are actually audited and logged.

2354.58 -> So we can go back and look and see what happened.

2358.27 -> And we use the principle of least privilege.

2360.1 -> So for example, the EC2 control plane that I show here

2362.83 -> is actually really the control plane

2365.47 -> that launches instances, there is a storage control plane,

2367.75 -> there is a networking control plane,

2368.86 -> so really three components.

2370.45 -> Each of them has a restricted set of API calls

2373.66 -> that they can make and that's it,

2375.55 -> they can't do anything more.

2377.89 -> Lastly, none of these APIs,

2380.02 -> and we go through this like every so often,

2382.24 -> we go look at these APIs over and over,

2384.91 -> none of these APIs

2386.17 -> ever expose customer memory or customer content.

2389.5 -> Just they don't, they just don't expose it.

2394.36 -> The second dimension, so with the first dimension,

2396.94 -> the things I just talked about,

2399.7 -> we're basically saying there is no way for an operator

2402.34 -> or a service an operator like myself or a service

2405.61 -> to actually get customer content.

2407.32 -> You can't SSH to a box, you can't just do weird things.

2410.68 -> The software that's running on that is trusted

2412.24 -> 'cause it's built by our system.

2414.25 -> The firmware is trusted 'cause it's measured and so on.

2417.61 -> The second dimension is Nitro Enclaves.

2421.42 -> An enclave is basically a isolated,

2424.63 -> hardened virtual machine.

2427.24 -> In fact, it uses the same underlying VMM technology

2431.11 -> that we use for virtual machines.

2432.97 -> Think of it this way:

2434.74 -> imagine you just launched a C5 8xlarge instance with,

2439.45 -> I'm just gonna make up a number,

2441.46 -> 16 VCPUs and 32 gigabytes of memory.

2444.82 -> And then you SSH into that instance and said,

2447.407 -> "Carve out four VCPUs and eight gigs of memory

2450.01 -> and create an enclave from it."

2452.5 -> We launch another virtual machine

2454 -> next to the main parent virtual machine

2456.73 -> with four VCPUs and eight gigabytes of memory,

2460 -> and that thing boots into a Linux like environment,

2462.04 -> but it's very, very limited.

2464.26 -> And that instance uses the same virtualization technology

2467.92 -> as any of our other instances

2470.83 -> uses VTX and VTD for isolation.

2473.35 -> But more importantly, it has no storage,

2475.81 -> it has no networking.

2477.55 -> The only thing you can talk to

2478.84 -> is it has thin straw between itself and the parent instance,

2482.62 -> it's called a VSOC.

2485.11 -> Why is this useful?

2486.13 -> Well, that instance when it comes up,

2489.49 -> it actually makes up a public private key pair.

2491.77 -> And the private key pair,

2492.85 -> the private key in that key pair stays in the enclave,

2495.28 -> it never leaves the enclave.

2496.93 -> The public key is actually exported

2498.28 -> as part of an attestation report.

2499.9 -> So now you can do things like, I can prove that the enclave

2503.32 -> is actually what you think it is.

2505.03 -> There is an attestation report with measurements

2507.28 -> and you can actually say

2509.38 -> these are exactly the measurements I expect

2511.09 -> because I know exactly what the enclave has in it

2513.22 -> in terms of an application,

2514.69 -> and now you can do secure things inside the enclave.

2518.41 -> So let me give you an example.

2520.54 -> Here's an example of

2521.8 -> a customer doing multi-party collaboration.

2523.84 -> So imagine a credit card company and a retailer

2528.46 -> and they wanna basically build

2529.66 -> some kind of analytics, right?

2530.95 -> So they have customers,

2532.12 -> they wanna look at credit card transactions

2533.71 -> and see what people are purchasing and so on and so forth,

2535.9 -> but they don't wanna share that data with each other,

2538.12 -> they wanna keep the data completely separate.

2540.04 -> So what happens is they both will write applications

2542.77 -> that will eventually go into an enclave,

2544.84 -> they will build the enclave image with the applications,

2547.18 -> they both will review the application

2548.71 -> and agree that it's okay, it's safe,

2550.96 -> and that they're not kind of stomping on each other.

2553.54 -> And then once they build the application,

2554.98 -> they get a set of measurements

2556.57 -> and that is the expected measurements.

2558.43 -> So when the enclave launches,

2560.08 -> it gets a secure hash measurements out,

2562.06 -> they can verify it's the same measurement.

2563.98 -> Now each one of them can encrypt the secret data

2568.12 -> that they want to send inside the enclave

2569.74 -> with the public key's enclave.

2571.63 -> They both can send their all individual data,

2574.12 -> that data as it goes to the parent instance

2576.13 -> and then eventually into the enclave,

2577.42 -> can only be decrypted by the enclave

2579.1 -> 'cause nobody else has the private key.

2581.35 -> And once the data is decrypted,

2582.61 -> it can be processed by the application in the enclave

2584.89 -> and the results can come out.

2590.08 -> I'll touch upon the next two really quickly.

2593.35 -> We also implemented, I think we announced this last year,

2595.96 -> we also implemented UEFI Secure Boot.

2597.79 -> I'm sure you guys know what UEFI Secure Boot is.

2600.94 -> If you're running on premises,

2602.65 -> more recently customers have had,

2604.99 -> operating systems that actually come up.

2606.94 -> There's a secure boot chain, it actually measures,

2610 -> it actually verifies

2611.5 -> the signature of each stage in the boot loader

2613.87 -> and grab like operating system boot loader and so on.

2618.58 -> And if everything's okay,

2619.57 -> that actually boots into the operating system and so on.

2622.69 -> So the reason we implemented this

2624.49 -> was a lot of customers said,

2625.397 -> "Hey, I've got UEFI Secure Boot on premises.

2628.12 -> How do I bring my that capability to EC2?"

2632.86 -> Same thing with Nitro TPM,

2634.313 -> a TPM is a trusted platform module.

2636.4 -> We have Nitro, we have TPMs on Nitro Controllers

2638.8 -> that I mentioned.

2639.85 -> But for instances, when you launch an instance,

2642.82 -> we actually expose a TPM device.

2644.62 -> And so with that TPM device,

2645.79 -> you can do things like extend measurements

2648.07 -> as part of your boot process,

2649.69 -> then you can tie to things like,

2651.01 -> you can actually encrypt your EBS volumes

2652.87 -> in addition to the encryption we do,

2654.52 -> you can use things like BitLocker

2656.08 -> or LUKS Encryption if you're using Linux,

2660.13 -> and you can tie it to the TPM's measurements.

2664.09 -> I also wanna share this paper actually published last week

2666.28 -> if you look at the date on it,

2667.78 -> it talks about all kinds of details about stuff I mentioned,

2671.02 -> and this is actually a wonderful read.

2675.55 -> The last thing I wanna cover in the last 15 or so minutes

2678.97 -> is EC2 Bare Metal instances.

2684.49 -> So the reason I wanna cover this topic is,

2688.24 -> I am in customer conversations often where

2690.94 -> customers ask me questions like,

2693.07 -> should I use Nitro or should I use Bare Metal?

2695.8 -> And I wanna clear that up.

2698.29 -> So the whole reason we built EC2 Bare Metal

2702.1 -> is back in 2016, VMware came to us and said,

2707.237 -> "Hey, we've got customers On Prem, they use VMware,

2709.63 -> they use vSphere and NSX and and VSAN and so on,

2714.4 -> and they have tooling around,

2716.02 -> all of the control pane APIs that VMware exposes

2719.23 -> and all the operational muscle that they've built,

2721.57 -> but they don't wanna manage their infrastructure."

2723.34 -> So what we wanna do is we wanna have those customers

2725.8 -> move to a public cloud like AWS,

2727.99 -> but we don't want have to have to,

2730.57 -> have them relearn EC2 APIs and do all of that stuff.

2735.13 -> So we sat down with them

2736.09 -> and they basically laid out a vision

2738.16 -> for a managed software defined data center as a service.

2743.95 -> Now, if you've been following the Nitro story

2745.96 -> I've been telling you so far,

2747.55 -> notice how we offloaded pretty much everything

2749.38 -> to the Nitro card?

2750.52 -> We got to the point where we said,

2751.847 -> "Do we even need a hypervisor?"

2753.82 -> The only thing that the hypervisor does

2755.62 -> is it carves up memory CPUs

2758.41 -> and basically protects the underlying platform

2760.54 -> from writes from instances, right?

2762.427 -> But we have a Nitro Security chip for that,

2764.26 -> we don't need a hypervisor,

2765.52 -> so let's get rid of the hypervisor.

2768.01 -> And that's what Bare Metal is,

2770.116 -> it's exactly the same thing as a virtualized instance

2772.78 -> without the hypervisor

2773.613 -> and I'll show you what that means in a minute.

2775.27 -> But there are lots of use cases

2776.44 -> that came out of this, right?

2777.85 -> The one I mentioned where

2779.89 -> you don't have to use our hypervisor if you don't,

2782.5 -> you can launch an EC2 Bare Metal instance

2784.72 -> and bring your own hypervisor like ESXI.

2787.87 -> Customers also use it for clear containers if you wanna use,

2790.45 -> if you wanna build containers that are more secure

2793.33 -> than basic containers.

2795.28 -> But the one that I like the most is the third bullet.

2799.15 -> Even if you launch a c524xlarge or a c6i32x large,

2805.21 -> the full instance, the full slaughter, right?

2808.96 -> You will not get exposed

2810.88 -> or the hypervisor will not expose

2812.98 -> all the performance monitoring counters for you,

2814.75 -> there are some that are kind of not fully exposed.

2817.72 -> And if you're doing things like

2819.07 -> performance measurements of your applications

2820.84 -> run near instance,

2821.95 -> you need access to performance monitoring counters on Intel,

2824.32 -> for example, on the Intel CPU.

2825.917 -> There are three types of counters, software counters,

2828.4 -> hardware counters and memory counters.

2830.35 -> Things like TLB misses and page cache misses and,

2834.97 -> what are my threads doing in terms of hotspots and,

2840.07 -> and cycles that I'm spending on things, et cetera.

2844.06 -> Well, when you're running on Bare Metal,

2845.41 -> you are running on Bare Metal,

2846.58 -> you are literally running on the CPU,

2848.32 -> there is no hypervisor

2849.19 -> so you get exposed to all the instances.

2851.26 -> And actually Intel has a really nice paper on this

2853.57 -> where they talk about

2854.86 -> virtualized instances with Bare Metal.

2856.39 -> But just to put it in pictures,

2859.27 -> this is what happens when you launch a C6I4xlarge instance.

2863.11 -> It is one of, in this case three instances

2865.54 -> on the physical box.

2867.07 -> You guys have already seen this picture,

2868.54 -> the Nitro Controller, the Nitro Cards,

2870.85 -> and you happen to be one of the instances,

2872.26 -> let's say Nitro instance one,

2875.53 -> I can also answer C6i32xlarge,

2877.39 -> which is the full machine, right?

2879.43 -> I'm occupying all these CPUs,

2881.44 -> the hypervisor doesn't use anything

2883.87 -> because it's just quiescent, it's just sitting there.

2886.36 -> But roughly the picture looks the same as the 4xlarge,

2889.09 -> except that it's all CPUs and all memory.

2892.21 -> What happens when you launch a Bare Metal

2894.1 -> is the hypervisor disappears,

2896.26 -> it's just that.

2897.58 -> Now if you think about what I said

2898.69 -> about how an instance is launched,

2900.19 -> that's exactly what happens here.

2901.387 -> The EC2 control pane is talking to the Nitro Controller,

2904.309 -> the Nitro Controller at some point says, turn on the CPU,

2908.1 -> I'm exposing EBSs and NVMe controller to BIOS,

2910.84 -> BIOS comes up and says, I need to boot from something,

2913.33 -> there is no SSD connected to the platform,

2915.64 -> the only choice I have is that NVMe controller I see,

2918.37 -> and it boots your AMI.

2919.93 -> That's exactly what happens when you,

2921.73 -> and if you notice, if I go back one step,

2927.25 -> notice the specs on the 32xlarge, 128 VCPUs,

2931.014 -> 256 gigabytes of memory, 50 gigabits of network,

2934.27 -> 40 gigabits of EBS, that's exactly what metal is.

2937.9 -> So it's exactly the spin specs,

2939.76 -> in fact, it's priced exactly the same for that reason.

2942.79 -> The only thing is you have to pick your use case

2944.59 -> and why you want it.

2947.11 -> So just as an example, I actually launched one last Friday,

2950.56 -> this is our sixth generation Ice Lake based metal instance.

2954.76 -> You can see I launched a c6i metal from the console,

2957.76 -> that's my instance ID and that's my public IP.

2960.88 -> I SSHed into it, I booted Amazon Linux.

2966.56 -> I ran an IMDs query, it's a c61 metal, there's 128 VCPus,

2970.78 -> roughly 256 kilobytes of memory.

2973.6 -> The one thing is

2974.433 -> when we launch virtualized instances with our hypervisor,

2976.66 -> we only expose a handful of devices, PCI devices, right?

2980.23 -> But when you're on Bare Metal, you see everything.

2982.9 -> So there's a bazillion devices,

2984.34 -> you can see all kinds of Intel devices.

2987.25 -> The three that are most important are,

2989.32 -> the 80 to 50 that we emulate,

2992.29 -> the EBS 0061 device and the ENA

2995.95 -> which is what you booted from,

2997.27 -> and that's your primary network interface.

3000.42 -> Lastly, EBS is exposed as NVMe,

3003.15 -> so I just, I installed the NVMe CLI

3006.33 -> and then I ran the identify controller command,

3008.55 -> and you can see it says it's my EBS volume

3011.22 -> and I can do LS block and C, so it looks pretty standard.

3016.2 -> A lot of times customers come to us and say,

3017.587 -> "Should I use virtualized 32xlarge

3019.766 -> or should I use a Bare Metal?"

3021.09 -> Well, it depends.

3024.84 -> 32xlarge is awesome.

3026.4 -> Like I said, our hypervisor is super thin,

3028.71 -> the performance of the hypervisor is practically Bare Metal.

3031.53 -> In fact, I pulled this picture off of a blog from 2018

3035.31 -> that compares four HPC workloads between C5 and C5N.

3040.077 -> C5N is our high networking instance, and,

3045.24 -> sorry, compared c5n18xlarge with our C5N Metal

3050.1 -> to show the overhead of the hypervisor.

3052.29 -> The gray bars are normalized, that's what,

3056.503 -> and the purple bars are what is,

3058.41 -> what happens when you run on the C5N18Xlarge.

3061.26 -> You can see that the difference is roughly 1%

3063.75 -> between the hypervisor based instance

3065.79 -> and the metal instance.

3067.95 -> And at the bottom I have this chart that shows

3070.05 -> kind of the specs of the c5n18Xlarge and the C5N metal,

3074.13 -> they're virtually the same.

3076.2 -> In terms of like the actual launch of an instance,

3078.93 -> it's exactly the same APIs,

3080.4 -> it's exactly the same AMIs,

3081.78 -> it's exactly the same instance lifecycle.

3084.33 -> What I mean by that is

3085.65 -> if you SSH into your Bare Metal Instance

3086.633 -> and say sudo shut down dash -h now,

3090.42 -> it's gonna behave exactly like you expect,

3092.13 -> like a virtualized instance, It's gonna shut down.

3094.5 -> If you say sudo reboot, it's gonna reboot and come back up.

3097.68 -> The way you stop your instance

3098.97 -> from the EC2 control plane is exactly the same.

3101.88 -> The SLA is a security bar because it's all running on Nitro,

3104.91 -> it's exactly the same.

3106.86 -> And you get elasticity so you can actually launch

3109.5 -> EC2 Bare Metal instances up

3111.57 -> and actually terminate instances and go back down,

3113.55 -> and it's completely on demand.

3115.65 -> It's not like you're reserving

3116.67 -> a machine for yourself somewhere.

3119.34 -> So which one should you use?

3121.29 -> Well, it depends on your use case.

3122.85 -> If your use case,

3123.87 -> in fact, a lot of times you tell customers,

3125.1 -> if you're asking us this question,

3126.54 -> chances are you don't need a metal,

3128.22 -> you're fine with virtualized.

3130.05 -> If your use case is one where

3131.91 -> you want to bring your own hypervisor

3133.23 -> or you want to do some deep performance analysis

3135.33 -> like I said, you probably want Bare Metal.

3139.32 -> One of the pain points with Bare Metal,

3141.24 -> in case you haven't caught it, is when you actually say,

3145.157 -> AWS space EC2 space instance type dash C5.metal for example,

3152.13 -> is it used to take like six to eight minutes

3154.41 -> for that instance to come up

3155.58 -> to where you can SSH into that box.

3158.1 -> And this was a huge pain point for our customers

3160.41 -> because they're like,

3161.243 -> "Well, when I launch a virtualized instance,

3163.26 -> it comes up in 40 seconds.

3164.76 -> How come your Bare Metal stuff takes so long?"

3166.56 -> Well, we worked on it

3168.06 -> and because we have a Nitro Controller

3170.25 -> and because we have system BIOS

3171.66 -> or the BIOS on platform that is owned by us,

3174.96 -> we actually implemented a protocol

3176.28 -> between the Nitro controller and the BIOS, that thing.

3180.78 -> And so what happens is,

3182.13 -> now if you think about BIOSes, right?

3183.69 -> System BIOSes has come up and they initialize the platform,

3187.47 -> they initialize the memory DIMs,

3189 -> they initialize PCIe,

3190.47 -> then they look for a Bootable device,

3191.91 -> they look for any kind of disc to boot from.

3194.31 -> Well what we told BIOS is, hold on,

3196.33 -> there is no device at the moment.

3198.27 -> When I get an instance launch

3199.5 -> and an EBS volume gets attached, I will release you,

3201.87 -> then you go scan for the PCI bus and look for an EBS volume.

3205.38 -> And so that's exactly what that interface does,

3207.27 -> it's able to hold BIOS at a place

3209.13 -> where it's just ready to go,

3211.35 -> and as soon as we attach that EBS volume, we say go.

3214.2 -> And so that effectively reduces the launch time

3217.11 -> of an EC2 Bare Metal instance to like 40 seconds.

3221.19 -> I, my C6i metal instance that I launched, it came up in,

3224.31 -> yeah, it came up in 40 seconds,

3225.57 -> so it was a 90% reduction in launch time.

3229.44 -> This was a fun one;

3230.85 -> in 2019, a lot of customers were coming to us saying,

3234.247 -> "You guys have Linux and you guys have Windows,

3236.73 -> you guys had Linux and Windows forever.

3238.98 -> When are you gonna bring MacOS to the cloud?"

3241.56 -> And the challenge we had was,

3242.91 -> in order to bring MacOS to the cloud,

3244.98 -> MacOS will only run on Apple branded hardware.

3248.79 -> So the challenge was,

3249.623 -> how do you bring Apple branded hardware into the cloud?

3251.19 -> Well, we have Nitro and I was the lead engineer,

3255.42 -> I actually built this thing.

3257.25 -> We took a standard Apple Mac Mini, it's not emulated,

3261.36 -> it's physically, I drove to the Apple store,

3263.67 -> bought a Mac Mini and brought it a lab.

3266.1 -> We actually connected the Nitro controller over Thunderbolt,

3269.91 -> we connected the HDMI port and the USB Type-C port

3272.76 -> to emulate keyboard, video and mouse.

3275.46 -> And that's what it looks like.

3278.73 -> From an instance launch perspective, same API.

3281.61 -> You just say dash dash instance type Mac one dot metal

3285.72 -> and you're a Bare Metal customer running on the Mac mini,

3287.73 -> physically on the Mac Mini.

3289.41 -> There were a ton of challenges,

3290.76 -> which I won't talk about right now,

3292.95 -> but we've been in production for about 20 months

3295.62 -> and lots and lots of customers.

3303.15 -> This is a chart that I won't go over completely,

3305.31 -> but the point of the chart is that in terms of elasticity,

3309.21 -> scalability, flexibility, security,

3312 -> you should think about EC2 Bare Metal instances

3313.98 -> exactly the same as virtualized.

3316.2 -> The only differences between virtualized and Bare Metal

3318.24 -> are those asterisks.

3319.83 -> You can obviously scale horizontally

3321.36 -> 'cause you can launch multiple Bare Metal instances.

3323.79 -> But there aren't multiple sizes of Bare Metal instance,

3327.99 -> it's a full slot,

3328.89 -> it's a full machine that you're getting, right?

3331.11 -> And then there is no hypervisor.

3332.73 -> That's the only difference

3333.66 -> between virtualized and Bare Metal.

3335.61 -> And if you compare that to sort of legacy Bare Metal,

3337.53 -> and this is where the confusion comes in,

3338.79 -> when we talk to customers about Bare Metal,

3340.68 -> customers automatically imagine

3342.69 -> that they're gonna send us a ticket

3344.13 -> and we're gonna reserve some machines for them

3345.75 -> in some lab somewhere and give them a private IP address.

3348.66 -> No, it's the same API, you launch a machine,

3351.21 -> you terminate a machine a second later,

3352.71 -> you get billed for one second, but it's Bare Metal.

3356.94 -> And so you don't have to like plan and procure and

3359.58 -> you don't have to sort of compromise on the elasticity

3363.27 -> that EC2 brings.

3368.13 -> Finally, my last slide; related breakouts.

3370.44 -> There are actually some really cool sessions on

3373.14 -> things related to Nitro and the silicon innovations

3377.46 -> where they'll talk about

3378.293 -> Inferentia and Graviton and Trainium.

3381.69 -> I think you should go listen to some of these things.

3384.51 -> So that's it.

3386.497 -> (audience claps)

Source: https://www.youtube.com/watch?v=jAaqfeyvvSE