AWS re:Invent 2021 - How Netflix is using IPv6 to enable hyperscale networking

Aug 16, 2023

AWS re:Invent 2021 - How Netflix is using IPv6 to enable hyperscale networking

Netflix continues to grow at an astonishing rate. With more subscribers and productions than ever, the network must continue to scale to connect it all. In this session, follow the Cloud Networking team on a journey to understand the scaling challenges that Netflix faces with IPv4 and the drivers for IPv6. Explore design patterns and architectural flaws that fundamentally shape the design of most VPC networks. Get insight into the close collaboration and co-innovation between Netflix and AWS that drove new features like prefix delegation and the tight feedback loop that takes features from unimpressive to amazing. Appreciate the simplicity that IPv6 can provide and gain motivation for your own journey.

Learn more about re:Invent 2021 at https://bit.ly/3IvOLtK

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#AWS #AmazonWebServices #CloudComputing

Content

2.06 -> I told my tech friend back there

3.27 -> I'd give him finger guns when I was ready to go,

5.12 -> so I'm ready to go now.

6.75 -> How's everybody doing?

8.15 -> I haven't been on a re:Invent stage in like two years,

10.29 -> and I'm super excited to see everyone in person.

12.36 -> It's fantastic.

14.33 -> We have some people trickling in,

15.47 -> but I think we can get started.

17.8 -> Welcome to how Netflix is using IPv6

20.68 -> to enable hyperscale networking.

23.59 -> My name is Rob Hilton.

24.423 -> I'm a Principal Solutions Architect with AWS,

26.56 -> and very fortunately for all of you,

28.27 -> you won't have to hear a whole lot from me today.

30.73 -> I'm joined on stage no less than six feet to my right

34.5 -> by Donavan Fritz, who is a Senior Networking SRE,

37.8 -> and a brilliant one at that,

39.22 -> who's helped us push the limits of cloud networking today,

42.31 -> and who's also helping pioneer some of the stuff

44.2 -> that we're working toward

45.05 -> building into for the next decade, or so.

48.18 -> Some of you may know that Netflix started

49.97 -> their cloud journey in about 2008,

52.44 -> and at the scale of their business,

54.1 -> I'm sure you can imagine how complex

55.71 -> and nuanced maintaining an always-on, resilient network

58.47 -> in that type of state would be,

60.15 -> and you would be correct.

61.75 -> In this talk, Donavan is gonna walk us through

64.03 -> some of the things that got Netflix to where they are today,

66.33 -> and some of the things that they're doing

67.56 -> to work toward what they need to be,

69.23 -> where they need to be tomorrow.

71.23 -> Ladies and gentlemen,

72.59 -> Donavan Fritz. All right, oh.

74.169 -> (audience applauds)

75.81 -> All right, thank you.

76.643 -> Oh, is it on?

78.13 -> Thanks, Rob, for that generous introduction.

79.89 -> Again, my name is Donavan Fritz.

82.26 -> I do cloud networking at Netflix,

83.69 -> and today, we're gonna be talking about

84.97 -> how we are at Netflix using IPv6

87.75 -> to enable hyperscale networking.

92.34 -> There we go.

96.66 -> Perfect, okay, so we're gonna start off a quick agenda

99.62 -> before we get started.

100.79 -> We're gonna be spending the majority of the time

102.68 -> talking about why we're doing this,

104.97 -> why IPv6, and just to give a little sneak peek there,

108.65 -> it's because the business really needs it now.

111.58 -> So after why, we're gonna discuss the co-innovation

113.92 -> between Netflix and AWS in this space.

115.923 -> I wanna talk a little bit about our progress

117.72 -> and where we are today with using IPv6 in the cloud.

120.88 -> I'll share some of my lessons learned and best practices,

123.36 -> and then we'll wrap up by communicating

126.93 -> how y'all can get started using IPv6 in the cloud,

130.16 -> and what I think is most important,

132.21 -> how to show that this is worthwhile to your business.

135.93 -> So as Rob mentioned,

136.763 -> we've been in the cloud for quite a long time.

138.88 -> And so, our story today starts at that time

141.44 -> over 10 years ago when Netflix started

144.3 -> operating inside the cloud.

145.21 -> We moved out of the data center and into AWS.

148.19 -> And the landscape that AWS had at the time

150.69 -> was very different from what exists today.

152.217 -> There was no concept of VPCs.

154.76 -> Internal load balancers didn't exist.

157.58 -> And so, as innovators in this space,

159.06 -> we had to build some foundations to enable us to work,

162.39 -> because this hadn't really been done before.

164.47 -> So we had to solve the service discovery problem.

166.39 -> How does one service find another service?

168.9 -> After service discovery,

170.08 -> we had to solve the load balancing problem.

171.98 -> How do you load balance between services?

174.67 -> So we have our own internal service discovery tool

177.13 -> that's open source on GitHub called Eureka,

179.15 -> and then we'd use client-side load balancing heavily,

182.8 -> again, another open source project called Ribbon

184.7 -> on our GitHub page.

186.49 -> So around the 2012 and 2013 timeframe,

189.26 -> in addition to us-east-1, we started using

191.45 -> two additional regions, us-west-2 and eu-west-1.

195 -> And we did this for a variety of reasons.

197.15 -> The main benefits by going into this

199.41 -> active-active-active model was that

201.31 -> it gave us better redundancy, better resiliency,

203.88 -> and in most cases, lower latency for our customers.

208.47 -> And what this enabled us to do was provide

210.5 -> a better service to our customers.

212.24 -> We started winning more of what we call

213.64 -> these moments of truth.

215.1 -> And as our customers continued to enjoy Netflix,

218.5 -> and continued to use our service, we grew.

221.23 -> And again, this is still in EC2-Classic.

223.61 -> The network mode of EC2-Classic is very simple

227.43 -> and very basic.

228.7 -> So if you haven't used EC2-Classic before,

231 -> the general concept there is that

232.87 -> it's a flat network shared by all customers.

235.42 -> So every AWS customer gets a private IPv4 address.

239.25 -> Every customer gets a public IPv4 address,

241.71 -> and the network is simply transport,

244.3 -> and policy is laid on top with AWS security groups.

248.45 -> And then around the 2016 timeframe,

250.73 -> we moved our entire service out of VPC,

253.52 -> and into, or I'm sorry, out of EC2-Classic and into VPC.

257.63 -> And when we made this transition,

259.62 -> in order to make this as easy as possible for our services

262.11 -> that depend on the network,

263.6 -> we built VPC to really mirror how EC2-Classic was built.

268.19 -> It's a flat network.

269.76 -> So all of our VPCs have non-overlapping IP space.

272.66 -> They're all peered together.

273.63 -> We enable routing between all of our VPCs.

276.08 -> We don't use any network segmentation.

279.06 -> The place in the network where services are deployed

283.37 -> doesn't really matter to us.

285.81 -> Again, we think of the network as just transport,

288.38 -> and we layer policy on top with security groups

290.83 -> to allow, or disallow communication.

294.23 -> So in order to accomplish this, again, like I mentioned,

295.9 -> we have to have non-overlapping IP space.

298.25 -> We do this globally across all of our three regions.

300.61 -> So in order to allow a globally routable network,

306.33 -> we have our own backbone.

308.01 -> So we use Direct Connect to connect

309.49 -> to each one of our regions,

311.58 -> and then we have non-overlapping IP space

313.07 -> across three regions, and we can,

315.07 -> we provide a network to our company,

316.75 -> where any node can communicate to any other node,

319.36 -> barring policy, like security groups.

322.03 -> And so, we maintain this model of non-overlapping IP space

325.28 -> not only in the cloud, but also on-premise as well,

327.51 -> so our colo facilities, our offices, what have you.

333.303 -> In the past four to five years,

334.52 -> we've been seeing an uptick in containers.

336.77 -> And so, now we don't think of our microservices

339.47 -> within the cloud as a collection of EC2 instances,

342.19 -> we think of them as a collection of nodes,

344.01 -> where a node could either be a container,

345.75 -> or an EC2 instance.

350.31 -> Again, to make this transition as easy as possible,

353.12 -> we focus on making the network contract

355 -> between a container and an EC2 VM the same.

359.41 -> And so, we use Titus as our container infrastructure

362.52 -> and management platform.

364.11 -> Titus is, again, open sourced on our GitHub page,

367.42 -> and what Titus allows us to do is it allows us to have

371.13 -> that same network contract between containers and EC2 VMs.

375.42 -> And again, this allows us to carry over a lot of things

378.03 -> that we use today.

379.48 -> So I mentioned client-side load balancing earlier.

381.85 -> We can continue to use our,

384.03 -> those existing techniques pretty transparently

386.11 -> as we move workloads from EC2 instances over to containers.

390.88 -> Now, Titus itself is operating on top of EC2.

393.57 -> So that orange box we see there itself is an EC2 instance.

398.27 -> And so, in order for Titus to allow containers

401.33 -> to have VPC IP addresses, there's a lot of acrobatics around

404.67 -> how many ENIs we can attach to an EC2 instance,

408.03 -> how many IPs we can put on those ENIs.

411.9 -> And it's a really big problem for us.

413.741 -> These aren't soft limits, either.

416.04 -> There's really hard limits,

417.11 -> and we can't just call up our AWS,

421.96 -> and ask for these limits to be increased,

424.04 -> because they're hard, and they're not very flexible.

429.02 -> Okay, so let's summarize what we've talked about so far.

431.48 -> First, the flat network,

432.96 -> and this is a carryover from EC2-Classic.

435.45 -> We built VPC to mirror that.

437.87 -> And one of the reasons why we did that is

439.47 -> because we use client-side load balancing.

441.6 -> And just to be clear, we actually like

443.24 -> client-side load balancing quite a bit,

446 -> and we've put a lot of business logic in there.

450.95 -> Client-side load balancing is really fundamental

453.22 -> to how we do canaries

454.33 -> and custom request routing, for example.

457.65 -> We also talked about containers.

458.97 -> We do an IP address per container,

460.77 -> and this is, again, to maintain the same network posture

462.81 -> as we do with EC2 instances.

465.56 -> So that's where we've been in the network,

467.31 -> and we also wanna think about where we're going

468.97 -> as a networking team.

470.47 -> So what we see is we're continuing to grow.

472.79 -> We're winning these moments of truth with our customers,

475.18 -> where they're choosing to spend their time with Netflix.

477.65 -> What this means for us is we need to be able

479.23 -> to accommodate more accounts,

480.87 -> more VPCs in our infrastructure.

484.32 -> We also see some on-premise use cases emerging.

487.82 -> We're also the largest studio in the world.

489.9 -> We're building our own originals,

491.37 -> and our own content for our service.

493.35 -> And what we're seeing is there are some use cases,

495.44 -> where we have to start pulling compute

497.56 -> and storage closer to our creatives

499.49 -> that are actually building our content.

502.08 -> We launched mobile games earlier this year.

504.61 -> Mobile games is interesting,

505.58 -> because it's something new for us.

507.69 -> We don't yet have great line of sight

509.16 -> into what exactly that means,

511.13 -> but I wanna make sure that I'm doing my job

513.44 -> and preparing the network for anything.

516.04 -> So I call these business requirements,

518.54 -> and I'm gonna translate each one

519.52 -> of those business requirements

520.59 -> into a technical requirement here.

522.54 -> So first off the flat network,

524.03 -> what this means for me as a member

525.35 -> of the cloud networking team is that I need to be able

527.84 -> to support hundreds of VPCs today,

529.76 -> all while maintaining full IP reachability.

532.76 -> We talked, for containers,

535.34 -> we need a number of IPs on an ENI

537.91 -> How many IPs can we put on an ENI?

540.04 -> And in some cases, these are really short-lived.

542.35 -> The fastest I've ever seen a container come up,

544.71 -> use the network, and go away is five seconds.

547.66 -> That's five seconds where an IP was needed,

550.67 -> allocated, used, and then went away.

553.2 -> So this is really incredibly dynamic.

559.45 -> So "N" IPs per ENI.

561.69 -> The more IPs we can put on an ENI,

563.58 -> the more efficient we become with containers,

565.37 -> the more efficient our business becomes

567.35 -> as our overall container platform becomes more efficient.

571.3 -> We think of this less as a container requirement,

573.37 -> so containers are driving how we use ENIs.

576.34 -> And so, this is more of an ENI density requirement,

578.6 -> and we'll refer to it as ENI density

580.49 -> for the remainder of this talk.

582.64 -> As we continue to grow, we need to change our mindset.

585.56 -> We're no longer thinking about 100 plus VPCs.

588.25 -> We're thinking 1,000 VPCs, thousands of VPCs.

591.23 -> How can we maintain a network to support this

593.93 -> all while having full IP reachability?

596.94 -> And the on-premise use case is similar.

598.87 -> We wanna be able to achieve on-premise

601.68 -> full IP reachability as well, again,

603.65 -> really teasing apart transport versus policy.

606.12 -> We wanna layer policy on top of our network,

608.68 -> and enable our network to just be simple transport.

613.08 -> So we've identified some flaws with our current approach,

615.36 -> and our current approach being

616.77 -> the exclusive use of private IPv4.

621.51 -> First, the flat network,

624.07 -> it's actually not that flat.

625.4 -> We have this concept of public and private IPv4 addressing,

628.66 -> and in some cases we'll see services will select public IPs

632.28 -> when they really should've been selecting the private IPs,

634.21 -> and traffic will go out a NAT gateway,

635.81 -> and loop back into our VPC.

637.65 -> This is suboptimal for a number of reasons.

639.53 -> It's costly, because NAT gateways are not super cheap,

643.36 -> and then it has latency,

644.48 -> and it has some bandwidth restrictions as well.

646.47 -> Overall, there's just a lot of room for improvement there.

649.92 -> For ENI density, I mentioned the ENI limits,

652.47 -> or not EN, well, AWS limits in general.

654.89 -> So again, how many ENIs can be attached to a compute node?

658.8 -> How many IPs can we put on those ENIs,

661.02 -> and then how fast can we change those?

662.94 -> Again, five seconds is incredibly fast.

665.53 -> If we have to contact the AWS API every time

669.15 -> we wanna use an IP, assign it to an ENI,

671.5 -> and then have it go away,

673.84 -> we actually can't mutate the network that fast at our scale.

677.7 -> So what we end up doing is we end up

679.97 -> predictively caching ENIs, or predictively caching IPs

683.21 -> that we think we're gonna need really very soon.

686.42 -> And then this creates an accounting problem.

688.53 -> This is starting to snowball here.

689.94 -> Now, we need to be able to think about

692.38 -> the lifecycle of IP address.

694.17 -> What was an IP address at a particular point in time?

697.987 -> And there are some, additionally, there are some workloads,

700.2 -> where we want to attach an EIP directly to a container.

703.6 -> So an EIP is an Elastic IP address,

705.547 -> and it's really a public IPv4 address.

708.494 -> And the way the contract works with the AWS API is

710.696 -> we take an EIP and we associate it to a private address.

714.99 -> And so, what we'll see is that if we wanna start a workload

717.98 -> using a public IPv4 address, if we start it too soon

721.17 -> after that association was successful,

723.5 -> we have intermittent connectivity.

725.27 -> So there's a propagation delay problem here as well,

728.07 -> and it gets more and more challenging

730.16 -> the more and more we use that technique.

733.51 -> So more flaws with our continued growth.

735.86 -> We're concerned around routing limits.

738.38 -> So I mentioned today, we're around the 100-ish VPC limit.

742.12 -> We can accommodate a full network

743.77 -> with full IP reachability using VPC peering.

746.74 -> But as we continue to grow,

748.15 -> we know we're gonna have to do something about that.

751.4 -> As we create more VPCs, we actually predict that

754.92 -> we're going to accelerate

756.38 -> our IPv4 address exhaustion problem.

759.87 -> So the more places in the network you have,

762.04 -> the more address space you allocate,

763.84 -> the less efficient you become

764.94 -> with that address space overall.

767.28 -> The on-premise story is similar.

769.82 -> We're concerned about address exhaustion,

771.43 -> again, that theme of, you know, more places in the network,

774.14 -> the less efficient you are overall.

776.72 -> But then we're also concerned about routing limits.

778.93 -> But in this case, it's less about VPC peering,

782.02 -> and more about Direct Connect.

783.4 -> How do we Direct Connect all of these VPCs

785.45 -> to provide reachability on-premise?

787.49 -> How do we, you know, how many disks can we put

789.76 -> on a Direct Connect?

791.55 -> How can we, you know, scale Direct Connect gateways,

794.43 -> those sort of things?

796.3 -> And so, what we did was we asked ourselves,

797.707 -> "What are we gonna do?"

798.54 -> We realized these problems were coming at us

800.54 -> a few years ago, and we collectively thought as a team,

803.61 -> you know, "What are we gonna do here?"

808.32 -> And so, anytime we're faced

809.28 -> with these really large architectural decisions,

812.38 -> what we end up doing is partnering with AWS.

815.995 -> As Rob mentioned earlier at the start,

817.56 -> we've been in AWS for a number of years,

819.62 -> since around the 2008 mark.

821.22 -> AWS has been a great partner with us for this so far,

824.65 -> and we know they're gonna continue to be

825.77 -> a great partner with us as we move forward.

828.67 -> So when we opened up this dialogue,

830.27 -> we started tossing out some ideas,

832.42 -> the first idea being this EC2-Classic mode of operation.

836.28 -> So we thought, you know,

837.567 -> "We're seeing some problems continuing to grow.

840.317 -> "Could we go back to this EC2-Classic model?

842.727 -> "We've done this before," we thought.

844.81 -> The general idea here is that we'll stop caring about

847.18 -> private IPv4 addresses.

848.89 -> We'll give everything a public IPv4 address,

851.28 -> and just communicate through the IGW.

854.19 -> Again, we've done this before.

855.94 -> We've fundamentally operated the streaming service

858.03 -> in this mode in the past.

860.93 -> What happened though is that once we moved into VPC,

863.24 -> we started using containers,

864.447 -> and we added this ENI density constraint

866.51 -> that we didn't have back then.

868.22 -> And so, this model doesn't address that constraint

870.47 -> that we now have.

871.8 -> In fact, it actually makes it worse,

873.11 -> because of that EIP update,

874.28 -> like the problem I mentioned earlier.

876.05 -> So this is a no-go.

877.93 -> The next idea we considered was

879.88 -> what I like to call tiny bubbles.

882.4 -> So we think about VPCs as bubbles, or islands,

885.35 -> and if you wanna interconnect the VPCs,

887.69 -> you have to use PrivateLink,

888.88 -> which is just a combination of VPC endpoints and NLBs.

893.51 -> And so, you have really explicit interconnection points

895.56 -> between your VPCs.

896.42 -> You stop caring about non-overlapping IP space

898.7 -> across your VPCs, and it's a great way to continue to scale.

901.63 -> Again, it's a well-defined pattern.

904.53 -> The issues with this is client-side load balancing is

907.27 -> so critical to our business that moving to this model means

910.52 -> that we have to start putting NLBs between our services,

913.27 -> which is a form of server-side load balancing,

915.03 -> which doesn't work well for us.

917.48 -> It also doesn't address this ENI density constraint

919.7 -> that I mentioned earlier.

921.33 -> It just keeps it the status quo.

923.95 -> In fact, no idea that we talked about

926.51 -> in these conversations, no, there's no option here

929.29 -> to solve this particular challenge.

931.95 -> And I recognize that all solutions

934.98 -> are a compromise to some degree.

938.71 -> But I have an idea here.

940.71 -> Before I focused on cloud networking,

942.31 -> I did what I would call traditional networking.

944.55 -> I worked in data center environments,

946.29 -> and there's a design pattern from those days of old

948.779 -> that I'm recognizing, I'm pulling forward here.

951.58 -> So if I asked myself how would I solve this problem

953.72 -> in a traditional network,

955.45 -> I would route a network block to a host.

958.39 -> We'd send a CIDR block to a particular compute node.

962.395 -> And so, we thought to ourselves,

963.977 -> "Could we do something similar inside AWS?"

967.95 -> So conceptually, what we wanted to accomplish is

969.84 -> we wanna move from this,

971.23 -> assigning individual IPv4 addresses

973.44 -> to an elastic network interface,

974.88 -> and move to a model where we assign a prefix,

977 -> a CIDR block, to a single ENI.

981.311 -> And we were actually talking with the AWS service teams

983.07 -> about that right here at re:Invent

986.38 -> in Vegas in 2019.

989.02 -> And when we were sketching this out on the back of a napkin,

991.81 -> we titled it prefix delegation,

993.4 -> and we've been talking about it in that way ever since.

996.27 -> And I'm really excited, because four months ago,

998.71 -> this actually became a reality.

1001.28 -> Now, Joe Mag was generous enough

1002.93 -> to let me put his tweet on the board here,

1005.24 -> and I wanted to do this because I think

1006.96 -> his words describe it best.

1008.46 -> This is a big deal.

1010.29 -> Being able to assign an IPv4,

1012.771 -> and/or an IPv6 prefix directly to an ENI

1016.2 -> is a game-changer for containers,

1018.3 -> which is exactly the pain point that we have today.

1027.434 -> A slight side effect,

1029.59 -> I mentioned earlier that all solutions are a compromise,

1032.31 -> and this is no different.

1033.45 -> So a slight side effect of using

1035.98 -> this new primitive prefix delegation is that

1037.87 -> we become overall less efficient with our IP space.

1041.35 -> In some cases, we may need to assign a single IP,

1043.88 -> or four, or six to a particular ENI,

1047.4 -> and with prefix delegation, we're really hard-coded,

1050.24 -> and we're set on a set amount.

1051.98 -> And so, we need to ask ourselves if we wanna buy

1054.55 -> all into this technique, how much IP space do we need?

1059.21 -> So let's think about that.

1061.29 -> The three primary address ranges that folks generally use

1064.1 -> in VPCs today are the RFC 1918,

1066.69 -> your 10/8, 172.16/12,

1069.426 -> 192.168/16.

1071.2 -> And so, we have those up on the board.

1073.06 -> The size of the box is the respective size

1075.06 -> compared to each other.

1076.82 -> For a variety of reasons that go back

1078.25 -> to when we initially moved the streaming service

1080.56 -> out of EC2-Classic and into VPC,

1083.11 -> we started using an additional address range.

1086.21 -> So this is RFC 6598.

1088.27 -> It's 100.64/10.

1089.98 -> It works equally as well for private IPv4 addressing

1092.86 -> inside AWS, or on-premise.

1096.73 -> Okay, so I mentioned earlier that

1098.83 -> we maintain non-overlapping IP space

1100.7 -> across our entire enterprise.

1101.97 -> And so, what I've done here is I've marked off on the board

1105.17 -> the IP space that is used on-premise.

1108.87 -> And now, in gray is the IP space

1111.54 -> that we are currently using inside VPC.

1114.07 -> And then the space in black is used for future allocation.

1117.44 -> So we'll put the two full blocks aside,

1119.5 -> the two smaller ones,

1120.333 -> and we'll just focus on IP planning

1122.19 -> using the open space here.

1125.03 -> So first and foremost, the single prefix is 16 IPs, a /28.

1129.2 -> Doesn't really register on the radar.

1131.17 -> When we're thinking about what is the upper bound

1132.86 -> for how many IPs we might need on an ENI,

1134.92 -> we you came up with a 64 number.

1137.02 -> So that's four prefixes on an ENI, 64 IPs, a /26.

1140.74 -> When we're just looking at one of those,

1142.8 -> it has no impact in our overall IP planning.

1145.92 -> Where things get interesting,

1147.29 -> as we think about how many elastic network interfaces

1149.97 -> we might need in a particular availability zone.

1152.27 -> So we landed on this 8K number,

1154.16 -> and we started to see some real big movement

1156.41 -> in terms of growth.

1157.95 -> So in order to satisfy this,

1159.26 -> we're looking at half a million IP addresses,

1161.84 -> which would be summarizable into a /13.

1165.35 -> Now, that's just one zone.

1166.78 -> I didn't mention this earlier,

1167.83 -> but we only operate our streaming service

1170.1 -> in regions that have at least three availability zones.

1172.55 -> So we need to multiply that, you know,

1174.983 -> four prefixes per ENI, 8K prefixes in a zone

1178.34 -> multiplied by three, three zones per region.

1180.84 -> We take that half million IPs.

1182.16 -> We get a million and a half IPs.

1184.27 -> In order to accomplish this,

1185.103 -> we would need an open /12 and a /13, which we have.

1189.04 -> We can do that.

1190.46 -> This is just one region now.

1191.92 -> I mentioned we have three regions.

1194.39 -> So as we multiply that by three regions,

1196.41 -> we switch from, you know, a million and a half IPs

1198.99 -> to needing 4 1/2 million IPs.

1200.97 -> So now we would need a /10 and a /13.

1204.36 -> We could do, we're out of space here.

1206.11 -> So this blue, these three blue boxes below the 10/8,

1210.08 -> that's IP space that we would need, but we don't have.

1212.88 -> We're out of addresses at this point.

1216.07 -> And I recognize that we're being fairly,

1218.8 -> very generous with our calculations here.

1220.79 -> So we could probably do something.

1222.45 -> We're a little bit over for our current size,

1225.68 -> but again, this is just our current size.

1227.92 -> We need to accommodate future growth,

1229.91 -> and my knee-jerk reaction would just be to double it.

1233.34 -> If I can't support doubling the size of my network,

1236.71 -> I don't think I'm doing a good enough job

1238.23 -> to be working at Netflix.

1240.43 -> So if we were to double it, we move from,

1243.86 -> you know, being a little bit over

1245.28 -> to something that we could do to just being grossly over.

1248.01 -> You know, we need to do something here.

1249.48 -> We know that the numbers aren't gonna work out for us.

1253.59 -> Oh, wanna go back.

1255.5 -> Okay, so I do need to caveat this a little bit,

1258.66 -> because these are not the exact numbers

1260.36 -> that we used internally for IP planning.

1262.38 -> I was advised that using the exact numbers

1264.9 -> would showcase our exact size as a company.

1269.84 -> So what we're doing here though is we're showing

1271.5 -> a general progression, because at the end of the day,

1274.1 -> this is showing the same result

1275.55 -> as what we came to internally.

1277.17 -> When we're asking how much IP space do we need,

1279.56 -> the answer is a lot.

1283.3 -> And I don't think it's any guess where we're going here next

1286.61 -> with this next logic step.

1290.06 -> We need IPv6.

1291.78 -> We need IPv6 to accommodate our future growth,

1295.41 -> but also to accommodate our container density on ENIs.

1300.45 -> And so, conceptually, when we go back to thinking about

1302.9 -> ENI density, we wanna move from this model

1305.06 -> with individual IPs and IPv4 prefixes

1307.9 -> to this model, thinking about individual IPv6 addresses

1311.237 -> and IPv6 prefix.

1314.55 -> Now, we've been on quite a bit of a journey here.

1316.01 -> I think we're down a bit of a rabbit hole.

1317.59 -> We've started talking about our network growing,

1320.57 -> and you know, we started adding containers,

1322.18 -> and how that's putting pressure on the network

1324.13 -> and ENI density, and we think we can solve this

1326.524 -> with prefix delegation, but we're gonna run out of IPs.

1332.38 -> So what I wanna do here is I wanna pull our heads above,

1334.97 -> back up above water, 'cause at the end of the day,

1337.23 -> we're trying to solve problems for the business,

1339.5 -> and I wanna make sure that we're still on track,

1341.52 -> because, again, we're trying to solve problems

1342.713 -> for the business.

1344.13 -> So we need to ask the question

1346.15 -> does IPv6 solve our business network requirements

1349.19 -> that we were talking about earlier?

1351.24 -> So with the flat network, absolutely.

1353.73 -> IPv6 is a single address space.

1356 -> We don't have this concept of public and private anymore,

1359.7 -> and no NAT, which is great.

1361.35 -> We don't have that awkward transition of, you know,

1363.78 -> things going out a NAT gateway, for example.

1367.08 -> Now, the first time I mention this to somebody,

1369.24 -> usually what I hear is, "Well, well, we need NAT.

1371.817 -> "We need NAT to protect us against things on the internet."

1376.4 -> And that's not true.

1377.75 -> NAT was, and NAT is not, and never was intended

1381.32 -> to be used as any sort of security barrier.

1384.3 -> What is used as a security barrier is the staple firewall.

1387.64 -> Now, typically today, what we see is that NAT

1390.19 -> and firewalls are bundled together in implementation.

1393.17 -> So firewalls, NAT gateways, for example,

1395.58 -> it's doing NAT-ing and firewalling.

1397.61 -> What we actually care about though for this concern

1400.12 -> around protecting ourselves from the internet

1402.66 -> is the firewall concept.

1404.45 -> And AWS also provides a primitive

1406.15 -> to be able to accomplish this.

1407.71 -> So the Egress-only Internet Gateway, or EIGW,

1410.832 -> is a complementary technique that,

1413.35 -> or primitive, I should say,

1416 -> that allows us to have internal, or private subnets

1419.48 -> that can send out to the internet,

1421.09 -> but won't receive connectivity inwards

1423.86 -> from, you know, the general internet.

1425.85 -> Anyway, flat network, no NAT, great with v6.

1430.58 -> Prefix delegation, ENI density,

1432.41 -> we talked about this quite a bit,

1433.86 -> and again, this is multifaceted.

1435.94 -> Prefix delegation not only solves our concern

1438.84 -> with how many IPs we can put on an ENI,

1440.83 -> but also with how fast we can change it.

1443.04 -> So prefix delegation allows us to do

1444.69 -> a single control plane operation,

1446.43 -> set up the prefix to an ENI,

1448.11 -> and then we're free to churn those IPs

1449.89 -> as much as we want after that.

1451.74 -> This two-for-one combination, again, is really powerful,

1454.04 -> and the reason why we wanna move forward

1455.49 -> with this technique.

1457.83 -> So continued growth.

1458.68 -> We talked about address exhaustion earlier.

1462.08 -> This is a 300-level course.

1463.24 -> I hope everybody understands that the v6 address space is

1465.93 -> sufficiently large, that we don't worry about this anymore.

1470.93 -> What we haven't talked about yet, though,

1472.44 -> is our concerns around routing limits.

1474.14 -> How do we actually interconnect all of our VPCs

1476.77 -> in this type of hypothetical world

1478.43 -> we're talking about so far?

1480.05 -> So let's think about that a bit.

1482.24 -> The first way I ever connected two VPCs together was

1484.98 -> with this concept of a Customer Gateway, or CGW,

1488.45 -> and the way this generally works is you create

1490.29 -> a VPN tunnel to VPC A, you create a VPN tunnel to VPC B,

1493.83 -> and you route traffic between your VPCs using

1496.13 -> your on-premise equipment.

1497.68 -> Has a lot of drawbacks.

1498.77 -> It's, you know, costly, and it has administrative,

1501.5 -> costly in terms of administrative overhead to set up,

1504.68 -> got some throughput limitations, a lot of latency there.

1507.21 -> But it is in fact an option to interconnect VPCs.

1511.02 -> The way that, the next way Amazon provided

1514.65 -> an option for connecting VPCs using private IPv4 was

1517.67 -> with VPC peering.

1519.05 -> VPC peering is really cool, because it's not so much a thing

1521.75 -> that you send traffic through,

1522.74 -> but more of a permission set is

1524.09 -> how I generally think about it.

1525.78 -> So it scales super well in terms of bandwidth

1530.26 -> and packets per second.

1532.7 -> It's super, super cost efficient,

1534.53 -> but it has a relatively low upper bound

1536.26 -> in terms of how many VPCs we can interconnect.

1538.66 -> I believe the limit's at 125 today.

1541.88 -> And so, if we're thinking about going above that limit,

1544.04 -> we really need to start thinking about Transit Gateway.

1546.61 -> Transit Gateway bumps that limit up quite high,

1548.86 -> moves from hundreds to thousands.

1551.741 -> I believe the limit's 5,000 today.

1553.63 -> And so, when you're thinking about that scale,

1556.29 -> Transit Gateway is really the way forward there.

1559.09 -> Now, a drawback of Transit Gateway is that

1562.55 -> you have, it's costly to maintain.

1564.29 -> Traffic's actually being sent through a thing today

1566.58 -> with how Transit Gateway is implemented,

1568.73 -> so there's bandwidth restrictions,

1569.93 -> there's a little bit extra latency increase,

1571.95 -> and it costs more than VPC peering.

1575.21 -> But what we're talking about here, if I stand back again

1577.8 -> and I think about the themes that we're talking about,

1579.8 -> is we're talking about ways to explicitly interconnect VPCs.

1583.01 -> I have to do a thing as an engineer.

1588.53 -> But what I actually want is I actually just want

1590.83 -> to accomplish my outcome.

1592.21 -> And the outcome that I want to accomplish is

1594.21 -> I want the things inside the VPCs to communicate.

1596.41 -> I actually don't care how that's done.

1599.298 -> We talked about an option earlier

1601.69 -> with the Amazon EC2-Classic model,

1603.86 -> where we could have a network set up

1605.82 -> by not setting up a network,

1608.13 -> just allow overlapping IP space in our VPCs,

1610.81 -> send traffic out the IGW,

1612.16 -> really lean into public IP connectivity.

1615 -> This works well for public IPv4 reachability,

1617.31 -> but it doesn't work for private IPv4 reachability.

1620.34 -> But with respect to IPv6,

1622.258 -> IPv6 is considered public in this regard.

1624.96 -> So we now have another option to interconnect our VPCs

1629.05 -> with IPv6 that we don't have with private IPv4.

1631.8 -> And I think this is really under-thought of as an option

1634.75 -> to interconnect your VPCs these days.

1641.03 -> Yes.

1642.16 -> So when we go back to thinking about how

1646.16 -> IPv6 can solve our business requirements,

1648.75 -> we move from thinking about

1650.67 -> having to explicitly set up our network

1652.66 -> to having this option to implicitly set up our network.

1656.36 -> I don't have to explicitly configure a thing,

1658.58 -> or do something.

1659.55 -> I get implicit reachability just by using IPv6.

1663.46 -> Now, I recognize there might be some reasons

1665.5 -> to do that explicit setup,

1667.1 -> if you need to use security group references, for example,

1670.18 -> but for some workloads, we actually don't need to do that.

1672.33 -> It might be fine.

1673.32 -> You could go to the extreme of doing

1674.85 -> a workload per VPC, for example.

1678.06 -> Anyway, the key takeaway here is that

1680.04 -> we have another option that we don't with IPv4.

1683.02 -> And the on-premise story is similar.

1685.06 -> We end up not having to have explicit setup.

1687.54 -> I don't have, if I don't have to Direct Connect

1689.38 -> all my VPCs, I think that's fantastic.

1694.41 -> For our IPv6, we can set up one public Direct Connect

1696.99 -> with one public VIF, and we can have reachability

1699.88 -> into all of our VPCs.

1701.36 -> That's super cool to me.

1704.04 -> So what we're seeing here is that IPv6

1707.26 -> is starting to check some boxes.

1709 -> There's a lot of business value.

1710.3 -> We're seeing a lot of business value in moving to IPv6.

1714.94 -> And so, what we need to do is we need to think about

1717.13 -> if this is actually something we wanna do at Netflix

1720.28 -> as a company.

1721.59 -> And the key here is as a company,

1723.99 -> because there are a number of things that

1725.27 -> a lot of folks are gonna have to do at Netflix

1727.24 -> to be able to support v6.

1729.78 -> So what I did as part of my job is I went out

1732.79 -> and I talked to all my fellow engineering teams,

1734.78 -> and I did what we call farming for dissent.

1737.07 -> So I'm talking to these engineering teams,

1739.1 -> trying to figure out if there's, if I'm missing something.

1742.16 -> From my standpoint, IPv6 seems to be ticking a lot of boxes.

1744.94 -> We should do this.

1746.01 -> Am I missing something?

1746.97 -> Is there any reason why we must hit the red button,

1749.62 -> and figure out a different solution?

1751.35 -> Because I don't see it yet.

1753.53 -> And in these conversations, again, I couldn't find a reason.

1757.55 -> So we ended up hitting the green button,

1760.26 -> and we decided to, as an engineering community at Netflix,

1765.57 -> move forward with IPv6.

1773.11 -> Now, we know we wouldn't,

1774.89 -> we knew that we would not be successful in this endeavor

1777.68 -> without partnering in co-innovation with AWS.

1781.075 -> And the first part of co-innovation I've already mentioned

1783.15 -> a time, or two, prefix delegation,

1786.07 -> the ability to assign a prefix to an ENI.

1790.03 -> What I haven't talked about yet is why it's a /80,

1792.297 -> and that's an interesting story that I think

1794.11 -> I should give some airtime to.

1795.93 -> So from past experience, if you were to route

1799.46 -> a network block to a compute host using a v6,

1802.01 -> generally, it would be a /64.

1803.728 -> It's a lot more common.

1806.33 -> But when we looked at what it would take to actually

1808.39 -> accomplish that using the existing AWS primitives,

1810.61 -> the math didn't really work.

1813.9 -> So a VPC today is a /56.

1818.536 -> Out of the /56, you get 256 /64s.

1823 -> And from our earlier session,

1824.39 -> when we were talking through

1825.42 -> private IPv4 address exhaustion,

1827.13 -> we know we need thousands of these things.

1829.18 -> So something has to give there.

1831.14 -> So we gave a different consideration.

1833.04 -> You know, what if we moved the boundary a bit over?

1834.73 -> We'll move it, you know, from 64 to /80, one hextet over,

1838.62 -> and things start to look a little bit different.

1841 -> So now within a particular VPC subnet,

1843.52 -> you can have 65,000 of these,

1846.58 -> so between 64 and 80 is 16 bits, two to the 16 is 65K.

1851.5 -> So in every single VPC subnet now,

1854.11 -> you get 65,000 prefixes in terms of optionalities

1857.81 -> for assigning to ENIs.

1860.01 -> The downside, what you're giving up there,

1861.83 -> is you're giving up the amount of unique IPs per ENI,

1865.62 -> and really the difference between 64

1867.617 -> and 80 is pretty minimal.

1869.8 -> So a /80 is 48 bits left in the address.

1874.07 -> So two to the 48, or between 80 and 128, you get 48 bits.

1878.94 -> So two to the 48th is a number that I don't have memorized.

1882.92 -> But that's the same bit space as MAC addresses,

1885.41 -> those hardware addresses on network interfaces.

1888.08 -> So in terms of order of magnitude, we are now in a situation

1891.99 -> where we have as many unique IPs on every single ENI

1896 -> as there are unique MAC addresses in the world.

1898.6 -> It's virtually limitless,

1900.54 -> and the amount of increased limitless from /80 to 64,

1905.11 -> we get a lot of diminishing returns there.

1907.68 -> So anyway, /80, landed on that.

1912.54 -> An open issue that I wanna talk about a bit

1914.34 -> is around transition mechanism.

1916.71 -> So IPv6 and IPv4 are not directly compatible.

1919.81 -> You need to do a thing to allow something that's v6-only

1922.78 -> to communicate to something that's v4-only.

1926.79 -> And what we're interested in at Netflix is

1929.13 -> driving for IPv6-only containers.

1931.4 -> And so, we need to think about how can we provide

1933.67 -> a backwards compatibility layer

1935.77 -> for things that are v4-only.

1938.54 -> The industry standard way to accomplish this

1940.19 -> is with DNS64 and NAT64.

1942.32 -> Quick overview of how this works,

1944.64 -> for our particular case, a container that's running v6-only

1947.68 -> will reach, oh,

1949.67 -> our container that's running v6-only will reach out

1952.66 -> and ask for a DNS record,

1954.07 -> and will do the v6 compatible record lookup.

1957.3 -> The DNS64 service will recognize that it only supports IPv4,

1960.587 -> and then will synthesize a response.

1963.6 -> And in that response, it'll put the route,

1966 -> or the destination to the NAT64 gateway, or instance.

1969.77 -> And so, when the container tries to send a packet,

1973.261 -> it'll send the package to the NAT64.

1975.78 -> Oh, I forgot to mention,

1977.51 -> that response from DNS64 will also have

1979.75 -> the ultimate IPv4 destination as well.

1981.56 -> They're both encoded in that response.

1984.69 -> And then in the NAT64, we'll move the segment data

1987.41 -> from v6 to v4, and the packet goes on its merry way.

1991.23 -> In order to use this technique,

1993.46 -> you're required to use DNS.

1998.18 -> And so, what I mentioned at the beginning of this talk is

2000.43 -> we couldn't rely on DNS when the streaming,

2002.25 -> when our product offering started,

2003.97 -> so we don't rely on DNS for interservice communication,

2007.27 -> so we don't have that hook to be able to use this technique.

2011.43 -> That's probably solvable, though.

2012.68 -> You know, there's a lot of incredibly talented folks

2015.64 -> at Netflix, and I think this could actually be solvable,

2018.15 -> but what's more difficult though

2020.2 -> is how security group references work.

2022.33 -> So again, we really think about our network

2024.53 -> in terms of transport versus policy.

2026.33 -> So we provide a simple transport network,

2029.04 -> and we layer policy on top with security groups.

2031.66 -> And so, anytime we have something that obfuscates that,

2034.62 -> that's a problem for us.

2036.15 -> NAT gateways and NAT64, when you send packets through them,

2040.4 -> they fundamentally lose your security group context.

2043.21 -> So now we lose the ability to do security group references,

2046 -> which is something that we do so much at Netflix.

2050.7 -> So this was such a problem for us that we actually decided

2052.72 -> to innovate in this area ourselves.

2054.74 -> So again, in the similar type of setup,

2057.023 -> what we do is instead of relying on NAT64

2060.4 -> to provide that transition between v6 and v4,

2064.41 -> we just run a transition mechanism

2066.24 -> on every single compute node,

2068.4 -> our Titus compute infrastructure, again.

2071.85 -> And so, in this particular situation,

2073.8 -> we can have a container that's IPv6-only,

2076.2 -> but the ENI that's attached to our compute node,

2078.905 -> or Titus EC2 instance, will be dual-stacked.

2081.84 -> It will have both an IPv6 prefix and an IPv4 address.

2086.9 -> And so, this transition mechanism will then allow

2090.06 -> that communication to go out.

2092.17 -> Now, it's still a transition mechanism.

2094.75 -> Oh, and security group references work,

2096.16 -> because we're, you know, co-tenanted on the same ENI

2099.25 -> and security groups are at the ENI.

2102.62 -> So I mentioned that this is a transition mechanism.

2104.45 -> It's not implemented as NAT64 and DNS64.

2107.99 -> It uses what we call internally as the TSA,

2110.49 -> or the Titus Syscall Agent.

2112.23 -> It's providing the same functionality,

2114.41 -> but it's done at the syscall layer.

2116.56 -> It's a bit more of a novel approach.

2119.76 -> Our team did an entire presentation

2121.41 -> on just this one component earlier this year

2124.95 -> at the Linux Plumbers Conference,

2126.73 -> and the short link to be able to view that presentation

2129.7 -> is up on the screen there.

2130.72 -> So it's bit.ly/nflx-tsa.

2135.1 -> So if you're interested for how we're solving

2137.74 -> this transition mechanism problem

2139.61 -> while maintaining security group references to work,

2142.42 -> I encourage everybody to check that out.

2145.05 -> Now, again, it's specific to Titus,

2147.03 -> but we are hoping to open source this in the future.

2150.73 -> But right now, it's still only at Netflix.

2158.02 -> Okay, so I wanna switch gears a bit

2159.88 -> and talk about our progress so far going towards IPv6.

2163.11 -> The techniques that I talked about so far are

2165.5 -> the techniques that we're moving forward with as a company.

2168.93 -> So today at Netflix, we are using IPv6.

2172.11 -> We have used prefix delegation in production.

2177.35 -> Kind of hoping for some claps.

2178.3 -> There we go, all right. (audience applauds)

2179.56 -> There we go. (audience applauds)

2180.73 -> Thank you, thank you. (audience applauds)

2183.38 -> And so, for us, we've been focusing

2184.94 -> just on IPv6-only containers,

2187.1 -> and by driving just that use case,

2189.57 -> we've driven 25% IPv6 adoption

2192.55 -> as measured by unique network flows inside of our VPCs.

2195.87 -> And to quantify that a bit more,

2197.27 -> we started this calendar year, 2021, with less than 1%.

2201.393 -> So that's quite a big, quite a bit of growth

2203.06 -> in a very short amount of time.

2206 -> We've learned a lot in this process,

2207.25 -> and I wanna share some of those lessons learned

2208.82 -> with you all today. (audience laughs)

2212.58 -> I got more applause, more laugh on that.

2215.103 -> (audience laughs)

2217.41 -> Cool.

2218.41 -> So it doesn't sound like that's contentious.

2220.94 -> Old code is definitely not fun, especially Java.

2223.8 -> What I see time and time again is that

2226.25 -> developers are tuning the JVM to explicitly disable IPv6,

2230.98 -> or at least prefer IPv4.

2232.91 -> And the streaming service, the Netflix streaming service,

2235.17 -> is primarily built in Java.

2236.77 -> And so, you can imagine how widespread this issue is.

2241.25 -> When I look in our internal code repository

2243.13 -> to the first time I ever saw this actually being committed,

2246.62 -> it lined up with when IPv6 was enabled

2248.83 -> on the corporate network.

2250.29 -> And at this time, IPv6 was relatively new.

2252.72 -> This was, you know, over 10 years ago.

2254.86 -> And the commit message was something like, you know,

2257.047 -> "I saw IPv6 on the corporate network today.

2259.347 -> "This library broke, disabling v6."

2261.8 -> And I don't blame them at all.

2263.44 -> I would probably do the same thing.

2264.67 -> At the end of the day, they're building software

2266.64 -> to support the business, and the network broke them,

2269.32 -> and they need to move forward.

2270.81 -> So that's great, I don't blame them.

2272.63 -> But what's tricky, though, is that those libraries

2275.13 -> that broke back then are fixed now.

2277.24 -> And so, unwinding that is the challenge.

2281.75 -> My next lesson learned that I wanna share with you all today

2284.71 -> is that assigning IPv6 to a node

2287.11 -> doesn't actually mean it's gonna be used.

2289.29 -> So similar to the Java problem, right,

2291.22 -> the JVM is explicitly preferring v4 over v6.

2295.43 -> But once we fix that, we end up seeing another problem.

2299.84 -> We operate nodes inside Netflix with sidecars,

2302.337 -> and these sidecars are generally very independent

2304.57 -> from the workloads.

2305.95 -> And what we'll see is that if we fix this JVM problem,

2309.35 -> for example, and this Java service is using

2312.15 -> both v6 for ingress and egress,

2314.59 -> we'll have sidecars on that node that are still

2316.75 -> preventing it from moving to a dual-stack stage

2319.74 -> operating with both v6 and v4 to going to IPv6-only.

2323.94 -> So when you're thinking about how to move to v6-only,

2326.53 -> you can't just think about the workload itself.

2328.95 -> You have to think about the entire compute node,

2330.78 -> everything that's running on that, sidecars included.

2336.44 -> The next one is that Happy Eyeballs masks IPv6 problems.

2343.09 -> So Happy Eyeballs, for those of you who may not be aware,

2346.56 -> is an IETF standard for giving preference to IPv6

2350.21 -> while still allowing fallback to IPv4.

2353.1 -> Now, don't get me wrong, this is great.

2354.83 -> This was required and needed to gain the level

2357.44 -> of IPv6 adoption that we have on the internet today.

2361.02 -> But we're using IPv6 in a data center environment,

2363.3 -> not over the general internet,

2364.73 -> and we see different results with Happy Eyeballs.

2367.61 -> So again, in VPC, we have tight control over the network,

2370.797 -> the TCP clients, the TCP servers,

2374.259 -> and Happy Eyeballs provides, is a fallback there.

2379.09 -> And we believe in IPv6.

2382.9 -> We believe in IPv6 so much that we don't want this fallback,

2386.95 -> 'cause this fallback is another if statement.

2390.5 -> It's something that we have to consider

2391.77 -> in terms of our resiliency exercises.

2394.82 -> It's something that should be regularly exercised.

2397.23 -> And so, again, we believe in IPv6 so much that

2399.45 -> we actually went through and we disabled Happy Eyeballs

2402.21 -> on some of our largest IPC platforms.

2404.52 -> And what this does is it provides a strong stance, saying,

2407.637 -> "We believe in IPv6."

2409.36 -> If there's an IPv6 problem, it is a network problem

2412.24 -> that needs to be fixed.

2415.6 -> My next lesson learned is how little IPv6 support

2419.28 -> there is today for all of these AWS Managed Services.

2424.25 -> I was kind of hoping for another, "Ooh, ah."

2427.517 -> (audience laughs)

2431.32 -> What wasn't surprise,

2432.153 -> or what was more surprising to me though

2434.37 -> was how many of these AWS services we use at Netflix.

2437.53 -> So if you think about things like RDS, ElastiCache,

2440.68 -> DynamoDB, all those great products that Amazon provides,

2444.75 -> very few of them support IPv6 today.

2447.5 -> And this is what makes the transition mechanism

2449.97 -> so important for us, because for those things that operate

2452.92 -> inside VPC, like ElastiCache, RDS,

2455.51 -> we can still use security group references

2458.07 -> with how we've implemented the transition mechanism,

2460.7 -> to allow us to move to IPv6 at our own pace,

2463.5 -> while still allowing these AWS Managed Services

2465.53 -> to get with the program at some point.

2470.72 -> All right, so I wanna share some of my best practices

2472.67 -> with y'all today.

2475.5 -> First and foremost, communication.

2477.71 -> Now, I mentioned earlier that as part of my job,

2480.153 -> I went through and I farmed for dissent

2481.71 -> I tried to figure out if there was a reason why

2483.92 -> we shouldn't use v6 at Netflix.

2486.37 -> And something interesting happened in that process

2488.41 -> that I didn't see right away, but I see now.

2491.58 -> So as part of that process, what I did was I socialized

2493.86 -> the problems that were seen at the network layer,

2496.35 -> and it kind of gave a heads up to all these different teams

2498.49 -> that they can expect to see IPv6 coming more and more.

2501.96 -> And what this did is it changes an equation.

2504.36 -> It changes a way of thinking.

2506.96 -> So now, when developers are seeing IPv6 show up in logs,

2510.9 -> and there's a log parser that has only ever seen

2513.22 -> a v4 address in this spot, it now breaks,

2515.61 -> because there's a v6 address in this spot.

2517.57 -> It changes that thought equation from,

2519.817 -> "The network broke my thing," to, "Oh, I can expect this.

2524.247 -> "This is happening, and I need to go fix that right now."

2526.86 -> So the communication upfront is really powerful,

2529.25 -> because it changes that equation down the road.

2532.52 -> My next best practice would be to give serious consideration

2535.13 -> for Bring-Your-Own-IP.

2538.49 -> So BYOIP, Bring-Your-Own-IP,

2541.22 -> is the ability to use

2543.07 -> your public IPv6 addresses inside AWS,

2547.14 -> and the reason why this is important,

2549.23 -> at least for Netflix we ran into this,

2552 -> is we have centralized services,

2553.36 -> where we just wanna have a network policy that's,

2556.18 -> you know, allow anything to communicate to me.

2558.04 -> I don't care what exactly you are,

2559.85 -> but if you're a Netflix node, you can come in,

2562.15 -> and you can talk to me.

2563.84 -> With IPv4, that's actually quite easy to do,

2566.18 -> because the address space is summarizable.

2567.89 -> You can put a 10/8, or for us, 100.64/10 rule in there

2571.6 -> to allow everything in.

2573.52 -> But with IPv6, by default, you will,

2577.43 -> by default, if you use the Amazon provided IP space,

2581.19 -> that address space is discontiguous.

2583 -> So where you could put a one-line entry in there

2586.28 -> to cover your entire enterprise,

2588.42 -> now, you're looking at doing one line per VPC,

2591.82 -> and the limitless scale of IPv6 means

2594.06 -> you're gonna run into that limit incredibly fast.

2597.56 -> So you know, definitely give some consideration there

2600.45 -> for those of you pursuing v6.

2604.04 -> My next best practice is to overlay v6 with v4.

2607.51 -> Recognize that you don't need to move to IPv6 all at once.

2610.73 -> You can overlay v6 and v4, take your time,

2614.43 -> move traffic over slowly, and get used to it.

2620.18 -> So then when you're doing that,

2622.05 -> make sure that you match your IPv4 range rules

2624.89 -> with the corresponding IPv6 range rules.

2627.37 -> So in the similar situation,

2628.78 -> where I was talking about a security group

2630.19 -> that allows your entire enterprise,

2631.93 -> if you only have that for a v4 and not for v6,

2634.96 -> you're kind of hurting yourself,

2636.3 -> because you'll put out this stance,

2639.35 -> where, you know, v4 works, but v6 doesn't,

2641.8 -> and that doesn't help our cause at all.

2643.98 -> So as your first best practice,

2648.11 -> make sure you go through your security groups,

2649.91 -> and if you're using IP range rules,

2651.34 -> make sure they're updated

2652.64 -> with corresponding v6 and v4 networks.

2659.54 -> Okay, so I really hope that everybody is thinking

2662.13 -> in this audience, "How do I get started?"

2665.73 -> The first step, or actually I would say step zero,

2668.26 -> is to get IPv6 working

2670.26 -> on your development machines, your workstations.

2672.91 -> Think about your corporate environment,

2674.82 -> your home office, your corporate VPN.

2677.8 -> Those are generally the areas today

2679.18 -> that are lacking IPv6 the most.

2681.02 -> You should start there, and focus on those areas first.

2684.99 -> After that, go through and enable IPv6 on your edge.

2689.08 -> So if you're using an ALB, or an NLB,

2691.61 -> it's one click in the AWS Console to turn on IPv6.

2695.31 -> Two things to keep in mind there is

2696.7 -> to update security groups.

2697.72 -> If you have an IPv4 all rule,

2699.41 -> make sure you have the corresponding IPv6 all rule.

2703.96 -> DNS-wise, if you're using Route 53 alias records,

2707.21 -> make sure you have a quad A in addition to the A record

2709.81 -> to enable v6 there.

2712.29 -> At that point, then start focusing on your workloads in VPC.

2717.73 -> Again, overlay IPv6 with IPv4.

2721.44 -> Realize that you don't have to move everything

2723.97 -> at the same time.

2724.96 -> Overlay it, get comfortable.

2727.34 -> And then my recommendation is to consider

2729.31 -> working from the edge inwards.

2731.27 -> So generally, the edge of your network

2734.16 -> facing the internet and facing your customers

2737.45 -> has the most concrete set of network dependencies.

2740.06 -> So picking out that area to focus on v6 adoption

2744.67 -> has worked well for us.

2746.01 -> And this is in contrast to looking

2748.42 -> from the data persistence layer up, for example.

2754.4 -> All right, so I saved the last question for last.

2756.9 -> Oh, actually, I wanna go back and mention one more thing.

2759.39 -> So my stunning colleague Joel Kodama is doing a Chalk Talk

2763.35 -> that goes into much more detail on this section entirely.

2766.53 -> So his talk is later this week.

2768.43 -> It's NFX401.

2771.897 -> And I highly encourage everybody to check that out.

2774.56 -> All right.

2775.92 -> Now, how do you show IPv6 is worthwhile to your business?

2779.84 -> I hope you've been seeing a common theme here

2781.88 -> in the way that I'm talking today is that

2783.97 -> this is a business requirement.

2785.49 -> It's required for Netflix to continue to grow.

2787.99 -> It's not because IPv6 is cool, or because I wanna use it.

2791.103 -> It's because the business needs it.

2793.72 -> It comes down to simple economics.

2797.78 -> So first off, IPv6 is faster than IPv4.

2801.3 -> I'm not here to, you know, debate the merits of this.

2805.85 -> But you can go online and just Google the sentence

2808.33 -> and find lots of great articles.

2812.69 -> Anyway, so IPv6 is faster than IPv4.

2815.46 -> So if you don't at least have IPv6 enabled on your edge,

2817.98 -> your, you know, your edge ALBs and NLBs,

2820.28 -> you're probably penalizing your customers,

2822.41 -> your users of your service.

2825.31 -> And there's an economic and business advantage

2827.54 -> to providing a faster service.

2830.31 -> And what we're generally seeing today is that IPv6 is faster

2833.53 -> because of how many NATs there are in the world.

2835.96 -> So think about your home router NAT,

2838.39 -> your NAT in your corporate environment,

2840.06 -> your, you know, your edge firewalls in your office.

2843.55 -> NAT gateways as well are example of a NAT,

2846.88 -> and then there's lots of carrier-grade NATs as well

2849.44 -> within ISPs' networks.

2852.07 -> And what we're seeing is that

2853.42 -> these NATs are an expensive boundary.

2856.08 -> They're providing a performance penalty

2859.3 -> for IPv4 versus IPv6.

2863 -> But it's also expensive.

2864.33 -> It's expensive for a couple reasons.

2868.66 -> Ah.

2874.16 -> Excuse me.

2877.11 -> Public IPv4s are also, they're a scarce resource.

2881.87 -> So what we see today is elastic IPs are more expensive

2886.02 -> under certain circumstances.

2888.677 -> And what this means is that we end up optimizing

2891.04 -> to reduce the use of public IPv4s.

2893.71 -> And so, we have that expensive boundary

2896.55 -> between using NAT gateways and public IPv4s.

2900.66 -> The tiny bubbles model that I mentioned earlier

2903.12 -> also optimizes to reduce the use of public IPv4s.

2907.5 -> The theme here that we're seeing is that these models,

2911.23 -> they introduce middle boxes.

2912.7 -> They introduce things in the network to process packets

2915.67 -> to reduce the, to either continue the growth

2918.33 -> of private IPv4, and allow overlaps,

2921.36 -> or to, in general, reduce the use of public IPv4 addresses,

2925.12 -> and these middle boxes add cost.

2927.59 -> NAT gateways, I think there are some folks

2930.01 -> maybe in this audience that are, you know, very adamant

2932.26 -> about how expensive NAT gateways can be.

2936.21 -> They also add operational risk.

2939.539 -> You know, the NLBs and the PrivateLink approach require

2942.01 -> DNS tricks in order to use.

2944.92 -> So what we're seeing today is that

2948.03 -> extending the life of IPv4 is in general

2951.66 -> putting lots of NATs and lots of middle boxes

2953.35 -> in your network.

2955.979 -> And what I want is I want just a simple flat network

2958.79 -> that's just simply transport, and NATs have broken that.

2962.29 -> They've broken that model, and I just really want it back.

2967.44 -> So at Netflix, what we've been doing is

2969.35 -> we've been starting the journey to IPv6 and VPC,

2972.19 -> and so should you.

2973.7 -> What you're gonna see is that the cost of maintaining

2976.33 -> an IPv6 network is gonna grow superlinearly

2979.01 -> as you continue to grow in the cloud.

2981.25 -> You can either proactively address that today,

2983.5 -> or you can be a victim to that change later.

2987.32 -> So that's it, that's all I have for y'all today.

2989.586 -> (audience applauds)

2997.71 -> Thank you.

2998.543 -> My contact information is up on the screen

3000.81 -> for anybody who wants to chat more on this topic.

3004.53 -> I am also compelled to invite everybody

3007.14 -> to complete the session survey.

3009.17 -> I will personally read all your feedback,

3011.07 -> so you know, don't hold back and be honest,

3013.35 -> because at the end of the day,

3014.183 -> it's gonna help me be a better presenter

3015.41 -> and provide better content in the future.

3017 -> So again, you know, thank you, everybody.

3020.134 -> (audience applauds)

Source: https://www.youtube.com/watch?v=1DF6yIFIx14