AWS re:Invent 2022 - Deploy modern and effective data models with Amazon DynamoDB (DAT320)
AWS re:Invent 2022 - Deploy modern and effective data models with Amazon DynamoDB (DAT320)
Amazon DynamoDB is a fully managed NoSQL database that provides consistent performance at any scale. Customers like Disney+, Zoom, and Snap use DynamoDB to handle some of the largest applications on the planet. In this session, AWS Data Hero Alex DeBrie and DynamoDB Senior Principal Engineer Amrith Kumar walk you through key data modeling concepts for DynamoDB and share why DynamoDB architecture and implementation can help your applications scale seamlessly from 30 to 300 million customers while maintaining consistent, single-digit millisecond performance.
ABOUT AWS Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.
AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.
#reInvent2022 #AWSreInvent2022 #AWSEvents
Content
0.06 -> hey everybody I'm Alex debris I'm
2.52 -> excited to talk about Dynamo DB with you
4.5 -> today I'm an AWS data hero work a lot
7.08 -> with dynamodb and I've talked at re
8.639 -> invent the last three years about
10.44 -> dynamodb I'm excited because this year I
13.019 -> have someone that actually knows what
13.98 -> they're talking about with me up here so
15.66 -> I'll be talking with amrith Kumar amreth
17.58 -> is a senior principal engineer on the
19.74 -> dynamodb team they're they're two senior
21.66 -> principal Engineers on the dynamodb team
23.1 -> it's the highest current engineering
25.32 -> position on the team so lots of deep
27.9 -> knowledge that he's going to bring here
29.519 -> we'll we'll do a little bit of
31.08 -> infrastructure a little bit of data
32.579 -> modeling a little bit of everything in
33.66 -> this talk to help you learn how to how
35.64 -> to model well with dynamodb so to kick
38.1 -> it off Amber's going to get us going
39.239 -> here
40.94 -> okay so good afternoon my name is amrith
45.18 -> I'm one of the senior principal
47.28 -> Engineers on the dynamodb team been here
49.86 -> for about two and a half years
52.16 -> the talk is about data modeling and
56.1 -> I'm just going to give you an
57.3 -> introduction about dynamodb and then
59.219 -> hand it off to Alex and he's going to do
60.66 -> the rest of it
62.1 -> let's see if this works okay
64.619 -> so I'll do the overview I'll hand it off
66.54 -> to Alex to do the data modeling and then
68.76 -> I'll finish with some talk about
71.22 -> indexing and transactions
73.56 -> if you were expecting this is going to
75.299 -> be an in-depth architecture talk it is
77.58 -> not
78.78 -> there's a couple of links which I've
80.7 -> provided
81.659 -> a colleague of mine jasso did a talk a
83.82 -> couple of years ago that's a very good
85.56 -> talk
87 -> um Alex's book and a couple of months
90 -> ago
91.32 -> on the 10th 10th year of nanomodb we
94.86 -> published another paper to follow up on
97.259 -> the original Dynamo paper and I'd
99.84 -> strongly recommend all of these if
101.28 -> you're interested in architecture
103.979 -> so without further Ado I expect that
106.68 -> everyone here knows what I'm going to
108.84 -> say dynamodb is a key value in document
111.479 -> store
112.619 -> it is completely serverless it is secure
115.32 -> durable available
118.02 -> and the simple promise we give you is
120.06 -> that you can get predictable single
122.1 -> digit millisecond response times at any
124.079 -> scale and I'm going to talk to you about
126.84 -> some of how dynamodb does this
130.039 -> and then you can build applications
132.42 -> which can leverage these things so
134.04 -> that's the general idea of the talk
136.5 -> all right
138.239 -> so when we were putting this
139.68 -> presentation together Alex sent me this
141.599 -> tweet and by the way I strongly
144.72 -> recommend the five minute video which is
147.48 -> linked there it talks about how Snapchat
149.599 -> built their application
152.16 -> and it talks about how they use
154.14 -> kubernetes and how much data they store
157.92 -> but the thing which really caught my eye
159.54 -> when I saw this video
161.879 -> was that they Scan they scan about 2
164.459 -> billion rows a minute
166.019 -> this is one customer of dynamodb
168.86 -> 400 terabytes is a fairly large table
171.18 -> not the largest table by any means but
173.76 -> that gives you an idea of the kind of
175.92 -> scale to which an application can go 300
178.2 -> million daily active users 400 terabyte
180.36 -> table and so on but
182.4 -> scanning 2 billion rows a minute yeah
186.06 -> so I thought I'd give everyone a little
188.58 -> bit of a glimpse at the scale at which
191.459 -> nanomodb operates
193.379 -> so this is a simple thought exercise
194.94 -> okay
196.14 -> wake up in the morning you have
197.459 -> breakfast that's the important part of
199.319 -> the thought exercise
201.239 -> for every request which dynamodb does
205.019 -> put one sheet of paper on that pile
207.599 -> okay
209.04 -> and work at this till lunch
211.92 -> can anybody give me an idea how tall
213.48 -> this pile is going to be
215.159 -> shout it out
218.459 -> Empire State Building okay
222.06 -> anyone else
224.4 -> to the Moon all right
226.799 -> the answer is yes it is to the Moon so
229.56 -> did we send you the slides for you to
231.48 -> review in advance yeah okay thank you um
234.84 -> so in terms of raw numbers dynamodb does
238.379 -> about 10 trillion requests a day easily
241.68 -> on most days
244.019 -> um
244.739 -> well over that
246.42 -> and each one of those requests is
248.879 -> predictably single digit millisecond
251.28 -> latencies
253.019 -> when you build applications which need
255.54 -> this kind of guarantee they don't happen
257.82 -> by accident you literally have to
259.919 -> engineer your application to it whether
261.959 -> it's a relational database or it's a
263.88 -> nosql database there's engineering
265.919 -> involved and a big part of that is what
268.02 -> Alex is going to talk about which is
269.82 -> data modeling but in order to data model
272.4 -> your application correctly you need to
274.62 -> know something about how dynamodb works
277.199 -> so let's talk about that
280.62 -> um
281.16 -> how does dynamodb scale
283.8 -> simple answer is two things one we scale
286.38 -> horizontally
288.06 -> and the second is with eventual
290.1 -> consistency
292.02 -> if your database requires synchronous
294.56 -> rights and you cannot tolerate eventual
297.54 -> consistency
299.04 -> your ability to scale will be diminished
301.86 -> if you can only scale vertically your
304.32 -> ability or scale is diminished but the
307.139 -> good news is most applications
310.08 -> which need to scale can tolerate both of
313.08 -> these you can scale horizontally and you
315.78 -> can tolerate eventual consistency
318.479 -> in dynamodb all of your data is stored
320.759 -> in tables and very much like a
323.34 -> traditional relational database
325.38 -> you need to have a primary key for every
327.72 -> item by the way we don't call them rows
330.06 -> we call them items so
332.52 -> if I use them interchangeably my
334.74 -> apologies my pedigree is relational
336.78 -> databases before I came here
339.139 -> so we store data in items
342.419 -> and being a nosql database there is no
345.78 -> strict schema in this sample data notice
348.96 -> that each item has slightly different
351.3 -> structure
352.44 -> but there is one thing which is required
354.36 -> that is the primary key
356.639 -> when you create a table in dynamodb we
359.1 -> ask you for two things well three we ask
361.56 -> you for a credit card that's different
363.419 -> um
364.199 -> you need a table name
366.72 -> and you need a primary key that's it
369.6 -> and the primary key can either be just a
371.82 -> partition key
372.96 -> or it can be a partition key and a sort
375 -> key here I've shown you one with just a
377.34 -> partition key and what we do with that
380.039 -> is we store your data
382.139 -> sharded on the storage nodes by
384.78 -> Computing a cryptographic hash on the
387.84 -> partition key
389.88 -> items which have which fall within a
392.759 -> range of hashes go to one storage node
395.039 -> that's easy
396.3 -> we try to keep each keep each partition
398.58 -> at about 20 Gigabytes
402.24 -> so whether your table happens to have
404.94 -> 400 terabytes or more your partition
408.479 -> size is limited
410.46 -> if there's a lot of traffic on your
412.199 -> table we we may even have smaller
414.539 -> partition sizes
416.039 -> that's one of the ways we give you
417.6 -> predictable response time
419.759 -> the thing we do really quickly 10
421.74 -> trillion times a day
423.6 -> is look at a partition key
426.12 -> figure out which partition it's on
428.4 -> go to the operation and return to you
431.28 -> over and over again
433.259 -> right
434.46 -> so
435.9 -> if I were to show you the same data this
438.18 -> way
439.259 -> three storage notes and a bunch of rows
443.34 -> dynamodb is also a region is a regional
445.62 -> service and I said it's durable and
447.66 -> highly available
448.8 -> so we replicate all of your data three
451.139 -> ways
452.4 -> every partition
454.56 -> there's three copies of it and each copy
457.319 -> is in a different availability Zone
459.539 -> so
460.74 -> if you're requesting your data and your
463.199 -> application happens to be in
464.58 -> availability Zone one we will try and
466.68 -> serve it from availability Zone one
469.319 -> and so on but it also means there's
472.199 -> three copies if one of them
474.599 -> goes away for some reason we can always
476.759 -> reconstruct it
478.199 -> and we'll talk about some of the things
479.819 -> which we do to make sure that that
481.8 -> actually works so yes we have three
483.72 -> copies of the data this is how these
485.46 -> partitions are laid out
488.58 -> by the way in case I'm going too fast
490.74 -> somebody just wave or stop me
493.86 -> um
495.419 -> I did also mention that dynamodb is
497.88 -> completely serverless
499.979 -> when you provision a table
502.08 -> we didn't ask you how many servers you
504.66 -> want to provision we do all that for you
506.34 -> oops can't do that
508.919 -> so on these storage nodes
511.68 -> alongside your data
513.659 -> is data for another customer
517.74 -> you may be running an application and on
519.959 -> the same storage node there may be some
521.459 -> of that 400 terabyte snap table
524.039 -> it's our job to make sure that every
526.62 -> partition
528.3 -> gives
529.5 -> customer predictable single digit
532.26 -> response time so we move these
533.76 -> partitions around on a regular basis and
536.459 -> yes there's three availability zones
539.64 -> there's no guarantee that you will be
543.6 -> getting a storage node with the same
545.7 -> partition
547.26 -> on a different availability Zone it's
549.06 -> completely random
550.98 -> okay you make a request to your table we
554.04 -> figure out which partition you're asking
556.2 -> for we get you your data that's
557.94 -> basically what we do
561.18 -> all right
563.519 -> so let's talk a little bit about the
565.019 -> architecture this slide is going to
566.22 -> build from left to right so on the left
568.62 -> you have all of the clients
570.54 -> so your client could be something which
572.1 -> is running on a mobile device it could
574.14 -> be another application running in AWS it
576.48 -> could be ec2 it could be kubernetes it
578.22 -> could be a Lambda whatever it is
580.38 -> dynamodb is a regional service
583.68 -> whether you use the SDK or you directly
586.32 -> talk to a restful endpoint the first
588.54 -> thing you do is resolve an IP resolve a
591.06 -> fully qualified domain name
592.98 -> so if you were in Us East one
595.62 -> region becomes Us East one
598.14 -> if you're in Usos too similar
601.38 -> when you resolve that
603.24 -> you will get an end point which points
605.7 -> to a collection of load balancers so in
608.459 -> this case I'm showing you that you
610.56 -> resolve this fqdn and you got a load
613.019 -> balancer in az2 all right
616.44 -> behind every load balancer is a
618.66 -> collection of hosts we call them request
621 -> routers
622.38 -> and there's a collection of request
624.36 -> routers which we call a gaggle
626.48 -> Collective is a gaggle and the job of
629.64 -> the request router which is completely
631.2 -> stateless is
633.3 -> of course between the load balancers and
635.399 -> the request router we do SSL termination
637.26 -> so your data in flight
640.98 -> your data in Flight let me see if I can
642.839 -> get this video pointer to work there you
644.459 -> go
645.42 -> nope I guess not the data in transit is
648.779 -> going to be SSL encrypted
651.24 -> it gets to a request router and the
653.339 -> request router's job is three things
655.98 -> first
657.6 -> is the request properly signed by the
661.14 -> user
663.66 -> second if it is properly signed and IT
667.019 -> addresses a particular table does that
669.12 -> user have access to the table
672.06 -> and if if the user does
674.82 -> does the user have access to the Keys
676.86 -> used to encrypt the data for that table
678.779 -> three things 10 trillion times a day
682.62 -> get a request
684.18 -> authenticate authorize
686.82 -> make sure you have keys and then
689.64 -> if you do
691.019 -> forward the request to a story mode
693.42 -> which is where the data actually lives
696.18 -> if you're doing a get
698.399 -> send it to the storage node the storage
700.74 -> node stores the data encrypted
702.959 -> so the storage node will decrypted
705.24 -> and send the data all the way back
709.279 -> if your application happens to be an az2
713.16 -> we will try and keep this entire thing
714.839 -> in az2 but remember
717.66 -> we also have the same infrastructure in
721.14 -> the other two azs
724.14 -> so in regions which have three azs this
726.72 -> is what you look like in regions which
728.94 -> have more than three azs we're in more
731.16 -> than three azs
732.72 -> and in the middle of the box there
734.82 -> on the middle of the slide there there's
736.8 -> these other hosts which we manage which
739.14 -> are the infrastructure metadata what
742.8 -> tables do you have
744.48 -> for each table where are the partitions
746.64 -> for that table and we cache a bunch of
748.92 -> this information so when you make a
751.26 -> request
752.399 -> we look up the partition map
755.399 -> we find out where the data is and we
757.44 -> send it to that storage
759.54 -> over and over again that's what we do
762.18 -> okay
765.54 -> um Auto admin is a component which is
768.06 -> listed in the middle there
770.04 -> I told you we try to keep partitions at
772.019 -> about 20 gig
773.7 -> if you insert a bunch of data and your
775.74 -> partition starts to grow
777.54 -> Auto admin will realize that and will
780.36 -> split your partition into two partitions
782.88 -> and put them into different places
784.86 -> update the partition map completely
787.079 -> transparent to you
789.06 -> you don't have to do any of that stuff
791.519 -> and we'll talk about transactions later
793.68 -> so I'll skip that one for now
797.94 -> if your requester was an az3
801.12 -> and you went to a request router in az3
804.779 -> we'll try and serve your request in az3
806.88 -> but
807.959 -> the request router in az3 can talk to a
810.12 -> storage node in az1 or az2 and there are
812.76 -> some cases where we actually do that
814.399 -> like rights or if you do a consistent
817.44 -> read we may have to reach across the
819.12 -> boundary
820.98 -> a lot of time thinking about
823.62 -> fractions of a millisecond
825.959 -> and the time difference between going
828.3 -> across AZ boundaries or within an AC and
831.36 -> we try to make sure that you get
832.92 -> predictable response times so those are
835.44 -> the things we do so you can build on top
837.48 -> of them
839.399 -> okay
841.38 -> I did mention that we were completely
843.18 -> serverless
844.26 -> so we have three azs worth of capacity
846.72 -> and something happens
848.82 -> maybe we do a game day we take an easy
851.16 -> out
852.54 -> there is enough capacity in the other
854.339 -> two azs to serve all of your requests
859.74 -> and we build and we engineer and we test
862.38 -> to this on a regular basis which means
866.22 -> this may be a little bit hard to see but
867.779 -> at the bottom right there's a little X
869.16 -> on the storage node we are continuously
872.16 -> deploying software
874.32 -> if we can take an entire easy out
876.959 -> we can take a storage unit anytime we
879.18 -> want
880.079 -> we can take a request router out anytime
882.18 -> we want
883.68 -> and it is there's no maintenance windows
886.86 -> because of this it's completely
888.48 -> serverless no maintenance windows and
890.579 -> all of this is completely invisible to
892.8 -> you as a user
894.54 -> and of course we do the same thing with
896.16 -> load balancers so
898.56 -> customers like snap
901.38 -> or Dropbox or
903.92 -> see Amazon themself
908.339 -> engineer applications and knowing how
910.74 -> this works
912.66 -> it's important for you to model your
914.579 -> data in a way that exploits this
917.04 -> architecture
918.18 -> so with that
920.699 -> let me hand it over to Alex
923.699 -> thank you
925.92 -> awesome thank you Amarth and yeah I love
927.899 -> that infrastructure stuff partly because
929.699 -> it's super interesting so go check out
931.56 -> those other resources that amberth
932.82 -> mentioned especially jasso's 2018 talk
934.92 -> and the 2020 2022 paper is really cool
937.68 -> if you want to get stuff but like
939.66 -> Amber's saying like this is really
940.98 -> useful for you in modeling your dynamodb
943.62 -> applications so that's what I'm going to
945.24 -> talk about here and you know if that
947.22 -> stuff flew by you we'll sort of refer to
948.899 -> it in different places and reinforce
950.639 -> where that that helps you in your data
952.56 -> modeling right so we'll do some data
953.699 -> modeling here I always like to go with
955.5 -> some sort of example I've done
956.579 -> e-commerce or social media in the past
958.44 -> we're going to do a game this year right
959.76 -> so last year Amazon Studios released new
962.519 -> world which is this you know multiplayer
964.86 -> online RPG game using a lot of AWS
968.339 -> Services behind the scenes and Werner
970.079 -> got up on stage during his keynote and
972.36 -> talked about all the different services
973.32 -> including dynamodb one of the things the
976.38 -> service is doing is you know if you're
977.94 -> playing every 30 seconds it's going to
979.44 -> increment your play time just to keep
980.699 -> track of how much time you've been
981.72 -> playing right so they're doing 800 000
983.76 -> rights to Dynamo every 30 seconds which
985.92 -> as we've seen is is really nothing for
988.26 -> Dynamo but but seems pretty big to me so
990.899 -> we're going to be talking about that
992.82 -> um as the example here I always like to
994.5 -> start off with some terminology so four
996.48 -> key terms table item primary key and
999.36 -> attributes we'll do it in the context of
1001.399 -> that game so we'll start off with some
1003.44 -> users in our game if you see we have
1005.3 -> some users here Amarth myself Janelle
1007.639 -> and Chad uh four records and they're
1010.1 -> going to be contained in what's called a
1011.899 -> table right so similar to a table in a
1014.54 -> relational database but as we're going
1015.8 -> to see different in a lot of ways as
1018.019 -> well
1019.519 -> now if you look at an individual record
1021.019 -> in a table as amorth mentioned that's
1022.759 -> going to be called an item so this you
1024.439 -> know amberth's user record that's going
1025.819 -> to be an individual item in your table
1027.74 -> similar to a row in a relational
1029.9 -> database
1031.88 -> when you create your dynamodb table you
1033.98 -> have to specify what's called a primary
1035.66 -> key and every item you write to your
1037.52 -> table must include that primary key and
1039.079 -> must be uniquely identified by that
1040.52 -> primary key so on the left hand side
1042.38 -> here we have the primary key for our
1043.819 -> table it's username that's going to
1045.62 -> uniquely identify every single user in
1047.54 -> our table right we're not going to have
1048.559 -> two users with the same username
1051.08 -> it's gonna be similar to primary key in
1052.64 -> a relational database but but different
1054.62 -> in a lot of ways as well
1056.6 -> in addition to that primary key we also
1058.52 -> have other attributes you can have here
1060.26 -> right and you can see we have a lot of
1061.94 -> different attributes similar to columns
1063.919 -> in a relational database with a big
1065.059 -> difference being you're not going to
1066.32 -> specify these up front so dynamodb is
1068.72 -> schemaless there's not going to be a
1069.919 -> schema that Dynamo is going to enforce
1071.539 -> for you outside of that primary key
1074.299 -> if you look at these attributes you can
1075.799 -> see we have some simple attributes right
1077.539 -> simple scalar strings numbers like class
1080.059 -> gold total Play Time Guild things like
1082.22 -> that but you can also have complex
1084.08 -> attributes you can have lists and sets
1086.539 -> and Maps we'll talk a little bit about
1088.1 -> when you'd want to use those
1091.039 -> let's talk a little bit more about the
1092.78 -> primary key because it's so important to
1094.46 -> how you work with dynamodb there are two
1097.039 -> types of primary key in dynamodb so
1098.9 -> there's first of all what's called a
1100.039 -> simple primary key which has one element
1101.539 -> called a partition key and there's
1103.28 -> what's called a composite primary key
1104.84 -> that has two elements that partition key
1106.82 -> and the sort key and that user table we
1109.58 -> looked at before that's an example of
1111.14 -> the simple primary key right it has just
1112.64 -> that single element that partition key
1114.2 -> that's unique identifying every record
1115.7 -> in your table this is going to be great
1117.98 -> for simple key value like access
1120.38 -> patterns you're only operating on one
1122 -> record at a time
1123.74 -> but if you have more complex access
1125.66 -> patterns then you might want to use a
1127.34 -> composite primary key and you know in
1129.14 -> addition to users in our application
1130.82 -> we're also going to have quests so let's
1132.38 -> look at a table with a composite primary
1134.66 -> key here we have some quests that our
1136.28 -> different users are going on you can see
1138.32 -> we're using this composite primary key
1139.88 -> it has two different elements here right
1141.38 -> we have that partition key of username
1143.66 -> but we also have that sort key of Guild
1146.12 -> ID
1148.28 -> now as you're looking at these records I
1149.78 -> just want to point out if you look at
1150.799 -> the top three records here those are
1152.179 -> three separate items there but they all
1154.64 -> have the same partition key so you can
1156.2 -> have multiple records that have the same
1158 -> partition key and if you're using this
1159.74 -> is nosql workbench this is what I use
1161.36 -> for modeling a lot if you have multiple
1163.4 -> records with the same partition key it's
1164.84 -> going to sort of merge that cell
1165.919 -> together because they're going to be
1167 -> contained near each other on the the
1168.799 -> same partition in most cases
1171.919 -> um but when you're working with that
1173 -> composite primary key there's sorry this
1174.98 -> is going to be called an item collection
1175.94 -> so if I refer to an item collection that
1177.799 -> refers to you know a set of items that
1179.66 -> have the same partition key
1182.36 -> and while you can have records with the
1184.34 -> same partition key it's the combination
1186.08 -> that still must be unique so you can
1187.7 -> have records with the same partition key
1189.02 -> but the sort Keys need to be unique
1190.66 -> within that combination
1194.66 -> if we look at the two types of primary
1196.52 -> key notice that both of them have a
1198.86 -> partition key right and that partition
1200.179 -> key is really important we can go back
1201.559 -> to what amworth was talking about when
1203.6 -> that request comes in it hits that load
1205.28 -> balancer it hits that request router
1207.32 -> right and that request router is is
1208.76 -> responsible for routing it to the Right