AWS re:Invent 2022 - Deep dive on Amazon S3 (STG203)

Aug 16, 2023

AWS re:Invent 2022 - Deep dive on Amazon S3 (STG203)

Amazon S3 and Amazon S3 Glacier storage classes provide developers and IT teams with cloud object storage that delivers industry-leading scalability, data availability, security, and performance. This session summarizes Amazon S3 and reviews the S3 storage classes and key features across security, access management, storage management, and more. Learn about the S3 durability model and how it differs from other cloud providers. Also explore common use cases, such as data lakes, mobile apps, backup and restore, archive, enterprise applications, IoT devices, and analytics.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

6.09 -> - Good evening everyone,

7.71 -> and thank you so much for joining us today.

9.93 -> My name is Oleg.

11.04 -> I will be joined by Sally

12.39 -> who will take over halfway through the presentation.

15.15 -> And before we begin, we'd like to thank you for being here.

18.63 -> It's always wonderful

19.5 -> to meet customers and colleagues in person,

21.72 -> and interact, and just exchange ideas, or just chat.

24.39 -> And it's been a little bit of a rough few years

27.3 -> for all of us, so it's wonderful to have you here.

30.15 -> Also, thank you for being here this late in the day.

33.12 -> I'm sure we're all a little bit tired,

34.41 -> so I'm gonna try to make it as entertaining as possible.

37.47 -> So, welcome to Deep dive on Amazon S3.

40.11 -> We do this type of session every year,

41.94 -> and every year, we discuss slightly different subjects,

44.22 -> all relevant, all interesting.

46.65 -> But this year we decided to take a step back,

48.48 -> way back, and ask ourselves,

50.137 -> "What would be the most important things,

52.08 -> most impactful things,

53.37 -> to focus on during Deep dive on Amazon S3?"

56.7 -> Because the subject is, well, as deep as it is wide.

61.68 -> We could have talked about many, many features in Amazon S3,

64.77 -> new releases, customer stories,

66.6 -> maybe even share some implementation details

68.91 -> and best practices.

71.37 -> But then we realized this is a really,

73.74 -> really big conference,

75.48 -> and all of the subjects are covered in depth

77.82 -> in other sessions, in key notes,

79.59 -> in builder sessions, and so on.

81.48 -> And I hope we, you guys had time to attend those

84.6 -> or will have time to attend those.

86.91 -> And so given that a lot of subjects already covered,

89.19 -> what would be the most important thing for us to focus on?

91.8 -> And so in the end, we realized that the most important thing

95.55 -> for us to focus on is fundamentals.

98.91 -> Those cross-cutting fundamental principles and mechanisms

102.51 -> that underpin everything that Amazon S3 does,

105 -> and actually, everything that any storage system does.

109.02 -> And there are actually quite a few.

112.56 -> And so after much thought, we focused on four.

115.29 -> Now, they're listed in a specific order,

116.97 -> but it by no means indicates they're import.

119.19 -> We just needed to listed it in some order.

122.28 -> All of them are very important

123.93 -> and existential for storage system.

126.21 -> Let's start with durability.

127.35 -> It is a core basic property of any storage system

130.77 -> that basically states that after an object, or a file,

135.06 -> or any kind of data that is written into it,

137.16 -> it can be eventually retrieved without alterations

140.25 -> and will be retained there.

142.41 -> Availability and performance,

144.12 -> availability and performance are really the,

146.1 -> exactly the same dial.

148.86 -> As you read and write data, you typically care a lot

151.95 -> about how fast you can do so.

153.72 -> And you also care how well you can parallelize your workload

157.26 -> to achieve better performance.

158.82 -> So, of course, we'll focus on that.

160.8 -> Now, most data is stored to benefit businesses or research,

164.22 -> and those institutions care deeply about cost management.

167.46 -> And so, we would be remiss if we didn't address how we think

170.82 -> about that with S3.

172.44 -> And finally, and inevitably, and existentially, security.

177.48 -> You want to be intentional

178.62 -> about who can read and write data,

181.98 -> and probably more importantly, who can't.

185.31 -> There are many other aspects we could have considered,

187.68 -> but we only have so much time,

189.197 -> and so we decided to focus on those four.

192.653 -> Durability is a natural place to start with,

195.36 -> because storage system.

198.99 -> The most important requirement for any storage system is

201.39 -> to retain your data, storing it without losing it.

204.78 -> Now, you have probably have heard that Amazon S3 is

207.54 -> designed through 11 9s.

210.03 -> We will discuss exactly what this means,

211.68 -> where this number came from,

213.21 -> and why it doesn't describe the full picture,

215.64 -> and why durability is more

217.32 -> than just this particular property.

220.23 -> And so to do so, let's go back,

222.24 -> as promised, to fundamentals.

225.69 -> Cloud is somebody else's computer,

227.94 -> and so all of your data is eventually written

229.98 -> on some kind of media.

231.42 -> And the first commercially available,

233.1 -> large scale media was a hard drive.

235.68 -> Now, it was pretty bulky.

238.23 -> I've seen some picture,

239.063 -> I've seen some pictures with people next to them,

240.78 -> and it requires three or four people to carry it.

242.433 -> It's the size, probably, vending machine.

245.28 -> And it retained a lot of data for its time,

247.5 -> close to five megabytes.

248.85 -> It was also very, very expensive.

250.44 -> It was close to $50,000, and it was back in 1956.

254.73 -> So, that was then.

257.37 -> A lot of things have changed now, but as we will discover,

261.84 -> some actually remain the same, and it's very significant.

264.99 -> Hard drives have become more compact,

266.7 -> more dense, and substantially cheaper.

269.4 -> Certainly not $50,000 for five megabyte.

272.85 -> And hard drives, to this day,

275.55 -> remain the work horses of the of the cloud.

278.347 -> And this, and while we use other media,

280.35 -> such as flash and SSDs, hard drives are very attractive

285.24 -> due to their cost and efficiency.

288.15 -> One thing they didn't change about hard drives, though,

290.73 -> is they are mechanical devices.

292.65 -> They have spinning discs, and they have a head

294.54 -> that travels along the spinning disc

296.4 -> and read and writes the data.

298.65 -> And so, they have, like any mechanical devices,

302.7 -> like any piece of equipment really, hard drives can fail.

306.03 -> Now, it's not a very common event, but it does happen.

308.82 -> Here are some reasons why hard drives may fail.

311.88 -> And they'll indicate that mechanics, electronics,

315.24 -> or perhaps, the electric aspect of it,

317.49 -> one way or another went wrong.

319.56 -> Stiction is an interesting phenomenon,

321.72 -> which I learned about recently,

323.19 -> which is basically a phenomenon of the head sticking

326.07 -> to the actual spinning disc and causing it to fail.

331.11 -> And when we reason about hard drive failures,

333.24 -> we use the acronym AFR, annualized failure rate.

339.36 -> And AFR varies widely, but these days it's close to,

343.74 -> it's typically measured in singular percents,

346.38 -> sometimes subsingular percents.

348.69 -> So why does all of this matter?

350.64 -> Well, your data needs to be stored somewhere,

353.85 -> and when you start to,

355.107 -> and and if you start building a storage system,

358.35 -> in the end, some kind of media like this hard drive,

360.45 -> will have to be used.

361.62 -> So, how would you try to build it at scale,

363.78 -> this hypothetical storage system, dealing with failure?

368.79 -> Let's consider maybe slightly exaggerated example.

371.97 -> Let's say we have our micro data center

374.73 -> that has 24 hard drives.

377.1 -> Now, something really bad happened and seven of them failed.

380.31 -> In reality in production systems, obviously,

382.17 -> this high rate of failure is very, very unlikely.

385.44 -> How do we deal with this?

387.42 -> Well, the approach has been around forever,

389.73 -> and it's fairly obvious.

390.72 -> Data duplication, right?

391.83 -> Redundancy.

392.97 -> You store your data on, or chunks of your data,

395.52 -> in multiple locations at once.

397.53 -> Something like this.

399 -> The color in this case indicates the the exact data bit.

403.41 -> And in this case, our replication factor is two, or three,

406.35 -> depending on how your reason about it.

407.82 -> But basically, the same data chunk is stored three times.

413.37 -> So, let's see how this particular distribution fared

415.98 -> for our failure pad.

417.99 -> Look at the yellow hard drives.

419.22 -> They actually look green, but they're meant to be yellow.

423.18 -> Now, it turns out they were completely spared by failure.

426 -> And so, when we access data, we can read it.

429.54 -> For example, if we're just reading the data,

431.37 -> we can read it from any hard drive.

433.14 -> That's great.

434.64 -> Blue ones didn't do so well,

436.68 -> but still very, but still durable,

438.69 -> because we still have one replica

440.25 -> from which we can read the data.

442.5 -> The two others are out,

443.52 -> and we probably need to do something to replace them.

446.58 -> And here we need to be very careful.

448.77 -> So, when the customer request arrives,

450.33 -> and we wanna fetch some data,

451.53 -> we route around the failed hard drives.

453.75 -> Otherwise, the customer will experience,

455.22 -> well availability issue in this case,

457.26 -> but it's also a very important factor.

460.29 -> And finally, the red ones.

461.7 -> As you see, they haven't really fared very well.

463.95 -> All three just happen to have failed.

465.96 -> We didn't distribute our data very well.

467.73 -> We didn't replicate it across, probably,

469.44 -> appropriate variety of media,

472.41 -> or perhaps we just waited too long

474.03 -> and allowed a lot of drives to fail.

476.73 -> So what does all of this mean?

477.75 -> These are fairly self-evident and obvious observations.

481.89 -> Well, at scale, media or anything really,

485.85 -> but media in this particular case, especially if it fails,

488.58 -> is relatively hard to manage.

490.29 -> You need to constantly distribute your data

494.43 -> across multiple media devices.

497.34 -> We haven't even covered the fact

498.66 -> that in order to implement this replication

501.27 -> that we talked about,

502.103 -> each write needs to lend to on multiple devices,

504.63 -> and you need to read about consistency.

506.76 -> And you need to be very careful about routing your data,

509.58 -> routing your request, to only appropriate devices.

513.3 -> Now, how do we solve this problem in Amazon S3?

516.12 -> You probably have heard of this,

517.11 -> but we will go very quickly through this exercise.

520.08 -> We also rely on redundancy,

521.58 -> but a specific kind of redundancy,

523.62 -> specifically erasure coating.

526.41 -> How does it work?

527.25 -> Well, for any object,

528.66 -> we start by chopping it up into chunks,

530.91 -> which we called shards.

531.81 -> They typically equally sized.

533.04 -> In this particular case, we have seven, seven.

536.25 -> We then use a specific algorithm.

537.84 -> And these types of algorithms have been around

539.97 -> since the sixties and seventies,

541.59 -> and they were born in the context

542.97 -> of reliable signal transfer.

545.46 -> And they're called parity shards,

547.44 -> additional calculated shards.

549.24 -> Why, why is it this way?

550.92 -> Well, as it turns out, you can assemble your original object

554.61 -> from any seven of these shards.

557.55 -> You can take all of the primary ones,

560.67 -> you can take all of the parity ones

563.13 -> and mix and a couple of primary ones, and so on.

565.5 -> You have some choices.

567.21 -> So, it's actually, the system is incredibly flexible

570.75 -> because you can select which shard which shards to use.

572.547 -> And this approach is really powerful, because it allows you

575.79 -> to control levels of redundancies beside beyond just,

579.66 -> we replicate data two times, three times, four times.

582.18 -> It actually really helps also

584.01 -> with durability and availability goals,

586.35 -> because it allows to work around, very effectively,

589.14 -> around things like drive failure,

590.91 -> availability zone failure, and so on,

592.71 -> relatively transparently to the customers.

595.47 -> Now, how does it work in practice?

597.69 -> Let's look at this pretty video.

599.49 -> So, when the customer puts an object in Amazon S3,

601.77 -> we erasure code it into small little chunks,

604.08 -> and we spread it as far, as wide as we can.

607.71 -> What does it mean?

608.543 -> Well, we distribute data, obviously,

609.74 -> on different hard drives.

610.95 -> We tries to, we try to place them

612.84 -> against different racks, different data centers,

615.57 -> different availability zones, and so on.

618.18 -> Conversely, on GET, we reassemble these shards

622.14 -> and put together into an object.

624.09 -> This approach has very substantial benefits.

626.85 -> One of them is any individual customer's data,

631.2 -> specific object, does not only occupy,

634.62 -> is spread across multiple media devices.

637.98 -> And any individual disc

640.17 -> actually contains a small part of it.

642.54 -> And what this means is that certain failure modes,

644.97 -> such as disk failures, SYS-E failures, and so on,

648.21 -> even in the presence of those,

649.68 -> we can still assemble data even if we don't attempt

652.44 -> to repair our hard drives.

656.22 -> I just mentioned repair.

657.42 -> Clever routing and distribution is not enough.

660.24 -> We need some kind of a complementing mechanism

662.1 -> that actually recovers your media.

663.66 -> Otherwise, that exaggerated picture

665.88 -> that I just presented may just become reality.

669.45 -> And so we call this process repair.

670.89 -> It's fairly common sense,

674.16 -> but it has some really interesting details

676.26 -> that are worth discussing.

677.55 -> So, how would we reason about it?

679.56 -> Well, as we discussed, hardware will fail over time,

682.56 -> not only hard drives, any equipment will,

685.77 -> network equipment, individual facilities, and so on.

690.24 -> So what, in order to keep up with this level of failures,

694.11 -> we need to measure failure rates

696.18 -> and then implement some kind of a mechanism

699.18 -> that repair repairs your media.

701.4 -> Now, the actual mechanic of repair can vary.

705.12 -> Like in practical terms,

706.35 -> we don't, obviously, pull out the broken hard drive,

708.96 -> and push in another one.

710.49 -> Most of the time, we maintain a pool

712.41 -> of the additional capacity, which then enters the fleet

715.44 -> and receives the replica of your data.

717.93 -> But the most important thing is that repair rates need

720.54 -> to be keeping up with the failure rates

722.4 -> to maintain durability.

724.74 -> You also need to track the worst case of failure rates,

728.1 -> because in order to ensure high durable posture

731.46 -> and strong durable posture,

732.81 -> you need to repair even in worst case scenarios.

735.24 -> And so this actually, as it turns out,

737.28 -> where 11 9s math comes from.

740.217 -> The AFR of the devices we use,

742.65 -> in the repair fleet we're utilizing,

744.51 -> in the repair mechanism we're utilizing,

746.373 -> allows us to keep up with any data decay

749.61 -> that might be happening.

751.86 -> So, let's look what the architecture

754.8 -> of such a repair system may look like.

756.45 -> It's also fairly simple, at least at this very high level.

760.71 -> So, here is the durable storage where your object live.

764.4 -> And so, what we do is we monitor the failure,

766.35 -> and we also maintain repair fleet.

769.35 -> And a repair gets failure detectors, report anomalies,

774.06 -> repair fleet picks them up, and goes ahead

776.13 -> and does whatever needs to be done

777.42 -> for for the data shard to be restored.

781.05 -> And as I mentioned,

781.883 -> this is what the magic number 11 9s come from.

784.56 -> But this is not actually enough,

786.6 -> because the world is a big and messy place,

788.85 -> and things change all the time.

790.71 -> And we have to think about things

792.15 -> that sometimes are not very obvious.

793.71 -> For example, it may be very hot weather

796.02 -> where the data center is, and maybe there is a power outage

800.28 -> in that particular moment.

801.84 -> And thus, and maybe cooling system lost its redundant.

804.45 -> It's a very unlikely event, but they do happen.

807.48 -> And in this case, the the level of failure may increase,

811.83 -> because some media devices can be temperature sensitive.

814.35 -> And so we need to be able to detect it and respond to it.

818.79 -> And we do.

820.53 -> Okay, so we've talked about redundancy,

823.92 -> we've talked about distribution of data,

825.78 -> we talked about, we talked about routing,

827.55 -> and we talked about repair.

829.41 -> Oh, we're done right?

830.61 -> We're durable at this point.

833.22 -> It's most more complicated than that, of course.

835.86 -> Of course, it was a trick question.

837.93 -> So, but there is something really interesting

840.21 -> what we just did without maybe realizing it.

843.18 -> We actually just created a threat model.

847.71 -> We ask ourselves, "What can go wrong?"

850.14 -> And then we ask ourselves, "How we gonna deal with that?"

852.72 -> And this is a mechanism that is very commonly used

856.59 -> in security, for decades,

858.24 -> when individual attack vectors are getting examined,

860.97 -> and mitigations get, and mechanisms get put

863.28 -> in place to address them.

865.05 -> So we ask ourselves, for example,

866.7 -> questions like, "Who is the adversary?"

868.507 -> "What are their capabilities?"

869.767 -> "What are they trying to do to us?"

871.65 -> And this is like,

872.483 -> in security context, pretty reasonable question.

874.29 -> But what does it mean in durability context?

876.51 -> In this particular situation,

878.43 -> the hardware failure, the adversary, is failure and decay.

882.75 -> Well, it's a, almost a super villain name,

885.12 -> but the reality is devices, any sort of devices

888.09 -> and network facilities, experience failure.

890.61 -> And so they can make our drives and servers fail

892.86 -> and destroy enough hardware.

895.26 -> Great, so what is the next step?

897.18 -> We discussed it, mitigation.

899.61 -> So we use data redundancy, failure detection,

902.28 -> scalable repair, clever routing.

904.8 -> Again, are we done?

907.89 -> Well, turns out you can't just do it once,

910.86 -> because the world is a big and messy place,

913.32 -> and a lot of circumstances change.

915.45 -> And so you, you can't just put mitigations

919.05 -> in place and stop there.

921.03 -> You have to keep iterating, measuring, improving.

925.05 -> And we call, at Amazon, we call this process a mechanism.

928.32 -> It's a complete process, a virtual cycle of sorts,

932.4 -> that reinforces and improves as it operates.

935.16 -> So, you constantly ask yourself, "How effective are we?"

937.567 -> "How cost effective are we?"

939.007 -> "How performant are we?"

939.84 -> "How reliable are we?"

941.107 -> "Resilient?"

941.97 -> And any other interesting cross-cutting concern

944.88 -> you may consider and constantly improve the processes.

948.39 -> And as the situation changes, and the system grows,

951.42 -> and the customer pattern changes, and S3 has existed

955.2 -> for close to 15 years, maybe more.

958.77 -> Sorry, I cannot do math right.

959.79 -> Actually, more than 15 years now.

962.13 -> So, a lot of things have changed since then for us.

965.97 -> And so this is why we don't only stop there.

968.94 -> So, we implement the mechanism.

970.137 -> We, just now we discussed the sort

971.7 -> of mechanism we can implement

973.62 -> for, specifically, hardware repair.

976.86 -> But we actually consider much more than that.

979.74 -> Here's just some other threats we consider.

982.17 -> We discussed facility failure, hardware repair data.

985.35 -> We're gonna talk a little bit about data corruption

987.87 -> and other failure modes.

989.07 -> But there are several things here which may be surprising.

992.22 -> Bugs, for example, and operational practices.

994.62 -> How is that relevant to durability?

997.02 -> Well, your hard drives may be perfectly healthy.

999.18 -> Your entire data center can be perfectly healthy.

1001.07 -> There is no fires anywhere,

1002.36 -> and all of the bits are piping across just fine.

1004.91 -> But you had a bug in your storage node,

1006.761 -> and you just deployed it.

1008.537 -> And that node starts flipping bits,

1011.36 -> or calculating incorrect checksums,

1013.55 -> or do some other terrible things to the data,

1015.53 -> as a result of which,

1017.3 -> your durability stance has been weakened.

1019.1 -> And, in fact, it may be writing something

1020.48 -> you may never be able to read.

1022.01 -> So, you need to reason about this type

1023.99 -> of failure modes as well.

1025.64 -> Same applies to operator error.

1027.14 -> When you think about what kind of tooling do you need

1029.33 -> to put in place to make these processes safe.

1033.05 -> And so, these are the things we think about,

1035.21 -> and there is a variety of mechanisms we employ.

1040.04 -> This is not necessarily designed to be read,

1041.99 -> but some of the examples

1043.67 -> of these mechanisms are facility isolation, repair,

1046.85 -> variety of guardrails, automation, and so on.

1050.54 -> One thing I would like to note is this process,

1052.64 -> this thread model building process,

1054.71 -> and creating mechanisms based on thread,

1057.25 -> on on the thread model, is universal.

1059.45 -> It doesn't need to apply to Amazon S3.

1061.19 -> It doesn't need to apply to durability.

1063.2 -> It can apply to just about any cross-cutting

1066.62 -> or specific characteristic of a system

1068.75 -> that you would like to continuously maintain.

1071 -> And so, you, too, can use a very similar process

1074.3 -> with a new software development cycle.

1076.94 -> So most of these mechanisms,

1078.65 -> they exist behind the scenes transparently to you,

1081.5 -> and ultimately, you don't even know they're there.

1083.54 -> There are a couple, however,

1084.98 -> that you can explicitly and intentionally participate on.

1087.95 -> And we consider one example, durable chain of custody.

1092.54 -> This is a very, very fancy word

1094.64 -> for a very, very simple idea.

1097.19 -> And the easiest way

1098.33 -> to talk about it is we'll probably just demonstrated it

1100.36 -> in a simple diagram.

1101.99 -> We have a user.

1103.1 -> We have a bucket.

1104.21 -> And let's assume that user has been authenticated,

1106.4 -> has an access to the bucket,

1107.51 -> and would like to issue a PUT.

1109.73 -> This is the interaction diagram.

1111.59 -> It's very fancy.

1113.15 -> The reality, big messy world, looks something like this.

1118.49 -> In reality, your PUT is traveling very, very far.

1122.72 -> It travels, maybe it's traveling from your own data center.

1126.05 -> And so, it needs to exit your facility,

1128.09 -> enter the public internet,

1129.38 -> go through the variety of intermediaries.

1131.69 -> Then, it reaches the AWS region.

1133.85 -> And then, finally, front door of Amazon S3.

1137.12 -> If you calling S3 from EC2, for example,

1140.24 -> the path is shorter, but nevertheless, it's not dissimilar.

1144.17 -> So, it is theoretically possible,

1145.73 -> even with network level data integrity support,

1149.72 -> that some of the bits may flip.

1152.48 -> They may, in fact, flip on your own machine in memory

1155.51 -> as you're sending this data,

1156.68 -> or in the NIC we see these failures a lot.

1159.26 -> We see failures, for example, also in network,

1163.73 -> on network devices.

1165.35 -> The interesting thing about Amazon S3 is

1166.956 -> that that it's so huge that any theoretical failure

1170.03 -> that could be experienced,

1171.62 -> we probably have experienced in practice.

1173.69 -> And because of that we had to think about it.

1175.97 -> But back to this picture.

1177.59 -> So okay, so we travel far.

1179.51 -> What is the problem with that?

1181.4 -> Well, if the bits flip on the wire,

1183.95 -> what might end up happening is

1185.39 -> that you end up storing the object

1186.59 -> you didn't mean to store, right?

1187.88 -> Because by the time we reached Amazon S3 front door,

1191.48 -> something else may have arrived.

1194.51 -> We use checksums inside for Amazon S3

1198.26 -> to actually control what happens after you enter front door.

1201.8 -> So, what it means is that for every PUT and for every GET,

1205.46 -> but let's start with PUT,

1206.93 -> when the object arrives at our front door,

1209.21 -> if it doesn't have a checksum already,

1210.74 -> and I'll get into it in a second,

1211.94 -> we'll calculate it,

1212.96 -> and then this checksum forever travels with the object.

1216.02 -> And so with any given point of time,

1218.18 -> we can immediately detect if something has flipped,

1220.76 -> and the object has altered,

1221.93 -> and that durability posture has been weakened.

1224.72 -> So, that's great.

1225.59 -> We seem to be safe, at least,

1228.35 -> in on on some part of this trajectory.

1231.62 -> Checksums, by the way, I mean,

1233.06 -> it's a relatively common concept.

1234.38 -> I'm really quickly go through it.

1235.94 -> It's basically signature, or fingerprint of sort,

1238.46 -> that you can generate based on an arbitrary ByteBuffer.

1242.06 -> They have this interesting property,

1243.47 -> is that they're relatively compact,

1245.51 -> and they also statistically built algorithmically,

1249.05 -> such that two different buffers are very, very,

1252.59 -> very unlikely to generate the same checksum.

1255.2 -> And because of that you can,

1256.73 -> if you generate checksum on something,

1258.44 -> and then read that something and check

1260.06 -> and recalculate the checksum again,

1261.47 -> if they match,

1262.64 -> you can be pretty certain that nothing has changed.

1265.34 -> Okay, so we are safe from the front door

1268.28 -> to the actual storage node.

1270.8 -> That's great, but what about all of these other guys?

1273.05 -> What happens over here?

1274.46 -> Again, all manner of things can happen.

1276.47 -> Well, this is where you, customer, come in.

1280.07 -> Turns out, you can participate

1281.81 -> in this fancy thing called durable chain of custody.

1285.8 -> And this is one of the reasons that this year,

1288.32 -> earlier this year, I believe it was March,

1290.21 -> with additional checksum semantics in Amazon S3.

1295.85 -> You may remember,

1296.96 -> that we had had MD5 checksum for a while.

1300.26 -> I don't remember exactly how long, but for years.

1303.17 -> The issue with that particular approach was

1305.15 -> first of all MD5 is relatively,

1306.65 -> can be potentially expensive.

1308.48 -> And secondly, the way the API was built is

1311.78 -> that you would need to send the checksum first

1314.42 -> and data second.

1315.89 -> And that means that you, effectively, will need

1318.35 -> to scan your data twice.

1320.03 -> Now, this year we delivered several new algorithms.

1323.3 -> A lot of them are much more performant and they can,

1327.256 -> they rely on HTTP trailer, a thing you send in the end,

1330.65 -> as opposed to the header.

1332.48 -> And because of that,

1333.62 -> you don't need to scan your data twice anymore,

1335.63 -> and AWS SDKs all do all

1337.91 -> of these wonderful things for you.

1339.65 -> And so because of that, if you start using checksums,

1343.07 -> which is relatively easy enough

1344.54 -> and SDKs enable it very, very simply,

1348.08 -> then you'll participate in a durable chain of custody,

1351.23 -> and you have a lot of assurances

1352.76 -> that your object never gets altered even if it travels

1355.31 -> across the wide expanses of the internet.

1361.4 -> There is a lot more things we can talk about

1362.9 -> in context of durability,

1364.34 -> but several points I would like you to take with you is,

1368 -> first of all, Amazon S3 is designed for 11 9s durability.

1372.68 -> And that is a function of data decay, AFR,

1377.21 -> and repair effectiveness.

1378.59 -> And this is where this number come from.

1380.99 -> Amazon S3 looks at durability as an ongoing process.

1383.84 -> It's a part of the culture.

1384.95 -> It's not something we do once.

1386.27 -> It's something that we do every day.

1388.43 -> And it's also something that you can do

1390.2 -> in your organization, utilizing threat model,

1392.69 -> and utilizing mechanisms for durability,

1395.45 -> and any other similar cross-cutting quality.

1398.87 -> And finally, for that specific example,

1400.64 -> durable chain of custody actually starts with you.

1402.68 -> It starts with the customer.

1403.61 -> You can participate in that.

1405.77 -> Okay, so that was durability.

1408.65 -> Now, let's get to performance availability

1411.53 -> and horizontal scaling.

1413.15 -> We discussed that Amazon is,

1414.77 -> how Amazon S3 durably stores your data.

1416.93 -> But as you start reading and writing it, of course,

1419.267 -> you you care how fast you can go.

1422.48 -> And that has several dimensions.

1424.52 -> Performance of individual operations,

1426.23 -> which can be latency

1427.34 -> or throughput if you're streaming an object

1429.38 -> or pushing an object to S3.

1431.45 -> But also there is the scaling factor.

1433.22 -> How many of parallel connections can you open

1435.98 -> to the same bucket, for example?

1437.99 -> And so, before we examine

1439.16 -> how Amazon S3 handles these things,

1440.78 -> let's go back to the basics

1442.37 -> and consider our hypothetical theoretical system we built

1445.64 -> of our hard drives.

1447.17 -> You remember this picture, right?

1449.135 -> And again, as we discussed, a lot of change,

1451.52 -> and we list some of the things we changed.

1453.92 -> But something else remains the same.

1456.59 -> Turns out, specifically for random reads and writes,

1460.85 -> for sequential reads and writes this number is much better.

1463.46 -> But for random reads and writes,

1464.84 -> you can only expect about 120 IOPs.

1468.53 -> And it was, this number is true today.

1470.57 -> It was true ten years ago, and it was true 20 years ago.

1474.38 -> So, what this means is that per terabyte,

1480.29 -> hard drives are actually slowing down.

1483.05 -> You can see, and specifically per terabyte,

1485.63 -> the speed of hard drives is not slowing down,

1487.31 -> but if you consider capacity,

1488.99 -> you will see that as the capacity growing,

1491.99 -> IOPs per terabyte is slowly going down.

1494.54 -> This is referred to as reduce excess density.

1498.65 -> And that, as it turns out, has a huge impact

1501.38 -> on how we design.

1503.63 -> And by we I mean,

1504.463 -> collective we the world, design storage systems.

1507.74 -> And to demonstrate how to think, how we think about it,

1510.38 -> let's consider two very different workload.

1512.66 -> One of them, it's a relatively small data set.

1515 -> Well, in the modern world, four petabyte is not that much.

1518.81 -> And you see that most of the time,

1520.4 -> the IOPs tps is relatively low.

1522.98 -> But occasional, there is this enormous spikes

1525.23 -> that go up to 2 million IOPS or so.

1529.55 -> So, let's do a very basic math for our homegrown system.

1533.15 -> And we're not even considering data redundancy right now.

1536.72 -> How many hard drives are we gonna need for this workload?

1539.57 -> Well, using our basic math,

1541.97 -> based on the capacity of 20 terabyte of hard drive,

1545.39 -> which is a relatively reason common capacity,

1547.43 -> we're gonna need 185 hard drives.

1549.8 -> Great. It's not a very big number,

1551.75 -> and they will do 22,000 IOPS for us.

1555.38 -> Is that sufficient?

1556.97 -> No, it's not, because we actually require two million.

1561.95 -> To support two million IOPs,

1563.51 -> we actually will need 19,000 hard drives.

1567.44 -> So, IOPs requirement actually pushes hard drive capacity

1574.227 -> up to a hundred times by compared

1576.74 -> with just the storage footprint requires.

1580.82 -> Let's consider a diametrically opposite example.

1584.33 -> Larger data set, close to 30 terabyte.

1586.88 -> The IOPS, pretty even, relatively small,

1590.277 -> 5k, maybe 8k in couple of spikes, but relatively low.

1596.36 -> And so, once again, to store this data set,

1599.24 -> we need many more hard drives.

1600.74 -> Now 1400, which again, it's a bigger data set.

1604.19 -> And it will give us 168,000 IOPs,

1607.64 -> which is far, far more than we need, right?

1609.927 -> And so in this case,

1611.15 -> storage actually requires 120 more hard drives

1614.96 -> than IOPS would.

1616.587 -> Okay, so, this was relatively basic math

1618.95 -> and relatively self-explanatory workloads, right?

1621.71 -> Small data set, a lot of spikes,

1623.36 -> large data set, relatively even.

1625.4 -> Great, so why is this relevant?

1627.35 -> Well, it demonstrate that scaling storage

1630.56 -> for different workload profiles is actually very hard.

1635.06 -> Turns out, we have a very unexpected friend

1639.68 -> in this space, and that is scale.

1643.58 -> Most of the time, things get much more complicated at scale,

1646.49 -> but some things can actually get simplified.

1648.8 -> Imagine that these two workloads were running

1650.69 -> on the same system.

1651.83 -> Imagine, then there were ten more workloads

1654.8 -> like the first one, and the second one,

1656.45 -> hundred, thousand, and so on.

1659.33 -> You may see a picture that looks something like this.

1662.39 -> When you start aggregating these workloads across variety

1665.51 -> of, once aggregating workloads across variety of data sets

1669.74 -> and specific spikes, things actually even out.

1673.37 -> The aggregate looks, beginning to look relatively flat,

1677.3 -> and that is called workload decorrelation.

1679.91 -> Aggregating a massive number of workloads,

1682.61 -> actually creates more smooth and predictable throughput.

1685.01 -> It is surprising, but it's true.

1687.56 -> There is one aspect that we still need to consider, though.

1689.96 -> How do we make sure

1691.4 -> that a spiky demand will not laser down

1696.11 -> individual hard drives?

1698.33 -> Well, this is where we go back to this picture

1700.16 -> and start talking about Amazon S3.

1702.08 -> Turns out that, yes, this is helpful for durability,

1704.69 -> but it's helpful for availability as well.

1706.97 -> Because remember,

1707.803 -> any individual customer data occupies only tiny amount

1710.78 -> of any given disc, which means

1712.61 -> that individual workload doesn't typically hotspot a disc.

1716.51 -> It is spread across many, many discs.

1719.48 -> And this also means

1721.22 -> that each disc doesn't serve only workload

1724.07 -> of this individual customer.

1725.33 -> In fact, it's workload of multiple customers.

1727.473 -> Decorrelation again.

1729.14 -> And also it allows us to maximize IO.

1733.07 -> And given the capacity of the number of disc,

1736.94 -> we have a relatively stable,

1738.56 -> relatively smooth running system.

1740.51 -> So, what does all of this mean?

1742.82 -> And why, and why did I mention all of this?

1746.297 -> If there is one thing that I would like for you to remember

1749.06 -> from the performance section,

1750.62 -> is that Amazon S3 is a massively,

1752.42 -> horizontally scalable system.

1754.7 -> It may be sort of a trivial statement,

1757.01 -> but what this means is that Amazon S3 is designed

1760.25 -> in such a way that it expects you, it wants you

1764.03 -> to parallelize everything about your workloads,

1766.64 -> and it handles it well.

1768.14 -> You can achieve very impressive performance

1770.12 -> through parallelization.

1771.17 -> Let's consider several aspects of that.

1773.27 -> One of them is parallelization across multiple endpoints.

1776.33 -> What is an endpoint?

1777.35 -> Well, when you connect to,

1778.52 -> when you open a connection to Amazon S3 to your bucket,

1781.49 -> you ultimately, at some point,

1782.99 -> gonna connect to the server someplace.

1784.4 -> This is one of our endpoints.

1786.8 -> Now we have tens, and in some regions,

1789.02 -> hundreds of thousands of those,

1790.79 -> and they run independently from one another.

1793.82 -> And so, if you distribute your operations

1796.25 -> across many endpoints,

1797.99 -> you actually achieve very high parallelization,

1800.12 -> because because you have completely decorrelated resources

1803.48 -> handling your workloads.

1804.98 -> So in practical terms, it means more threats,

1808.22 -> more open connections, more workers, and so on.

1811.28 -> That is typically very effective mechanism to reduce latency

1814.19 -> and improved throughput.

1815.42 -> And Amazon S3 tends to scale very well with that.

1820.73 -> So we discussed, so this is on individual operations, right?

1824.03 -> So, but turns out, the same logic can can apply

1827.21 -> to a single operation as well.

1829.97 -> Just, it's a very trivial graph once again.

1833.54 -> If you upload a one gigabyte object,

1836.72 -> it'll probably take you much longer than to upload.

1839.9 -> But for example, one megabyte object, right?

1843.98 -> But what if you were able to upload the same object

1848.12 -> in parallel with itself?

1851.12 -> And what if, after you have uploaded an object,

1854.15 -> you were able to read chunks of object,

1856.79 -> chunks of this object also in parallel

1859.07 -> and assemble them on the client?

1861.35 -> Well it turns out, S3 does allow you to do that.

1864.23 -> And we have a feature called multipart uploads

1867.47 -> when you can basically chop up your original object

1869.57 -> into small chunks, completely unrelated to the concept

1872.24 -> of shards we discussed before,

1873.86 -> and upload them in parallel with one another.

1875.93 -> And then, complete upload, and the object will materialize.

1879.35 -> Similarly, we have byte-range GETs that allow you

1882.83 -> to fetch data out different chunks of the same object

1887.54 -> from Amazon S3.

1888.373 -> And if you superimpose this with the fact

1891.2 -> that you can also open multiple connections,

1893.75 -> then you can achieve really very impressive parallelism.

1898.25 -> As you increase the parallelism of workloads,

1900.26 -> it helps to understand what impact it had, might have

1902.9 -> on a S3 name space.

1904.04 -> Now, this may be a surprising statement to make,

1906.47 -> but S3 name space, in addition to its actual byte storage,

1911.03 -> also has its own scalability mechanism.

1913.16 -> And it scales with the request rates based

1915.2 -> on what we call prefixes.

1917.09 -> Prefix is basically a leading part of your key

1921.29 -> combined with the preceding bucket name.

1924.29 -> And so, the keys with the beginning of which are the same,

1929.06 -> are likely gonna end up in the same prefix.

1931.49 -> Prefix is an actual backed up by a set

1933.62 -> of physical resources in Amazon S3,

1936.44 -> and the set of physical resources

1938.391 -> imposes certain throughput limitations,

1941.3 -> specifically 35 on PUTs and 55 on GETs.

1944.93 -> So what happens if you go faster?

1946.64 -> Well, Amazon S3 will scale out your prefix for you.

1949.37 -> It'll create more.

1950.84 -> This process takes several minutes typically,

1953.48 -> and from workloads that grow gradually.

1956.3 -> This is a mechanic you don't even need to worry

1958.55 -> or care about because Amazon S3

1961.04 -> will proactively scale your prefixes

1962.54 -> as tps is slowly increasing.

1964.94 -> But if you have a spike your workload, for example,

1968.21 -> if you go from zero to 15,000 tps on a relatively

1973.67 -> on a relatively narrow set of keys,

1976.25 -> then the scaling may kick in immediately,

1979.1 -> but might take minutes to catch up.

1980.99 -> And so in these cases you might experience some pushback,

1983.96 -> which generate 503s and tell you to slow down.

1986.72 -> And retries will resolve them.

1988.22 -> And maybe all of this is okay.

1989.75 -> In fact, for vast variety of customers, it is okay,

1993.311 -> and they don't need to worry about.

1994.73 -> But if you have particular sensitive performance workloads,

1998.09 -> this is something to be aware of.

1999.587 -> And the easiest to demonstrate it is

2001.15 -> with a very simple example.

2003.58 -> Let's consider several,

2006.55 -> so let's consider autonomous vehicles.

2009.25 -> I realize it is just not written on the slide. (laughs)

2011.71 -> And let's say, they sit in the garage, and they kind,

2014.02 -> and they drive out in the beginning of the day.

2016.3 -> And then, they drive back in at the end of the day,

2018.157 -> and start uploading data for all of the,

2020.95 -> for whatever just happened to them.

2022.9 -> Now, this is one of the possible prefix designs.

2025.3 -> It's a relatively natural way to organize data.

2028.57 -> But as you can see, it shares a lot of common prefixes.

2031.33 -> So, they're likely gonna end up in the same S3 prefix.

2033.94 -> And so, the performance picture that you may see

2035.787 -> on the upload may look something like this.

2038.5 -> So you would reach a certain limit.

2040.42 -> Ignore the numbers on the the tps numbers

2042.94 -> on the vertical axis.

2044.02 -> They're definitely not quite right.

2045.55 -> I don't know why it says at 1000.

2047.86 -> But you will achieve a plateau at first,

2050.41 -> especially if you have a massive burst of goodput,

2054.22 -> as in as in the requests that have successfully completed.

2058.06 -> And during this plateau, we will scale out your prefix,

2061.21 -> and then you will fully scale up your your goodput.

2064.06 -> And again, in many cases, this is completely fine.

2067.15 -> But if you want to prevent this,

2068.8 -> if you wanna prepare for this proactively,

2072.19 -> there is multiple things you can do.

2073.51 -> But it ultimately means design your prefixes intentionally

2076.3 -> for your workloads.

2077.71 -> And one of the common mechanisms we suggest utilizing

2080.71 -> in this case is what we call entropy.

2082.93 -> Entropy is something that is uniformly distributed

2086.17 -> and then not obviously correlated

2087.55 -> with the prefix name.

2088.9 -> Some, some customers interpret it as a random number.

2091.87 -> It is not, but it is typically something

2093.61 -> you can deterministically calculate because, well,

2095.71 -> otherwise, you cannot really build that key.

2098.56 -> But in algorithms like hashes, MD5,

2102.34 -> or any other similar algorithms,

2103.78 -> actually can be very helpful with that.

2107.83 -> To sum up as a final recap,

2110.11 -> parallelize everything.

2112.24 -> Amazon S3 loves you to do that.

2115.06 -> It's a massively horizontally scalable system,

2117.37 -> so you can parallelize across endpoints,

2120.19 -> across individual operations, and using MPUs and range GETS.

2124.87 -> And you can, and as you do so, there is some caveats

2128.41 -> around scaling workloads across prefixes,

2130.63 -> which you also can overcome by variety of mechanisms,

2133.299 -> such as entropy.

2135.16 -> Now, I'm handing the torch torch over to Sally

2137.95 -> who will discuss cost management and security.

2142.465 -> (audience clapping)

2148.18 -> - All right.

2149.05 -> Who.

2151.66 -> Thank you, Oleg.

2152.68 -> Hi, I'm Sally.

2154.21 -> I'm a software development engineer at Amazon S3.

2157.3 -> Now that we've covered S3's durability model,

2160.27 -> performance, and scaling,

2161.98 -> let's talk about actually storing your data in S3

2164.86 -> and choosing the right storage class for your use case.

2169.87 -> Amazon S3 provides what we call storage classes.

2173.14 -> You can think about them as sort of a dial that allows you

2176.29 -> to optimize your cost based on your use cases.

2180.04 -> When evaluating which storage class to use,

2182.74 -> there are several factors to consider.

2185.05 -> The first one is access pattern.

2187.09 -> How frequently is the data accessed?

2189.49 -> What sort of latency and availability do we expect?

2193.75 -> The other one is duration of storage.

2196.42 -> How long do we expect to store that data?

2199.27 -> S3 provides a wide range of storage classes

2201.91 -> for you to choose from.

2203.71 -> And here, we'll talk about just a few.

2206.98 -> The first one I wanna cover is S3 Standard.

2210.49 -> In some sense, it is a default.

2212.86 -> It is designed for 99.99% availability,

2216.88 -> milliseconds latency, and protection against loss

2219.94 -> of a single availability zone.

2222.07 -> So, this class works well for hot data,

2224.44 -> or data that needs to be frequently accessed.

2228.13 -> The next one is S3 Standard Infrequent Access, or IA.

2232.36 -> So, if you know that your data is less frequently accessed

2235.51 -> and not deleted within a month,

2237.61 -> you can save up to 40% storage costs compared

2240.28 -> to S3 Standard, but with a retrieval fee

2242.68 -> for accessing the data.

2244.6 -> So, this works well for cool data

2246.61 -> that requires low latency and high availability.

2251.47 -> For non-critical data,

2252.79 -> we also have Single-AZ options like S3 One Zone-IA,

2256.57 -> which can further bring down the cost.

2260.41 -> You can also archive objects into Amazon Glacier

2263.53 -> at a 60% lower storage cost compared to S3 One Zone-IA.

2269.56 -> Glacier Deep Archive is our storage class

2271.87 -> with the lowest cost,

2273.25 -> designed for long term archive data that is rarely accessed,

2276.97 -> but need to be kept securely.

2280.48 -> And we have also introduced a new Glacier Storage class,

2283.75 -> Glacier Instant Retrieval,

2285.55 -> which is great for archive data that needs to be accessed

2288.4 -> within milliseconds.

2290.11 -> These are all great options for archive data.

2294.67 -> And finally,

2295.87 -> S3 Intelligent Tiering is the only cloud storage class

2299.05 -> that delivers automatic storage cost savings for customers.

2303.28 -> S3 will automatically move data

2305.5 -> to the most cost effective access tier

2307.87 -> when data access patterns change

2309.85 -> without performance impact or operational overhead.

2315.16 -> So as data ages, their access patterns tend to change,

2318.46 -> and a storage class transition is sometimes appropriate.

2321.82 -> How can we do that?

2323.53 -> So, if you have predictable access patterns,

2326.17 -> you can use S3 Lifecycle

2327.85 -> to transition to a different storage class.

2330.67 -> And you can do so based on a variety of rules

2333.28 -> that can also include your tags and prefixes.

2337.15 -> But if your access patterns are constantly changing,

2339.76 -> or you simply don't know your access patterns,

2342.34 -> you can have Amazon S3 analyze your access patterns

2345.31 -> automatically and move your objects

2347.65 -> to Infrequent, Archive, and Deep Archive access tiers

2351.55 -> with S3 Intelligent Tiering.

2353.5 -> So, let's take a closer look.

2356.47 -> For a small monthly object monitoring and automation fee,

2360.43 -> S3 Intelligent Tiering automatically optimizes storage cost

2364.12 -> based on the access pattern of that data.

2367.27 -> You can also optionally activate one or both

2370.36 -> of the Archive Access tiers,

2372.01 -> which are designed for asynchronous access.

2374.98 -> So, Amazon S3 will move your objects

2377.77 -> that haven't been accessed in the last 90 days

2380.11 -> to the Archive Access tier,

2381.91 -> and after 180 days, to the Deep Archive Access tier,

2385.72 -> all within the S3 Intelligent Tiering storage class.

2389.53 -> Any time an object in the Archive Access tier

2392.29 -> or the Deep Archive Access tier is accessed,

2395.08 -> that object is moved back to the Frequent Access tier

2398.14 -> in as little as a few hours.

2401.02 -> So in summary,

2402.28 -> S3 intelligent hearing can further optimize storage costs

2405.4 -> when access patterns change without any analysis,

2408.88 -> operational overhead, or retrieval fees.

2413.83 -> In order to manage cost effectively

2415.81 -> and make decisions about your storage classes,

2418.36 -> you'll need insights into your cost structure.

2421.03 -> And to help with that,

2422.38 -> S3 Storage Lens provides storage usage and activity metrics

2426.55 -> where you can drill down by region,

2428.41 -> storage class, bucket, and prefix.

2431.86 -> You have up to 15 months of historical data to analyze,

2435.34 -> which enables you to analyze long-term trends and patterns.

2439.78 -> Storage lens also gives you recommendations

2442.09 -> for cost efficiency and data protection best practices.

2446.32 -> And last year, we released CloudWatch integration

2449.26 -> that allows you to, among other things, utilize alarms

2453.04 -> and events rules triggered actions.

2456.73 -> And this year,

2457.57 -> we've more than doubled the existing Lens metrics,

2460.36 -> adding 34 new metrics

2462.13 -> across the free and paid advanced tiers,

2464.8 -> bringing total metrics count to more than 60.

2468.25 -> With the new metrics,

2469.87 -> Storage Lens gives you deeper visibility

2472.24 -> into cost optimization, data protection,

2475.21 -> access control, performance, and more.

2480.1 -> So to summarize,

2481.54 -> S3 provides a wide range of storage classes for you

2484.3 -> to choose from.

2485.74 -> S3 offers tools like Storage Lens

2488.38 -> to help you understand, analyze, and optimize storage.

2492.97 -> When you need to move objects

2494.77 -> from one storage class to another,

2497.05 -> you should use S3 Lifecycle rules for predictable

2500.08 -> or known access patterns.

2502.12 -> But if you have complex or unpredictable access patterns,

2505.45 -> you can use the Intelligent Tiering storage class,

2507.94 -> so S3 can automatically place your objects

2510.49 -> in the right access tier.

2514.75 -> Here's our last topic of the day.

2516.52 -> Let's talk about how to secure your data in S3.

2520.69 -> At AWS, security is at the core of everything we do.

2524.8 -> The objects and buckets that you store

2526.63 -> in S3 are secured by default,

2529.54 -> and we give you granular controls

2531.13 -> to share that access with others.

2533.77 -> For this section,

2534.603 -> I wanna first go through two types of policies

2536.83 -> that you can use for access controls on your buckets,

2539.92 -> IAM policy and bucket policy.

2543.04 -> Then, we'll move on

2544 -> to talk about how you can share your data

2545.98 -> with others within your account

2548.11 -> or across different accounts.

2549.76 -> And we'll talk about how to do it securely,

2551.92 -> so you're only sharing your data with the intended audience.

2556.12 -> We'll also talk about how you can review

2558.07 -> and audit the access controls on your buckets

2560.41 -> within the AWS account.

2562.75 -> And finally,

2563.74 -> we'll talk about the server side encryption options

2566.14 -> that Amazon S3 offers

2567.97 -> and how to choose the best encryption mode

2570.22 -> for your use case.

2573.55 -> When you have data in Amazon S3

2575.47 -> and have different users using that data,

2577.66 -> you need a way to give them access.

2579.94 -> To do that, you can either define an IAM policy

2583.15 -> in AWS Identity and Access Management

2586 -> or a bucket policy in Amazon S3.

2589.6 -> An IAM policy is an access policy that you attach

2592.96 -> to your IAM user or IAM role, hence it is identity based.

2597.79 -> A bucket policy is an access policy that you attach

2601.03 -> to your S3 bucket, hence it is resource based.

2604.75 -> Both IAM policy and bucket policy use the JSON format,

2608.44 -> so we can easily relate one to the other.

2611.47 -> So, let's first talk about IAM.

2615.07 -> Within your AWS account,

2616.54 -> here are some basic IAM concepts.

2619.12 -> First, you have principles.

2620.8 -> These are the identities in the IAM system.

2623.92 -> You have the actions that they take.

2625.99 -> Specifically for S3, say you're reading an object.

2629.38 -> That action is called GET object.

2632.08 -> And then you have the resource,

2633.49 -> which is what a principle is performing the action on.

2637.36 -> And in S3, you can be acting on the buckets

2639.64 -> for some actions like listing out the content of a bucket,

2642.88 -> or you could be acting on the objects

2644.43 -> in other cases like reading a particular object.

2649.96 -> So, how does this model translate into something

2652.33 -> that IAM can understand and enforce?

2654.91 -> That'll be an IAM policy.

2657.19 -> An IAM policy defines

2658.72 -> what actions can this IAM user perform.

2662.08 -> It has three mandatory elements,

2664.3 -> Effect, Action, and Resource.

2668.44 -> So for this IAM policy, the first element,

2671.02 -> the Effect statement, specifies whether the policy results

2674.44 -> in a deny or allow.

2677.53 -> The second element,

2678.46 -> the Action statement, specifies the action

2681.07 -> that is being denied or allowed.

2684.19 -> And the third element, the Resource statement,

2686.5 -> specifies which AWS resource the policy applies to.

2690.46 -> So, that's all you need for an IAM policy.

2692.83 -> This IAM policy effectively allows this particular IAM user

2696.85 -> to read and write objects

2698.53 -> to the bucket called sample-bucket-reinvent.

2702.88 -> So, who gets to apply IAM policies?

2705.43 -> Well, if you are the IAM administrator

2707.08 -> for your AWS account,

2708.67 -> you define IAM policies and attach them

2711.34 -> to IAM user or IAM role within your account.

2715.21 -> And you can further allow an IAM user to assume an IAM role.

2719.35 -> With Role Assumption, the IAM user inherits the permission

2723.28 -> of the IAM role.

2726.73 -> Instead of using IAM policy,

2728.68 -> you can use bucket policy in Amazon S3.

2731.257 -> So, this is a resource based approach,

2733.45 -> because it defines permissions for S3 bucket.

2738.858 -> A bucket policy defines who can access my bucket

2741.91 -> and what actions can that user perform.

2744.7 -> It is very similar to the IAM policy

2746.8 -> that uses the JSON format,

2748.54 -> but it requires an additional principle statement

2751.21 -> that states who the policy covers.

2753.94 -> So here in this example,

2755.5 -> the bucket policy allows AWS account ID 111111111111

2758.98 -> to read and write objects

2760.42 -> to the bucket called sample-bucket-reinvent.

2765.22 -> The principle statement

2766.33 -> in a bucket policy can accept a variety of input,

2769.51 -> including AWS account and root user,

2772.33 -> IAM users, federated users, IAM roles,

2775.93 -> assumed-role sessions, AWS service principles,

2779.14 -> and anonymous user, which is the same as public,

2782.08 -> but we do not recommend that for obvious reasons.

2786.82 -> And you can further add granularity to your IAM policy

2790.18 -> or bucket policy with a condition statement.

2792.94 -> Here in this example,

2794.11 -> we add a condition that only allows access to objects

2797.08 -> that have the Re:invent tag on them.

2800.8 -> And we have many more condition keys that you can work with

2803.29 -> for granular controls.

2804.91 -> For detailed descriptions,

2806.35 -> you can go to the AWS documentation page

2808.84 -> and search for condition keys for Amazon S3.

2815.29 -> Next, we'll move on to the second main concept

2817.33 -> and talk about access

2818.53 -> within your account versus access across accounts.

2822.04 -> When you only have IAM users within your AWS account,

2825.22 -> reading or writing data to your S3 buckets,

2828.13 -> you'll either need an IAM policy or a bucket policy

2831.46 -> to provide that permission.

2833.59 -> But as you expand your use of Amazon S3,

2836.2 -> sometimes you wanna share your data with someone

2838.57 -> from an external account.

2840.52 -> If you need to set up cross account access,

2842.77 -> you'll need both IAM policy

2844.63 -> in the account that owns the IAM user

2847.12 -> and bucket policy in the account that owns the S3 bucket.

2851.05 -> This is because S3 buckets are private to the account

2853.9 -> that owns them by the default.

2857.44 -> For cross-account write,

2858.91 -> the writer of the data owns the data.

2861.22 -> So in this example,

2862.57 -> we have an account in, a user in account A writing data

2866.29 -> to a bucket in account B,

2868.42 -> The resulting objects are owned

2870.07 -> by the object writer, account A.

2872.71 -> The objects cannot be read

2874.33 -> by the bucket owner, account B.

2876.49 -> And the objects cannot be shared by the bucket owner

2879.34 -> with another account, say account C.

2883.24 -> Now you might be thinking,

2884.477 -> "I'm not a fan of this limitation.

2886.45 -> When other people put buckets into my..."

2888.598 -> (mic feeding back) Woo.

2889.607 -> "Put objects into my bucket,

2891.46 -> I wanna be the owner of those objects."

2893.74 -> And for customers using bucket policy

2895.72 -> to enforce access control, bucket policy is only applicable

2899.14 -> to objects owned by the bucket owner.

2901.78 -> How can you ensure that you own all the objects

2904.63 -> in your bucket without relying

2906.22 -> on the permissions configured by the object creator?

2911.2 -> Here's where the Bucket-Level Object Ownership setting

2913.57 -> comes in.

2914.65 -> When you set Object Ownership to bucket owner enforced

2917.74 -> on your S3 bucket,

2919.27 -> every object in that bucket becomes automatically owned

2922.6 -> by the bucket owner.

2924.37 -> If you were previously using access control list,

2927.04 -> they can be disabled, as you can now exclusively rely

2930.07 -> on IAM policies for access control.

2932.77 -> Access control list will no longer affect ownership

2935.8 -> or permissions for your S3 objects with this setting.

2939.4 -> So, here's what you'll see on the AWS console.

2942.25 -> For this bucket, once you select bucket owner enforced,

2945.85 -> you as the bucket owner automatically own

2948.19 -> and have full control over all the objects in this bucket.

2951.76 -> So, this setting can be applied to a new or existing bucket

2955.15 -> and can be turned off at any time.

2959.32 -> As your bucket evolves and becomes a central location

2961.96 -> for storing shared data sets,

2963.85 -> your bucket policy can become more complex.

2966.55 -> You'll probably spend more time to audit and manage it.

2970.03 -> Amazon S3 Access Points,

2971.68 -> a feature that helps you simplify access management

2975.01 -> for shared data sets.

2977.26 -> In a nutshell,

2978.25 -> you divide access to your bucket into multiple access points

2981.34 -> for different groups of users.

2983.38 -> And here in this example,

2984.76 -> we divide cross-account users into three different groups,

2987.91 -> Finance, Sales, and Dev.

2990.37 -> Each group reads data from a designated access point

2993.4 -> that has its own access point policy.

2997.57 -> With Amazon S3 Access Points,

2999.73 -> you effectively decompose your bucket policy

3002.64 -> into separate access point policies

3004.56 -> that you can manage more easily.

3007.38 -> Within each access point policy,

3009.27 -> you can further add granularity to your permissions.

3013.32 -> In addition, you can use access points to restrict access

3016.62 -> to specific virtual private clouds.

3020.88 -> Now, here's a screen that you may all be very familiar with.

3023.85 -> In the S3 console, we have a total of five buckets.

3026.7 -> The access column

3027.63 -> in the bucket overview page shows the status for my buckets.

3031.78 -> And here we have one bucket that has public access

3034.83 -> and another bucket that has objects that can be made public.

3038.22 -> That doesn't sound right.

3039.6 -> I can probably just go ahead and click on that second bucket

3042.21 -> and change the bucket policy.

3044.1 -> But what if I have 500 buckets,

3045.9 -> and half of them have public access?

3048.06 -> How do I easily correct this mistake

3049.77 -> and prevent it from happening again in the future?

3054.06 -> To guard against accidental public access,

3056.58 -> Amazon S3 Block Public Access is a feature

3059.34 -> that applies a blanket protection against public access.

3063.21 -> This feature can be set at the bucket level

3065.43 -> or at the account level.

3067.05 -> It provides protection against permissions

3069.27 -> from access control list, bucket policy, or both.

3075.63 -> Going back to our previous example,

3077.52 -> after I turn on block public access at the account level,

3080.52 -> all of my buckets are protected.

3082.5 -> The two buckets that were previously flagged are now secured

3085.62 -> and not publicly accessible.

3090.03 -> Oops.

3091.95 -> Once you turn on block public access at the account level,

3094.83 -> this setting gives you the overarching protection

3097.02 -> across all of your buckets

3098.46 -> and overrides your individual bucket setting.

3102.69 -> Customers wanted a way to review their existing permissions

3106.08 -> and prevent unintentional access permission.

3108.75 -> So for this, we introduced Access Analyzer for S3.

3112.62 -> This feature analyzes permissions for your S3 buckets

3115.68 -> in an AWS region and provides a simple dashboard

3119.1 -> that shows buckets with public access,

3121.41 -> as well as buckets that are shared with external entities.

3124.68 -> So, let's take an example.

3128.1 -> Here in this example,

3129.39 -> you can select Access Analyzer for S3 on the left panel.

3133.26 -> Once you click that, you'll see a dashboard

3135.36 -> that contains Access Analyzer findings.

3138.06 -> Access Analyzer reviews your bucket policy,

3140.88 -> access control list, access point policy,

3143.7 -> and block public access settings to produce findings.

3147.63 -> It'll list buckets that have public access on the top panel

3150.654 -> and buckets that are shared with external accounts

3153.12 -> on the bottom panel.

3154.8 -> It'll also tell you the means that granted the access,

3157.23 -> such as bucket access control list, in this example.

3162.15 -> When you share your buckets with external entities,

3164.61 -> we generally recommend three best practices.

3167.43 -> So first, run Access Analyzer

3169.83 -> to understand how your buckets are being shared.

3172.53 -> And second, you should keep public buckets

3174.93 -> in a dedicated AWS account,

3176.97 -> so you can enable block public access

3179.1 -> at the account level everywhere else.

3181.77 -> And third, you should enable bucket owner enforced

3184.74 -> in S3 Object Ownership to standardize object ownership

3188.16 -> in your bucket in order to leverage bucket policies.

3194.22 -> Finally, let's talk about encryption.

3196.47 -> S3 offers several options for server side encryption.

3201.69 -> First, we have SSE-S3,

3203.61 -> where S3 manages your encryption keys.

3206.25 -> When you select SSE-S3 encryption,

3208.74 -> S3 generates a unique encryption key,

3211.62 -> encrypts your object with that key,

3213.87 -> and then encrypts the key with the regional root key

3216.54 -> that is regularly rotated.

3218.7 -> Customers do not manage their own key policies,

3221.43 -> and encryption is not logged in CloudTrail.

3224.077 -> SSE-S3 is offered to customers free of charge.

3228.87 -> Alternatively, you can choose SSE-KMS,

3231.57 -> where AWS key management service, or KMS,

3234.66 -> manages your encryption keys.

3236.94 -> With SSE-KMS, S3 calls KMS on behalf of the customer

3241.2 -> to generate a unique encryption key,

3243.27 -> which is then used to encrypt the object.

3246.15 -> The encryption key itself is protected by a KMS key.

3250.2 -> When you use a customer managed KMS key,

3253.05 -> you can manage key policies,

3255.09 -> configure how often you'd like to rotate the key,

3257.7 -> and any usage of the KMS key will be reflected

3260.67 -> on CloudTrail.

3262.11 -> And you'll be charged for each request made to KMS.

3267.21 -> Another option is to use SSE-KMS with an AWS managed key,

3271.68 -> but you won't be able to manage key policies

3274.17 -> or control the rotation schedule of that key.

3277.11 -> And since you cannot change the key policy,

3279.42 -> you will not be able to share your data

3281.4 -> with external accounts,

3282.69 -> as they'll require access to your KMS key

3285.18 -> in order to successfully encrypt or decrypt the object.

3288.96 -> So, the best option is to use SSE-KMS

3291.9 -> with a customer managed KMS key,

3294.09 -> as it gives you the most control and visibility

3296.91 -> into how you encrypt your data.

3300.93 -> Default encryption is a bucket level setting

3303.12 -> that ensures all, all the objects stored

3305.55 -> in the bucket is encrypted on the server side.

3308.52 -> When you turn on default encryption on your S3 bucket,

3311.7 -> all new objects will be encrypted

3313.59 -> with the type of encryption key that you configure.

3316.62 -> We support SSE-S3 and SSE-KMS

3319.41 -> for bucket level default encryption.

3323.91 -> So as we just talked about, for KMS encrypted objects,

3327.051 -> S3 calls KMS on behalf of the customers

3329.94 -> to get encryption keys to in order to encrypt

3332.73 -> or decrypt the data.

3334.95 -> Due to the volume of their workload,

3336.63 -> some customers could encounter KMS requests rate throttling

3340.26 -> and unexpectedly high cost due to repeated calls

3343.28 -> to KMS from S3.

3345.973 -> So Amazon S3 Bucket keys is a feature

3348 -> that is most useful for high volume workloads

3350.58 -> accessing KMS encrypted objects in S3.

3353.76 -> It can help you reduce your KMS calls,

3355.89 -> resulting in an improvement in encryption performance,

3358.59 -> and reduction of KMS costs by up to 99%.

3362.64 -> So if you want to use KMS backed encryption,

3365.31 -> using bucket keys is highly recommended

3367.77 -> to reduce your cost of encryption.

3372.36 -> So to summarize, you can control access

3374.79 -> to your bucket with either an IAM policy

3377.13 -> or a bucket policy within your account,

3379.32 -> and you'll need both to share your account,

3380.997 -> to share share your bucket with an external account.

3384.63 -> To share your bucket with different groups of users,

3387.48 -> consider using S3 Access Points.

3390.54 -> You can ensure that the bucket owner owns all the objects

3393.69 -> inside a bucket by enabling bucket owner enforced

3396.69 -> in S3 Object Ownership.

3399.09 -> You can prevent public access with S3 Block Public Access

3402.93 -> and review access controls with Access Analyzer.

3406.11 -> And finally, we recommend enabling server side encryption

3409.23 -> to increase your security posture.

3411.54 -> If you're using SSE-KMS encryption,

3413.91 -> you should use a customer managed key

3415.74 -> instead of the AWS managed key,

3417.87 -> and consider enabling S3 bucket keys to save on KMS costs.

3423.87 -> If you want to continue your AWS Storage learning,

3426.69 -> please visit https://aws.training/storage.

3429.84 -> You can find the AWS Skill Builder to build a learning plan,

3433.35 -> increase your knowledge using the ramp-up guides,

3435.9 -> and earn digital badges to demonstrate your knowledge.

3440.34 -> Give every one a second.

3443.22 -> Thank you. (audience clapping)

3444.45 -> Session survey should already be on in your inboxes

3447.15 -> or on your app.

3448.08 -> Just as we love getting your feedback on Amazon S3,

3450.48 -> we'd really appreciate your feedback on this session.

3452.88 -> Thank you.

3453.981 -> (audience clapping)

Source: https://www.youtube.com/watch?v=v3HfUNQ0JOE