Amazon EFS: Secure data persistence with Amazon ECS and AWS Fargate

Aug 16, 2023

Amazon EFS: Secure data persistence with Amazon ECS and AWS Fargate

In this demo, we’re going to use Amazon EFS to create a shared file system, and then attach it to existing Amazon ECS and AWS Fargate services. We’ll also explore how AWS IAM Authorization and EFS Access Points can simplify configuring more granular access to our data for different scenarios.

Amazon Elastic File System (EFS) provides a simple, scalable, and fully-managed shared file system for use with AWS Cloud services and on-premises resources. Whether you’re developing a new containerized application or modernizing an existing monolith, there are many types of applications where a shared persistence layer is an essential ingredient in achieving a resilient and durable architecture.

Learn more about AWS Storage at - https://go.aws/3e7MdSF

Persistent File Storage for Modern Applications: https://go.aws/2LKgsmd
GitHub sample code repository: https://bit.ly/2TlBewC
Tutorial on using Amazon EFS File Systems with Amazon ECS: https://amzn.to/3g5FPNr
AWS News Blog launch announcement: https://go.aws/3bLWLoP
Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate: https://go.aws/3e53pZ0

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

#AWS #AmazonEFS

Content

0 -> Amazon Elastic File System (EFS) provides a simple, scalable, and fully-managed shared file

6.4 -> system for use with AWS Cloud services and on-premises resources. Whether you’re

11 -> developing a new containerized application or modernizing an existing monolith, there are many

15.1 -> types of applications where a shared persistence layer is an essential ingredient in achieving a

20 -> resilient and durable architecture. These include stateful, long-running applications like content

25.1 -> management systems, whose files change frequently as editors publish new content or

29.14 -> administrators install plugins, and applications that need to share data with one another, possibly

34.1 -> spanning multiple availability zones, such as machine learning frameworks or collaborative data

38.16 -> science workspaces. Amazon EFS is the natural choice for these types of use cases. As a

44.6 -> fully-managed service, it’s simple to set up, and it will scale to petabytes of capacity and several

48.18 -> gigabytes per second of aggregate throughput automatically as you add and remove data. And if

54.1 -> your workload requires higher throughput than that, you can achieve even more by using

58.6 -> Provisioned Throughput! When you use Amazon EFS in conjunction with your containerized

63 -> applications on AWS, you can feel confident that your shared storage solution is as elastic as the

68.2 -> container infrastructure that depends on it. Let’s take a look at an example scenario of one of

73.8 -> these use cases: We have an Amazon Elastic Container Service, or “ECS”, cluster running a mix

79.4 -> of tasks using both ECS on EC2 and AWS Fargate. The containers are all running a web

84.14 -> application called Cloud Commander. We’ll use it as a hypothetical production application for today.

89.14 -> An application such as this might need varying degrees of read and write access to shared

93.16 -> storage, and the familiar interface of a file manager will allow us to visualize these ideas

98 -> more easily. The ECS on EC2 version of this application is running behind an Amazon

103.2 -> Application Load Balancer, and the Fargate version is running behind another. If we open up

108.8 -> the first load balancer URL, we’ll see an empty directory where we can drag-and-drop a file to

112.12 -> upload it, and then double-click on it to view it. Notice though, that if another user, represented by

122 -> this other browser window, loads the same application, the file we just uploaded isn’t there

126.8 -> anymore. The other version of the application running on Fargate has the same behavior, and

132.6 -> neither application can see what has been uploaded to the other. What’s happening here

136.1 -> is that the files we’re uploading aren’t being saved to a common location - they’re being saved locally

140.16 -> to whichever container happens to process the upload request, so there’s no guarantee that

144.18 -> different users would see the same files. That’s definitely not what we want here - so let’s fix it!

149.18 -> In this demo, we’re going to use Amazon EFS to create a shared filesystem, and then attach it to

155.4 -> our existing Amazon ECS and AWS Fargate services. This will enable our example file

160.8 -> manager application to interact with the same files, no matter which container handles our

164.4 -> requests. Once we’ve got EFS hooked up to both versions of our application, we’ll then explore how

169.4 -> AWS IAM Authorization and EFS Access Points can simplify configuring more granular access to

174.16 -> our data for different scenarios. If you’d like to try this out on your own later, you can use the

180 -> AWS Cloud Development Kit, or “CDK”, to deploy the initial state of this demo environment to your

185.8 -> own AWS account. A link to the sample code repository for that is available in the video

189.16 -> description below. But for now, let’s head over to the AWS console, open the Amazon EFS

195.18 -> dashboard, and create our filesystem! Creating a basic EFS filesystem is easy. First, we

202.4 -> choose the VPC the filesystem will be created in and its mount targets. In order for all of our

207.4 -> containers to be able to access it, we must create a mount target in each Availability Zone. As a

212.2 -> general best practice, we’ll choose the private subnet for each Availability Zone for our mount

216.2 -> targets. We also need to select a security group. The default security is pre-selected for us, but

221.18 -> we’re going to choose this “efs-demo-fs” security group that was created by the AWS CDK sample

227.12 -> code so that we can limit access to the filesystem to just our containers. Next, we’re shown

233.4 -> several different filesystem settings that we can configure. We’ll leave most of them alone for this

237.6 -> demo, but let’s give the filesystem a name and enable encryption of data at rest using the default

242.6 -> AWS Key Management Service key for Amazon EFS before moving on. We’ll come back to the

248.2 -> additional client access settings later on in this demo. For now, let’s continue, and then click

252.16 -> “Create File System”. Our EFS filesystem was successfully created! Now let’s go make it

259.6 -> available in the Cloud Commander application running on our cluster! To do that, we open

263.16 -> the Task Definitions link in the Amazon ECS menu, select one of the task definitions, and click

269 -> Create new revision. First, we’ll scroll all the way down to the bottom and click on the name of the

274.4 -> existing volume to modify it. Instead of Bind Mount, we’ll choose EFS as the volume type and

280.4 -> then select the File system ID for the EFS filesystem that we just created. We’ll get into

285.4 -> Access points later, so let’s leave that alone for now. The root directory value tells us which

289.16 -> directory in the EFS filesystem will be mounted as the root directory inside the ECS host. Leaving it

296.2 -> as just “/” will mount the entire EFS volume’s contents. We’ll enable Encryption in transit, and

303.8 -> finally, we’ll leave EFS IAM authorization disabled for the time being and come back to it later.

309.2 -> Since we already had the Bind Mount volume set up earlier that we changed to an EFS volume, we

313.14 -> don’t need to make any changes to the container definition. For reference, if we were adding a

318.6 -> completely new volume, we’d now need to go up to the list of container definitions, click “cloudcmd”,

324.6 -> and add a new mount point under the Storage and Logging section. That’s not the case here, though,

329.6 -> so we’ll cancel out of that, scroll down to the bottom, and click “Create” to create the new task

334.8 -> definition revision. We can start deploying this by choosing the Update service option within the

339.14 -> Actions dropdown menu. Please note that at the time of this recording, it’s necessary to select

344.2 -> 1.4.0 as the platform version. We can otherwise leave the rest of the inputs alone as we click

349.14 -> through each of the screens, and then keep an eye on the deployment status until ECS has

353.14 -> finished starting our updated tasks and stopping the old ones. This process is identical for both

359.2 -> ECS on EC2 services and as well as for Fargate services, so we won’t walk through it step-by-step

364.2 -> in this video. Same as with the previous walkthrough, we just need to wait until ECS has

368.16 -> finished deploying our updated tasks and cleaning out the old ones, and then we’re ready to try

373.2 -> everything out. Now that the cluster is showing only the updated tasks running, let’s

378 -> open up both of our browser windows from earlier and see what happens. The first thing we notice is

382.1 -> that the test files we uploaded at the start of the demo are gone. This is expected, since the file

387.8 -> was previously being saved on a data volume that was no longer available once the container was

391.4 -> shut down, which is one of the things our new EFS filesystem is going to help us avoid. Let’s try

396 -> uploading a few new files. OK, we can upload files and view them when we double-click. After

403.1 -> refreshing the listing a few times, and pretending to be a different user using a different browser, we

408.1 -> can see our files are still there. Now let’s see check and see what happens if we refresh the

413.1 -> window using the other application running on AWS Fargate. Great! Our files are still there

419.16 -> too. By simply creating an EFS filesystem and mounting it to use as a volume in our containers,

424.18 -> we have persistent, shared storage accessible by both of our example Cloud Commander

429.4 -> applications running in multiple containers, and we know that our data will still be there even if we

434.2 -> terminate these containers and launch new ones. But this is a pretty basic use case. In a

439.1 -> real application, we’re going to have several different services that don’t all need the same

443.6 -> level of access to the same filesystem. The traditional approach of relying on POSIX

447.14 -> filesystem permissions can make this very difficult to configure and may even require creating

452.6 -> multiple EFS filesystems to ensure that everything is locked down correctly. Instead, we’re going to

457.6 -> take a moment to dream up a very complicated scenario, and then see how easy it can be to set

461.14 -> up if we incorporate IAM Authorization and EFS Access Points into our solution! So, what’s our

467.14 -> scenario going to be? Well, a service is going to have one of three types of access to a filesystem:

472.16 -> read-only, read/write, or none at all. We have two services in our demo, so let’s have our

478.6 -> complicated scenario involve all of these combinations! We’re going to set things up as

482.6 -> follows: (1) each service will have its own private storage, shared with containers of its same type,

488.6 -> that only it can read and write; (2) each service will have a location it can write to, that the other

494.4 -> service can read from, but not write to; and (3) both services will have access to a common

499.14 -> location that they can both read to and write from like they have now. OK, let’s go back to EFS

506 -> and configure this! We’ll create the Access Points first. An Access Point can be thought of as

510.12 -> an application-specific entry point into a certain location in an EFS filesystem, this lets us manage

516.8 -> things like data access and user identities more easily. We’re going to create some Access Points

521.2 -> for each of the three scenarios mentioned earlier. First, let’s configure the private Access

525.12 -> Points for each of our services that only they are able to read from and write to. Let’s start with

531 -> Fargate. We’ll call the Access Point “FargatePrivate” and have its root directory begin

536 -> at “/private/fargate”. We also have the ability to specify a POSIX User and Owner. The values

543 -> entered under POSIX User will tell the mounted filesystem who we are, and the values under

547.6 -> Owner will define the ownership and permissions for the directory we specified when it gets created

552.6 -> the first time we connect. To keep things easy to remember, let’s use “7000” as the Owner ID and

558.4 -> Group ID for anything related to our Fargate tasks, since the word “Fargate” has seven letters. As for

564 -> permissions, we only want our Fargate tasks to be able to read from and write to here, so let’s set the

569.16 -> permissions to 750. The configuration for “EcsPrivate” is the same, except we’ll input “ECS”

576.18 -> instead of “Fargate”, and we’ll use “3000” for the IDs, since “ECS” has 3 letters. Next, we’ll add

584 -> Access Points for each service that they can write to and the other can read from.

595.4 -> And finally, we’ll add a common location where both services can read and write freely, and then click “Save

602.16 -> access points” to create them. It’ll take a few moments after saving for them to become

607 -> available, at which point we can define how access is going to work. Let’s scroll back up and

611.14 -> set our filesystem policy. There’s no policy in place right now, so our filesystem is using the

616.18 -> default policy of allowing any clients with network access to connect, which is why we were able to

621.12 -> hook up the filesystem to our containers so easily before. To get our policy started, let’s check off

626.14 -> these three settings to configure new defaults: “Disable root access”, “Enforce read-only access”,

632.8 -> and “Enforce in-transit encryption”. After clicking “Set policy”, we can see the JSON tab has been

638.12 -> populated with the familiar IAM-style policy configuration. Now we need to configure the

643.16 -> policy for our own scenarios. We’ll do one together step-by-step, and then fast-forward

648.12 -> through the rest. If you’d like to review any individual policy statement more closely, a copy of

653.2 -> the entire policy is available at the same repository linked in the video description. Let’s

658.6 -> configure “FargatePrivate”. We want to allow our Fargate service, or more specifically, the Task

663.18 -> IAM Role that it’s using, to be able to mount and write to the FargatePrivate access point that we

668.1 -> defined below. We can get that Task IAM Role’s ARN from the Fargate task definition.

678.6 -> It needs to be able to mount and write to the filesystem, and we’ll specify that this policy statement applies to

684.16 -> that “FargatePrivate” access point.

690 -> Since it needs this same level of access for the “FargateShared” scenario, we’ll go ahead and

694.4 -> add that access point here as well. The final policy looks like this, and we can use the “Sid” values to

700.1 -> help us tell at a glance what each section is for. Let’s save this policy and go back to our task definitions to

708.6 -> make the access points available. We’ll quickly run through how to create the new task definition

714 -> revision for the Fargate application. First, let’s add the volume for the “common” access point, making

719.12 -> sure to also check the “Enable IAM Authorization” box so that the filesystem policy we created earlier gets

725.6 -> evaluated. Next, we’ll do the “private” access point for Fargate,

734.18 -> and finally, we’ll do both of the “shared” scenarios - Fargate and ECS on EC2.

749.18 -> We don’t need our “files” example volume anymore, so we’ll remove that before going to the container definition

755.8 -> to add all of these new mount points. Note that since our sample application container uses the “/files” directory

763.8 -> as its root, we need to be sure to mount our access points within that “/files” path in order for them to be accessible.

771.18 -> There’s the mount point configuration for “common”...

778.6 -> both “shared” volumes...

783.1 -> and the “private” volume just for the instance of our application that’s running on Fargate.

789.2 -> The process to do this for the ECS on EC2 application’s task definition is identical, aside from the names, so

795.16 -> we won’t go through that step-by-step in this video. After creating both of the new task definitions and

800.16 -> updating the services, though, we can open up our two browsers again and see that they each have three

805.8 -> directories - “common”, “private”, and “shared”.

810.6 -> They can both read and write from each other’s files under the “common” directory...

815.8 -> read, but not write, each other’s files under the matching “shared” directories...

823.2 -> and they each have a private space for storing files under the “private” directory that the other can’t

828.12 -> interact with at all.

831 -> Support for Amazon EFS on both AWS Fargate and Amazon ECS for EC2 is available in all AWS regions.

838.12 -> To learn more about using Amazon EFS with your containerized applications, including documentation,

843.8 -> blog posts, and the sample code from this demo, please visit the links in the video description below.

848.12 -> Thanks for watching!

Source: https://www.youtube.com/watch?v=FJlHXBcDJiw