AWS re:Invent 2022 - SaaS microservices deep dive: Simplifying multi-tenant development (SAS405)

Aug 16, 2023

AWS re:Invent 2022 - SaaS microservices deep dive: Simplifying multi-tenant development (SAS405)

At some point in building a SaaS environment, the attention shifts to how multi-tenancy will influence how the builders on your team design and code their multi-tenant microservices. Multi-tenancy requires you to introduce new mechanisms to address authorization, data access, tenant isolation, metrics, billing, logging, and a host of other considerations. In this session, dive deep into multi-tenant microservices, looking at the various patterns and strategies that can be used to bring a multi-tenant microservice to life without imposing added complexity on your SaaS builders.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

4.35 -> - All right, we all set?

5.64 -> Great. Good morning, everybody.

7.65 -> Thank you for joining me.

9.66 -> We're gonna talk about building

11.37 -> SaaS microservices this morning

15.48 -> and I'm gonna try and show you some ideas

18.33 -> and some tricks and some patterns

19.8 -> to help simplify the development process

23.13 -> to make your product teams more efficient,

26.19 -> building the things that your customers are asking you for.

29.79 -> My name is Michael Beardsley.

31.38 -> I am a solutions architect and I work

34.17 -> on a team at AWS called the SaaS Factory.

37.47 -> And the goal of the SaaS Factory is really to make AWS

40.23 -> the best place to build your SaaS solutions.

43.83 -> And we do that by interacting with our partners

47.07 -> and our customers, giving best practice and guidance

49.83 -> and we also build a lot of reusable content,

52.83 -> both written content, but also a lot of code samples

55.68 -> that you can find out there on GitHub

57.21 -> that you can take advantage of

58.38 -> to help accelerate your journey.

61.38 -> Just as a reminder, this is a 400 level session.

64.08 -> I'm gonna be showing a lot of source code on the screen.

67.08 -> I hope everybody's excited about that.

69.81 -> And with that, let's dive in.

73.92 -> Let's set the stage here.

75.75 -> Let's make sure we're all talking about the same thing.

78.87 -> So up on the screen, I've got a very generic

83.46 -> microservices architecture and there's a couple of things

86.52 -> that we can notice about this.

88.08 -> One is that there is going to be some sort

91.08 -> of user interface client that you have.

94.44 -> Usually, these are written in a JavaScript framework

97.68 -> like React or Angular, but this might be

100.56 -> a mobile application or some other way

102.69 -> to interact with your microservices.

105.36 -> This user application, this client talks to an API gateway

110.46 -> and the API gateway sits in front

112.26 -> of all of your microservices and presents

114.48 -> a cohesive look at your solution

117.99 -> that's made up of all these different services.

120.42 -> And that API gateway talks to some identity provider.

124.38 -> Here, I've shown Amazon Cognito,

126.42 -> but it could be any identity provider.

128.88 -> The point is that we want to make sure that our requests

132.87 -> against our microservices are authenticated.

135.45 -> We know who's asking to use them and they're authorized,

139.35 -> that they can actually do what they're asking to do.

142.47 -> And then that proxies down to your microservices.

145.2 -> And you'll notice that all of my microservices

148.08 -> are independent of the other.

151.26 -> And this gives us the ability to scale

154.08 -> these microservices independently,

156.33 -> it gives us the ability to deploy

158.76 -> these microservices independently

160.89 -> and it gives us flexibility to use

164.04 -> the right tools for the job.

166.11 -> So you'll notice that each of these microservices

168.81 -> might be using a different type of compute.

171.42 -> Maybe we're using containers in one,

173.79 -> maybe we're using EC2 instances in another,

176.82 -> maybe we're using Lambda functions

178.83 -> and putting all of that behind an API gateway.

182.13 -> You'll also notice that each microservice

184.38 -> owns its own database, owns its own data source.

187.41 -> It's super important for microservices

190.02 -> to maintain their independence and flexibility.

192.45 -> And it also gives us the ability

194.22 -> for each microservice to use the right kind

196.26 -> of data technology for whatever that service does,

199.5 -> whether it's a NoSQL solution, a purpose-built database,

203.1 -> or a traditional relational database.

205.71 -> So this is the kind of architecture

207.72 -> that we want in our heads

209.07 -> as we go through the rest of the talk.

213.93 -> So as we start thinking about moving

216.57 -> to delivering our products through a SaaS delivery model,

221.58 -> or if you're already there, what are some of the things

224.07 -> that might be different that you need to think about

227.85 -> versus a more traditional delivery model?

231.87 -> One is tiering.

232.86 -> So we often like to package up our SaaS offerings

238.47 -> in a way that delivers an experience

241.71 -> that our different customer segments are interested in.

244.59 -> So the classic example here might be like

246.78 -> a standard tier versus a premium tier,

249.39 -> or maybe you've got a free trial or a paid trial tier

252.6 -> and these tiers are gonna impact

254.73 -> how you're writing your microservices

256.53 -> and they're certainly going to impact

258.69 -> how your provisioning your infrastructure

260.49 -> and utilizing resources.

264.09 -> Data is key to succeeding as a SaaS vendor.

269.16 -> You have to know what your customers are doing

271.8 -> with your services and maybe more importantly,

274.32 -> you need to know what they're not doing with your services,

277.14 -> so that you can gain insights into what's going on

281.34 -> so that you can increase their loyalty

283.98 -> and retain their subscription and grow your business.

287.07 -> So gathering up these actions, this audit information

291.18 -> and these metrics that tell you what's going on

293.64 -> with your running system are key in a SaaS solution

296.85 -> and you have to have tenant context

299.34 -> when you gather up this information

301.74 -> and that's what's different about this

303.69 -> than a traditional model.

306.99 -> Now, most customers expect a SaaS solution to be billed

311.1 -> with some concept of consumption-based pricing.

316.53 -> Customers want to pay for what they use,

318.27 -> they don't really want to pay for anything more than that.

320.7 -> So now we have to think about gathering up the data

324.12 -> that identifies what each tenant is consuming

327.75 -> so that we can invoice them properly.

329.49 -> We call that metering.

332.61 -> We're gonna spend a lot of time

333.75 -> this morning talking about identity.

335.88 -> We have to know who's interacting with our microservices,

339.72 -> and more importantly, what tenant they

342.87 -> are interacting with us under

345.21 -> so that we can build isolation into our solutions.

349.26 -> We have to isolate each tenant's data and their actions

353.58 -> from every other tenant in our system.

356.7 -> So these are some of the ideas that we want

358.5 -> to keep in mind as we're building SaaS solutions.

361.98 -> So how does this impact our microservices architecture

364.71 -> that we were just looking at?

366.93 -> Well, on the front-end, we need to be thinking

369.03 -> about things like how are we authenticating these users

371.97 -> that are coming in to access our system?

374.67 -> This is often where we're also gathering up

377.31 -> the information that's gonna be important for routing

379.86 -> these requests to the proper backend resources.

383.34 -> So the metadata, whether it's a subdomain

385.83 -> or it's part of the headers or some other piece of metadata,

389.07 -> usually, this is defined on the front door

392.64 -> as it's coming through that front-end.

394.59 -> And because in SaaS, we want to be running

397.95 -> a single version of the same code base for everybody,

402.57 -> but if we want them to have a little bit

404.19 -> different experience, you'll find that this is also

406.71 -> where we're implementing feature flags

408.39 -> to turn some things off, to turn some things on.

411.87 -> As we go into the API gateway level,

413.85 -> now we're talking about authorization.

416.1 -> Are they authorized to do what they're asking to do?

418.35 -> This is also often a place

420.15 -> where you're looking at throttling,

421.89 -> especially if you've tiered your service.

425.19 -> Caching also happens at the API gateway level.

428.52 -> And as we get into the microservices,

430.47 -> now, we're really getting into the meat

432.18 -> of this data gathering with tenant context

436.14 -> for metrics, logging and of course,

439.74 -> consumption-based billing metering.

443.43 -> As our microservices talk to our data layer,

446.25 -> now, in a multi-tenant world, we have to think about

449.04 -> how are we accessing those data resources?

451.89 -> How have we partitioned our data resources

455.07 -> so that each tenant's data, so that we know

457.59 -> where it is and how to access it?

459.66 -> How are we isolating that data from one another?

462.6 -> Because partitioning is not the same as isolation.

465.93 -> And we're gonna talk about that more a little bit later.

468.24 -> And of course, back up and restore

469.74 -> with data gets really difficult,

471.54 -> especially if you've pooled your data sources.

475.17 -> And of course, all this is running on infrastructure.

478.8 -> How we provision that infrastructure?

481.44 -> What techniques are we using

483.72 -> to implement isolation on that infrastructure?

486.72 -> How are we maintaining it and how is it changing

489.81 -> and evolving and having to be modified

492.63 -> as we bring on new tenants

494.73 -> and as those tenants go through their natural life cycle

497.88 -> of interacting with us as customers?

502.08 -> So there's a lot going on here.

504.45 -> And who has to worry about all this?

506.97 -> The product development teams,

508.68 -> the software engineers that are out there

510.6 -> and we want to make our developers productive

513.06 -> and this is nothing new.

514.17 -> Software engineering has been trying to figure out

515.91 -> how to make developers productive

517.23 -> since the very first programs were written

520.5 -> and one of the main ways that we do that

522.21 -> is through encapsulation.

523.89 -> And encapsulation really just means gathering up

526.59 -> both data and logic into some unit that can be reused,

531.66 -> whether that's a shared library

533.28 -> or whether that's more of even just a pattern

535.08 -> of writing software like object-oriented code

538.17 -> and encapsulation hides away the complexity

541.2 -> of what's really going on underneath

543.6 -> and it gives us a way to be more flexible

547.23 -> as we build our systems and it promotes reuse

551.1 -> and every time we can reuse something,

553.38 -> we're gonna save costs.

555.36 -> We're gonna save costs, both hard costs,

557.7 -> but we're also going to reduce defects,

560.4 -> because now, we just have one copy of this thing

562.38 -> going out there, we can test it and we can reuse it.

566.94 -> It also starts to develop common patterns

569.94 -> of how we want our systems to be built.

572.73 -> This increases the efficiency of your team

575.91 -> who's building this stuff and it makes it easier for you

579.24 -> to bring new people onto your project

581.73 -> and get them up to speed more quickly.

586.5 -> Let's take a look at a common example.

589.77 -> I bet everybody here who's ever written a line of code

592.68 -> has written some sort of logging statement before.

597.12 -> So here's this super simple logging statement.

600.09 -> It's just gonna print this sentence out to its output.

605.01 -> But think about all the complexity

606.99 -> that's happening underneath

608.64 -> that his logging module is helping us do.

613.05 -> This logging library,

614.34 -> it's gotta figure out how it's configured,

616.38 -> should it even be logging at the info level?

620.01 -> If it is, it's gotta figure out what the current thread is,

623.22 -> it's gotta go get the stack trace,

625.41 -> it's gotta get a bunch of metadata

627 -> from the line number and the file name,

629.07 -> it's gotta wire up together

630.93 -> all this metadata like timestamps

635.28 -> and then it's gotta format the output

637.71 -> and then it's gotta manage the actual I/O,

640.59 -> which probably means it's got a buffer, it's gotta flush,

643.29 -> it's gotta synchronize and it's gotta clean itself up

646.86 -> to get ready for the next logging statement.

649.14 -> So there's a lot going on here.

652.2 -> Imagine if, as a developer, you had to do all of this

657.39 -> every time you wanted to write a logging statement,

659.97 -> what would happen?

662.22 -> Well, you got a lot less logs. (chuckles)

665.1 -> Our CloudWatch bill would go down.

668.31 -> But what else would happen?

671.01 -> Mistakes would happen, people would forget a step,

673.95 -> or they'd get a step outta order,

676.38 -> or they wouldn't implement something with best practices.

680.34 -> And so the advantage here of encapsulating

684.63 -> all of this complexity into a logging module,

687.54 -> it makes us more efficient, it makes our software better.

691.95 -> So this is the kind of concept

694.11 -> that we want to keep in mind as we start talking

696.27 -> about some of these things that we need to think about

698.73 -> when we're building SaaS microservices.

703.08 -> So let's start with a non-SaaS microservice.

707.97 -> Let's make a really simple one here.

710.37 -> We're gonna define a function in our microservice

713.717 -> to get a list of orders.

718.2 -> If you don't recognize the syntax that's up on the screen,

720.99 -> this happens to be Python.

723.72 -> I chose Python not because I'm promoting the use of Python,

726.96 -> although it's a wonderful language,

728.28 -> I chose it because even if you have never seen

730.44 -> a line of Python before in your life,

732.12 -> it's pretty easy to understand

734.55 -> and it's also a pretty terse language

736.65 -> and PowerPoint is not the best IDE, so bear with me.

740.262 -> (chuckles)

741.9 -> So what is this microservice doing?

743.7 -> What is this function doing?

745.77 -> Not a whole lot, it's opening up

747.06 -> a connection to Amazon DynamoDB,

749.73 -> it's scanning a table and trying to find

751.92 -> all the order items in that table for that user

755.31 -> and returning a list of order objects to the client.

759.21 -> Nothing about SaaS, nothing about multi-tenancy

762.15 -> going on in here, nothing about tenancy at all.

766.68 -> So what's our first step then?

768.9 -> How do we make this multi-tenant?

772.32 -> We should probably add a tenant.

775.83 -> Seems pretty basic, so here we go.

778.98 -> Here's our same function, not a whole lot's different,

781.8 -> but we've added tenant_id and by adding tenant_id,

785.85 -> we can now ask the real question,

789.48 -> which isn't just give me all the orders for this user,

792.75 -> it's give me all the orders for this user

794.91 -> in the context of the tenant that they belong to.

799.95 -> This also has allowed us to now communicate

803.97 -> with DynamoDB differently.

805.26 -> We can now do a query instead of a scan.

808.89 -> A query is gonna be more efficient,

810.54 -> it's gonna cost you less, it's gonna be faster

813.48 -> and it's going to allow us to use

817.2 -> some more advanced security features

819.18 -> that we're gonna talk about a little bit later.

822.27 -> So this is great.

823.103 -> We've got our tenant_id in here, we can now use it.

825.84 -> How do we get a tenant_id? Where does it come from?

830.82 -> Well, I suggest that the tenant_id

832.77 -> should come through the request.

835.2 -> So let's take a look at what a natural

838.38 -> request flow might look like.

840.66 -> So our users, they ask our application for a list of orders

844.65 -> and the very first thing that our software does

846.6 -> is it says, "Well, I need to know

848.13 -> you are who you say you are," so we ask them to log in

852.63 -> and we interact with some sort of identity provider here

857.64 -> and usually this, nowadays, is implemented

860.7 -> with open ID connect, which is a process

864 -> built on top of OAuth, but you could be

866.22 -> using SAML assertions or something else.

868.44 -> The point is, is that your identity provider,

870.9 -> after authenticating that user,

872.91 -> is going to return a set of tokens.

876.15 -> And using those tokens then,

878.58 -> we're going to redirect our call back into our service

881.52 -> and we're gonna say, "No, we want a list of orders."

883.59 -> And our API gateway then is going to verify

888.21 -> that that security token is valid.

890.58 -> It's gonna make sure it isn't expired,

892.92 -> it's gonna make sure it hasn't been messed with

895.23 -> between the time that it was issued and right now

899.37 -> and a bunch of other checks.

901.17 -> And assuming that that all holds true

903.6 -> and the call is authorized, then of course,

905.58 -> we talk to our microservice,

907.05 -> which does its work and returns to our customer.

912.39 -> So this is our standard flow

913.95 -> and somewhere in here, we're gonna add the tenant_id.

920.37 -> What are these security tokens that I'm talking about?

923.31 -> I'm talking about JSON web tokens

926.7 -> or JWT tokens as some people call them or J-W-T.

930.09 -> So what is a JWT token?

932.52 -> It's really nothing more than a Base64 encoded string

935.61 -> that's split up into three parts.

937.77 -> The first part is the header, gives us the type

940.08 -> and the hashing algorithm that was used

943.59 -> and then the middle part is the payload

945.6 -> and the payload is a list of key value pairs

948.93 -> that tell us about the token, tells us about who issued it,

952.41 -> some of the details about when is it gonna expire,

955.65 -> when should you start being able to use it,

958.14 -> who's the intended audience, stuff like that.

960.54 -> There's a whole list of standard claims.

962.76 -> We call these key value pairs claims, but importantly,

967.17 -> you can add your own key value pairs to these tokens.

972.42 -> And that is how we are going to pass along the tenant_id.

976.44 -> We call these custom claims.

979.47 -> Last part of the JWT token is the signature.

981.45 -> The signature is a combination of the header

983.97 -> and the payload and a secret and we can use that

987.9 -> to ask the identity provider to verify

990.9 -> that no one has modified the packet

993.54 -> between when it was created and when we're using it.

1000.02 -> So now we know about these tokens,

1001.49 -> we know we're gonna use them to get our tenant_id,

1003.83 -> so let's take a look at how we might do that in code.

1006.08 -> So here we are back at our function.

1010.76 -> And now up online, too, you can see I am pulling

1015.23 -> the authorization header out of the incoming HTTP request.

1019.79 -> I'm extracting that bearer_token,

1022.28 -> I'm splitting it up into its parts,

1024.71 -> I'm grabbing out the payload and I'm getting the tenant_id

1029.3 -> as a custom claim out of that payload.

1033.23 -> Now, you might ask why do it this way?

1037.7 -> Why override the JWT token from the identity provider

1041.87 -> with custom information?

1044.69 -> If I didn't get the tenant_id

1045.95 -> as part of the incoming request,

1049.04 -> I have to get it from somewhere,

1052.07 -> which would mean that when my request hits

1054.77 -> and I don't have that tenant context,

1056.57 -> I'd have to go ask some, probably, other microservice

1059.75 -> that I wrote, like a tenant context microservice, let's say.

1063.5 -> Guess what, you're gonna have to do that

1065.15 -> every single time you need the tenant_id

1067.82 -> and by the end of this morning you'll understand

1069.5 -> that you need the tenant_id for everything

1071.87 -> and now you've created a microservice

1074.63 -> that has a single point of failure

1076.46 -> and it's a hotspot in your architecture

1078.95 -> and it should not be what you are worrying about.

1081.56 -> You should be worrying about writing

1083.18 -> whatever your intellectual property and features

1085.43 -> and functionality are that your customers want.

1087.86 -> So we can take advantage of the fact

1089.54 -> that this step has already happened

1091.55 -> and it's happened before we've had our microservice

1094.46 -> and it's secure.

1096.59 -> So that's why we promote this pattern.

1100.49 -> So there's a lot going on in here.

1102.77 -> There's string manipulation, there's regular expressions.

1107.3 -> Who here has ever had a bug in a regular expression?

1110 -> So this is ripe. This is ripe for some sort of reuse.

1113.9 -> We should encapsulate this.

1115.64 -> There might even be more going on here.

1117.32 -> This assumes that the API gateway

1119.06 -> has already verified the token

1120.35 -> and we're not talking to the identity provider.

1124.61 -> You'd think decoding Base64 would be easy.

1127.22 -> Sometimes you gotta pat it out,

1128.81 -> depending on how it was written.

1130.31 -> There's all kinds of stuff going on here.

1131.87 -> So this is the perfect example of where we should create

1135.74 -> some sort of helper for that process that can be reused

1139.61 -> between all of our microservices.

1141.95 -> So let's create a token helper

1143.66 -> that we can get a tenant_id out of any request.

1149.84 -> What next? What else should we do?

1152.78 -> Did anyone notice that there was no error handling

1155.436 -> (chuckles) in our microservice?

1158.24 -> Let's add some error handling.

1160.43 -> And if there's an error, we should probably log it.

1163.28 -> We should probably do something

1164.63 -> about the fact that there was an error.

1166.64 -> So here you see I'm building up my logging message

1171.164 -> and I'm building up a structured logging message.

1175.22 -> I'm not just logging a plain string,

1178.58 -> which means I've got this pattern here

1180.41 -> that I need to follow.

1181.67 -> I need to add tenant context to it

1184.28 -> and I don't want my developers having to remember

1187.25 -> how to build this every single time.

1190.07 -> That's too hard, that's too much.

1192.08 -> And in fact, it's probably more complicated than this.

1195.11 -> I'm probably trying to use

1196.79 -> an even more complicated structure than this.

1198.77 -> So again, ripe for encapsulation.

1203.75 -> So let's add a logging helper.

1206.33 -> So now we've got a token helper

1208.19 -> that can help us with a request.

1209.57 -> We've got a logging helper.

1212.12 -> What else is on our list?

1214.64 -> How about our discussion about consumption-based pricing?

1218 -> Let's say that we wanted to bill

1220.76 -> some of this service based upon the consumption

1224.9 -> of the resources that we have to pay for.

1228.8 -> We can ask DynamoDB for a little bit more information

1231.29 -> about the query that we just ran

1233.36 -> and we can bring back some information

1236.69 -> about the capacity that we consumed

1238.67 -> from that table, the number of records.

1242.21 -> Again, a somewhat complicated package of data

1246.86 -> that we want to build up here.

1248.09 -> We want to build it the same way

1249.56 -> and then we've gotta deliver it to our backend.

1253.79 -> Now here, this is showing an example

1255.68 -> using Kinesis Data Firehose,

1257.45 -> but you could imagine all kinds of different ways

1259.55 -> to ingest data into AWS.

1261.38 -> We have a rich set of services

1263.48 -> and features to pull that off.

1264.83 -> The point is this should also be

1268.13 -> encapsulated and easy to use,

1270.08 -> because you want to be gathering this data as a SaaS vendor.

1273.95 -> So make it easy for your developers

1276.26 -> to gather up a whole lot of data.

1278.69 -> We get asked so often, "Well, what metrics should I have?"

1284.81 -> I don't know. You probably don't know either.

1287.27 -> So just start gathering stuff

1289.49 -> and put it in a structured format

1291.8 -> so that you can query it and learn and find signals

1296.45 -> and find patterns in your data

1298.76 -> and then you can turn off the stuff that you don't need

1301.19 -> and you can add more stuff that you do need.

1306.74 -> How about accessing our data?

1311.12 -> Not only do we have the tenant_id in our security token,

1315.35 -> we can also grab the tier,

1317.27 -> because when you onboard a new tenant,

1319.04 -> a tenant's actually onboarding into a tier.

1322.19 -> And so we know this information when we onboard the tenant,

1324.86 -> so we can add it to the identity provider

1327.14 -> and now we can capture the tier and using that tier,

1329.54 -> we can make some decisions.

1332.66 -> Let's imagine that we've got three tiers to our system here.

1335.72 -> Let's say we've got a trial tier and a standard tier

1338.39 -> and maybe people pay us a little bit more money,

1341.18 -> they get some different features

1343.25 -> and we've got a premium tier

1345.2 -> and for the premium tier tenants,

1346.79 -> let's say they each get their own DynamoDB table,

1350.27 -> different isolation, different noisy neighbor problems,

1353.81 -> different partitioning.

1356.57 -> And so now we can use this tier

1358.25 -> to impact how we access our data,

1360.05 -> so now when we give the DynamoDB client

1363.5 -> the table name, it's a variable at runtime.

1366.74 -> It's not static.

1368.45 -> And again, this whole idea here should be encapsulated

1374.24 -> into some data access layer in your microservice.

1378.38 -> You should simply ask the data access layer,

1380.517 -> "Hey, here's this request.

1382.97 -> Go figure out how to get the data for me."

1389.06 -> Earlier I said partitioning does not equal isolation.

1393.02 -> What did I mean by that?

1395.81 -> Let's take a look at our example here.

1397.33 -> So we've got a couple of different tables.

1398.99 -> We've got our trials table, we've got our standard table,

1401.06 -> maybe we've got a few premium tenants.

1403.52 -> So we've partitioned the data here.

1406.55 -> We've either partitioned it physically

1411.38 -> in the example of the premium tiers

1414.29 -> where they have literally two different DynamoDB tables,

1418.22 -> or we've partied it logically

1420.02 -> by adding some sort of discriminator to the data.

1423.02 -> In this case, we would add the unique tenant identifier

1426.32 -> as the partition key in these pool tables

1429.2 -> and we would use that to filter out

1431.15 -> and know which items go with which tenant.

1434.06 -> So we've split them up, we've partitioned them,

1437.78 -> but we've done nothing about preventing

1440.36 -> somebody from talking to any of it.

1444.17 -> Why is that?

1446.96 -> It's because when you create an AWS SDK client,

1452.3 -> by default, it uses the permissions

1456.47 -> and privileges from what we call

1458.6 -> the default credentials provider chain.

1462.29 -> What the heck is that?

1463.85 -> Well, that's the EC2 instance role.

1468.5 -> If you're running containers, that's the task role.

1471.56 -> If you're running Lambda, that's the execution role.

1474.2 -> So it's whatever IAM permissions you've given

1476.84 -> to the compute that's running your microservice.

1481.25 -> And so that SDK client is gonna have access

1483.8 -> to all of these tables.

1486.2 -> So they're not really isolated, they're partitioned,

1489.83 -> but there's nothing preventing me from accessing something

1493.13 -> that I shouldn't be accessing

1494.96 -> in the context of an incoming request.

1498.02 -> So how can we fix this problem?

1501.2 -> It turns out that you don't have to create

1504.44 -> the SDK client with the default credentials.

1506.87 -> You can create the SDK client

1509.06 -> with credentials that you provided at runtime

1512.66 -> and therefore, we can change the security posture

1515.21 -> of this SDK client.

1516.89 -> This is true for any of the SDK clients, not just DynamoDB,

1521.9 -> before we start using it.

1524.27 -> So how do we do that?

1526.25 -> Here we are back at our flow.

1529.67 -> We've gotten to our microservice,

1530.96 -> we have an authenticated authorized request,

1533.99 -> we're ready to go.

1535.16 -> We're ready for the microservice to start doing its work,

1537.38 -> but first, our microservice is going to go get

1542.21 -> a set of scoped down permissions

1545.84 -> that align with the tenant context of the incoming request

1550.82 -> and then we're gonna use those permissions

1552.98 -> to create a temporary set of AWS credentials

1557.36 -> and these temporary credentials are gonna time out,

1559.61 -> they're gonna be short-lived,

1560.63 -> so even if something horrible goes wrong,

1563.63 -> you've got a small blast radius 'cause they go away quickly.

1566.81 -> And we're gonna use those credentials

1568.28 -> to access our AWS resource in the context

1571.37 -> of the tenant who's accessing our microservice

1574.79 -> and then we're gonna return the response.

1579.74 -> So how do we do that flow?

1581.39 -> How do we get those runtime credentials?

1586.19 -> There's two main ways to approach this

1590.45 -> and they both have their pros and cons.

1593.24 -> So one way would be to create an IAM role

1598.22 -> and a set of permissions for each of your tenants.

1602.87 -> And really, you'd probably have to do this

1604.88 -> for each of your microservices, for each of your tenants.

1608.78 -> Potentially, for each tier.

1612.02 -> But you set this up and then you use

1615.71 -> a feature of Amazon Cognito called identity pools.

1620.39 -> Not to be confused with, but always is, user pools,

1624.17 -> identity pools are completely different.

1626.93 -> It's completely different service

1628.37 -> and what identity pools do is it allows you

1630.77 -> to exchange a Cognito identity for a set of AWS credentials.

1635.54 -> Perfect. That's what we need.

1638.48 -> And so what are the benefits of this model?

1641.48 -> The benefits are that it's really easy to understand

1645.502 -> all of the security posture of all of your tenants,

1648.86 -> it's right in your face, you can go look it up at any time,

1651.41 -> it's sitting in IAM and you're using a managed service

1656.42 -> to do some of the wiring to get you

1657.89 -> the right one at runtime, that's nice.

1662.93 -> What are some of the downsides?

1664.34 -> Like pretty much every AWS service,

1667.37 -> there are account limits in IAM.

1670.22 -> You can only have so many roles and so many policies

1673.22 -> and so if you have a lot of tenants

1675.2 -> and you've got a lot of microservices,

1676.76 -> this might not work for you.

1679.13 -> Plus, if you want to change any of those policies,

1682.49 -> right now you're thinking about a batch process

1684.74 -> across a whole lot of IAM roles, that gets shaky, too,

1688.97 -> so this doesn't scale super well,

1693.38 -> but it is easy to understand and it's convenient.

1696.83 -> So what if you can't work with those limitations?

1699.62 -> What are our other options?

1701.06 -> Our other option is to do some of the work ourselves.

1707.24 -> So here, we can use what we call an identity policy,

1712.58 -> which is the green circle on the left.

1715.16 -> The identity policy contains all the permissions

1718.25 -> that our microservice might need

1720.59 -> and then at runtime, we can create

1722.81 -> what we call a session policy

1725.63 -> and we can lay that session policy

1728.12 -> on top of the identity policy

1731.45 -> and the intersection of those two,

1733.82 -> which is the little yellow shape in the middle,

1736.28 -> represents the set of permissions

1738.74 -> that we want to be able to use at runtime

1742.76 -> and we get to this by calling one of the assume_role methods

1747.44 -> from the Security Token Service API.

1750.77 -> STS is part of the IAM service.

1756.8 -> Let's take a look at what these policies look like here.

1759.71 -> So here would be the identity policy.

1761.72 -> This would be the policy for your microservice.

1764 -> Here we're saying we're gonna allow, read and write

1767.48 -> and update and delete, all the CRUD methods, on DynamoDB

1771.5 -> for the different tables

1773.3 -> that our order service has to worry about.

1777.08 -> So this is our identity policy

1778.67 -> and now we're gonna lay on top

1780.38 -> of this identity policy our session policy.

1785.78 -> And what our session policy is gonna do

1787.85 -> is it is going to restrict, it's going to take away

1793.82 -> some of what's up here in our identity policy.

1798.8 -> So our session policy, same format,

1801.59 -> still a IAM policy JSON document.

1807.53 -> And so now we can start limiting

1811.19 -> what we want out of that identity policy.

1813.38 -> So instead of all the actions, let's say,

1815.27 -> now, we just wanna read only.

1816.8 -> So now, we're just gonna allow query here.

1820.31 -> Now, note that you can't add actions

1824.54 -> in your session policy that don't already exist

1827.93 -> in the identity policy.

1829.88 -> So for example, I couldn't add, say, S3 getObject here,

1836.36 -> because my identity policy doesn't have

1838.49 -> any permissions to talk to S3.

1840.95 -> So at runtime, I can't increase my permissions,

1845.48 -> I can just reduce them.

1848.93 -> And now here, I can restrict which tables.

1852.62 -> If I know that, in this case, this should be

1854.57 -> a read-only action against a tenant

1856.43 -> that's in my standard tier,

1857.72 -> I should only be talking to the tenant table.

1861.83 -> And remember earlier when I said

1863.81 -> when we switched from a table scan to a query,

1867.86 -> it opened up some other options for us

1870.35 -> and those other options, IAM calls request conditions.

1875.12 -> And so I can add a request condition on top of this policy

1880.46 -> that says for leading keys in DynamoDB,

1884.72 -> which means the partition key,

1888.68 -> the value of the partition key for that item

1891.32 -> has to equal this unique tenant identifier

1895.79 -> for me to be able to do a query against this resource.

1899.81 -> So we've now basically implemented row level security

1903.14 -> inside of DynamoDB and we can do it at runtime.

1910.85 -> So what does this look like in code?

1913.58 -> We've got our two flavors.

1915.2 -> We've got our identity pools flavor

1917.39 -> and then we've got our do-it-ourselves flavor.

1922.64 -> So identity pools are not, maybe, the easiest to understand,

1926.84 -> but the idea is is that you get

1928.49 -> this common Cognito identity object

1933.38 -> and then you give that identity object to the identity pools

1937.64 -> and the identity pools will return you

1940.46 -> what it calls the authenticated role for that identity.

1946.52 -> And it can do role mapping by looking at the claims

1949.22 -> and the JWT token to figure out which IAM role

1952.04 -> should be the authenticated role for this identity.

1956.72 -> So these tenant credentials that you see here,

1959.39 -> these are the scoped down credentials.

1961.37 -> In this case, they're the ones that you pre-populated

1964.07 -> when you onboarded that tenant for this microservice.

1969.29 -> Now, every SDK does this a little bit differently,

1973.01 -> but all the SDKs have the ability to do this,

1977.15 -> which is to override the default

1980.96 -> credential provider chain at runtime.

1983.81 -> So in Python, we create what's called a boto3 session

1989.63 -> with these runtime credentials

1992.84 -> instead of whatever the default credentials are

1995.12 -> that it's running on your compute.

1997.1 -> And then using that session,

1999.29 -> we ask for the SDK client that we want.

2003.22 -> And so now this DDB client that I've got here

2006.79 -> at the bottom of the screen,

2008.74 -> all of its actions are going to be restricted

2011.8 -> to whatever permissions are available

2014.11 -> inside of the set of credentials that I just gave it.

2019.66 -> What does it look like if we're gonna build our own?

2025.45 -> I don't have a policy, so I gotta build one at runtime.

2028.18 -> So I should make some template out of my policy.

2031.54 -> So here, I've got a template object

2033.37 -> and you see it's got four variables in it.

2037.75 -> And at runtime, I can fill out those four variables.

2041.02 -> Remember, I got which table to talk to

2043.42 -> from looking through my tiers and I got my tenant-id

2047.08 -> and maybe I need a couple other little pieces of information

2049.57 -> to build up the ARN for the resource that I want to talk to.

2054.25 -> And now, I'm not using identity pools.

2058.24 -> So I'm not using Cognito anymore.

2060.13 -> Now, I have to interact with STS myself.

2065.71 -> So I ask STS to assume this identity role.

2072.46 -> And because I'm now calling assume_role myself,

2076.03 -> I can pass in my scoped down policy.

2080.26 -> Now, when you call getCredentials

2082.96 -> for identity on identity pools,

2085.12 -> it's doing assume_role for you behind the scenes.

2090.28 -> So it's saving a little bit of latency,

2092.29 -> because it's making that call

2094 -> to the STS service on your behalf,

2096.97 -> up on AWS before the return from Cognito comes back.

2100.84 -> But this gives us more flexibility.

2106.45 -> And then just as before, same thing,

2108.52 -> you now have a set of scoped down credentials.

2110.95 -> You create a session with that

2112.87 -> and you create your SDK client with it.

2117.85 -> Now you may be thinking, "This is great. I get it.

2120.85 -> It's complicated. There's a ton of steps.

2122.65 -> I'm gonna encapsulate this and I'm gonna have

2124.66 -> some sort of shared library that vends out

2127.96 -> my SDK client and my developers are gonna say,

2130.48 -> 'Hey, some sort of factory builder pattern

2133.33 -> or something, give me a DynamoDB client,

2135.55 -> here's the request that I want

2136.87 -> this client for,' and that's great.

2139.87 -> What if my developers forget or don't know

2142.81 -> and they just call boto3.client, what's gonna happen?

2147.16 -> They're gonna circumvent all this fancy security

2149.38 -> that I just spent all this time trying to build."

2152.56 -> One of the neat things about this pattern here

2157.15 -> is that because we're no longer using

2159.55 -> the default credentials,

2161.95 -> we don't have to give those default credentials

2165.91 -> any permissions to talk to DynamoDB.

2171.01 -> So even if your developer creates a DynamoDB client

2175.63 -> outside of the scope of your patterns here,

2179.5 -> they're not gonna be able to do anything,

2180.82 -> because those credentials aren't gonna

2182.41 -> have access to any DynamoDB actions.

2190.09 -> So we've talked a lot about code

2193.3 -> and I wanna leave you with some thoughts about DevOps.

2197.68 -> So in a traditional model, DevOps is really

2200.2 -> about provisioning the infrastructure

2203.86 -> and deploying your workloads

2206.47 -> to get your software up and running.

2210.88 -> And that's essentially a one time deal.

2213.07 -> It's one time every time you update your software.

2216.13 -> Some companies have super fancy CI/CD pipelines

2219.46 -> and they're constantly setting up

2221.11 -> the infrastructure and deploying their code,

2225.19 -> but you're only doing it when your code changes.

2230.47 -> In a multi-tenant world, in a SaaS world,

2233.41 -> you have to think about what impacts

2235.78 -> does onboarding a new tenant make to your DevOps world?

2241.06 -> When you onboard a new tenant,

2243.01 -> are you gonna have to provision some infrastructure

2245.35 -> to support that tenant?

2247.78 -> You're probably gonna have to create

2249.43 -> some of these security policies

2250.81 -> that we've just been talking about,

2252.04 -> whichever flavor you want.

2255.43 -> You're gonna have to set up those custom claims

2257.86 -> in your identity provider.

2260.23 -> You might have to deploy some workloads.

2262.24 -> It depends on how your architecture is set up.

2265.3 -> You're probably gonna have to wire up a billing integration

2268.87 -> or some other third party integration

2271.18 -> that your solution depends on.

2273.49 -> And it's not just when you onboard a new tenant,

2276.37 -> you also have to be thinking about what happens

2278.83 -> as the tenant continues to exist as a customer for you

2281.92 -> and goes through some of its natural life cycling.

2285.55 -> What if a tenant moves tiers?

2288.55 -> What if they're in your trial experience and they love it

2291.97 -> and they're like, "I am ready to commit, I'm ready to go."

2294.88 -> Are you gonna bring along that experience that they've had

2298.66 -> or are you gonna make them start over again?

2302.14 -> What if you're moving data from a pooled resource

2305.8 -> into a siloed resource or vice versa?

2308.14 -> How are you gonna deal with that?

2310.3 -> What does it mean to your infrastructure

2312.91 -> to disable a tenant?

2316.57 -> It might mean changing a load balancer rule

2321.61 -> to have a fixed redirect to say,

2324.947 -> "You haven't paid your bill. Please call customer support."

2327.272 -> (chuckles)

2328.51 -> It may mean moving them to a different tier.

2332.62 -> It may mean throttling them.

2334.36 -> It could mean all kinds of things

2335.71 -> and it's gonna be different for everybody's solution.

2338.56 -> What does it mean if a tenant decides to leave?

2343 -> What if they unsubscribe? What if they ask to be forgotten?

2346.29 -> (chuckles)

2348.46 -> So these are the kinds of challenges that we have

2350.53 -> to think about in a SaaS world with DevOps.

2357.25 -> Now, we've talked about some of the things

2359.32 -> that we should be encapsulating and some of the things

2361.33 -> we should be figuring out how to reuse,

2364.33 -> but depending on whether we're using EC2 instances

2368.38 -> or containers or lambda functions,

2371.32 -> the way that we can share

2374.29 -> those constructs might be a little bit different.

2377.41 -> And so I want to cover that here.

2379.96 -> And I've chunked these up into three categories.

2382.48 -> The first category, I will call shared libraries.

2386.32 -> And this is the basic, this is the traditional thing

2389.02 -> that all of us know about.

2390.82 -> This is the traditional packaging,

2392.5 -> whether it's a Java archive and the Java world

2395.35 -> or a dll.net world or node modules, what have you.

2399.58 -> So these are shared libraries in your technology space.

2404.32 -> They're packaged with your microservice, they're versioned,

2409 -> and importantly, they run in the same process

2412.12 -> as your microservice.

2414.1 -> So they're running in the same memory space,

2417.19 -> in the same process, in the same virtual machine

2419.77 -> as your microservice and because they're running

2421.78 -> in the same process, it means

2423.64 -> that you pretty much have to use the same language.

2427.81 -> So whatever you wrote your microservice in

2429.43 -> is probably what you've written your helper in.

2434.71 -> This works great on EC2 and in containers.

2438.7 -> When you build your container,

2439.81 -> you have to remember to build your shared libraries

2442.69 -> and include them when you build your image,

2445.12 -> so that it's all there at runtime.

2448.63 -> But this is pretty straightforward.

2450.07 -> We've been doing this for years and years and years.

2453.01 -> Now, with lambda functions,

2455.86 -> this gets a little bit more complicated,

2457.39 -> because a lambda function is packaged up

2462.28 -> individually as each function.

2464.44 -> So your microservice is probably made up

2466.99 -> of a whole bunch of Lambda functions.

2469.18 -> It's one microservice, it's the orders microservice,

2471.73 -> but maybe there's 10 functions

2473.23 -> that make up the orders microservice.

2475.51 -> Each of those 10 functions need to be able

2477.22 -> to access your shared library.

2479.41 -> So now you've gotta build that shared library,

2481.48 -> bundle it into your Lambda function and deploy it.

2488.02 -> But that now means you've got 10 copies

2490.27 -> of your shared library floating around out there.

2493.42 -> And one of the values of using Lambda functions

2496.3 -> for your microservices is you don't have to change

2499.09 -> all 10 of them all at the same time.

2500.65 -> Again, it just it just makes our agility

2505.18 -> more and more and more fine grain.

2506.68 -> So if a microservice can be independent

2508.87 -> and you can deploy and scale

2512.98 -> and deal with each microservice independently,

2515.26 -> now with serverless microservices,

2518.86 -> you can do it at a function level.

2521.65 -> So you could find yourself in a place

2523.42 -> where you've got drift in the versions

2526.42 -> of your shared libraries if you're not careful.

2530.98 -> And boy, wouldn't that be a hard bug

2533.89 -> to figure out what the heck is going on.

2535.96 -> Why one little piece of your orders microservice

2539.41 -> is doing something than all the rest of it.

2541.42 -> So how do we solve that problem in Lambda?

2543.4 -> We came out with Lambda layers.

2545.14 -> So Lambda layers is a mechanism

2548.14 -> for you to do shared libraries in Lambda.

2551.2 -> Lambda layers are independently versioned,

2554.98 -> provisioned and deployed,

2557.647 -> and you attach them to your function.

2559.39 -> So now you have one copy of your shared library

2562.87 -> that you can attach to all the functions

2564.82 -> inside of your microservice.

2569.74 -> The next bucket, I'll call background processes

2572.23 -> generically here.

2573.28 -> So the classic example of this would be

2575.89 -> a background process running as an agent

2579.31 -> or a daemon on the operating system.

2582.46 -> Now, these shared constructs do not run

2586.66 -> in the same process as your microservice.

2589.57 -> So these are now independent of your microservice

2592.78 -> and your microservice is gonna communicate

2595.06 -> with these background processes,

2597.46 -> either through the network stack

2600.01 -> or through whatever the operating system uses

2602.56 -> for inter-process communication,

2604.48 -> whether that's pipes or sockets or what have you.

2607.84 -> So a good example of this might be

2609.43 -> something like the CloudWatch logs agent

2613.09 -> that is running on all your EC2 instances

2616.39 -> if you built them from an Amazon AMI

2619.24 -> and its job is to sit there and watch for log files

2624.16 -> and log output and gather it all up

2626.95 -> and buffer it all up and figure out

2628.81 -> how to submit it out to CloudWatch.

2633.37 -> So this works great on EC2.

2636.49 -> You're in control of the operating system.

2639.58 -> You're in control of all the processes

2641.32 -> that your microservice is running next to.

2644.32 -> It doesn't work so hot on containers,

2646.33 -> because, by definition, containers only run one process.

2652.54 -> That's why we have to give our image definitions

2655.6 -> the command that we want it to execute when it runs.

2659.77 -> If anybody's ever tried to run a cron job

2662.23 -> inside of a container that's also trying to run

2664.96 -> something else, you know what I'm talking about,

2666.76 -> the whole point of Docker was to only run

2669.4 -> one process at a time.

2671.83 -> So how do I run a background process in a container?

2677.08 -> We can use the sidecar pattern.

2679.99 -> And the sidecar pattern is really just another container

2683.86 -> next to the container that is your microservice,

2686.47 -> but because they're running together

2689.14 -> inside of Docker, Containerd

2692.32 -> or whatever is running your containers,

2695.41 -> now, these two containers can inter-operate with each other,

2699.85 -> usually over the local host loop back address.

2703.6 -> So you would expose your different helpers,

2705.55 -> maybe on different TCP ports,

2707.29 -> and because you're not actually going out on the network,

2710.92 -> it's a very low latency high performance.

2717.52 -> Maybe a good example here might be the Fluent-Bit sidecar.

2722.74 -> Again, having to do with logging,

2725.29 -> you send all your log messages to Fluent-Bit or Fluentd

2730.54 -> and then it deals with gathering stuff up.

2735.64 -> I should mention that Lambda functions

2738.01 -> aren't a long running process.

2739.39 -> So there is no concept of background processes in Lambda,

2742.9 -> because there's no background.

2745.12 -> So the last category here, I'll call it interceptors.

2751.51 -> So the first flavor of interceptor would be network proxies.

2756.43 -> So what I mean by interceptor is something

2759.7 -> that is intercepting the flow,

2761.98 -> the natural flow of your microservice code

2766.87 -> before it is invoked and after,

2771.52 -> so before the request comes in and after the response

2774.1 -> is going out to your client.

2776.8 -> So a good example here might be like the Envoy proxy,

2779.98 -> which is what enables service meshes

2782.5 -> in Istio in the containers world,

2787.54 -> or maybe a Squid reverse proxy

2791.26 -> for people who have been doing this

2792.49 -> a little bit longer than containers.

2794.41 -> Same idea. It's intercepting the incoming traffic.

2797.68 -> It can affect that incoming request

2799.93 -> and it can affect the outgoing request.

2801.76 -> The downside of this is it doesn't really have

2804.1 -> as rich of context about what the request is,

2806.98 -> because it's not running alongside your microservice

2810.25 -> and it only has the information that's available

2813.16 -> on the incoming network request.

2818.8 -> So how do you get beyond that?

2821.29 -> You can run the same concept of intercepting

2824.44 -> the request coming in and the response going out,

2826.87 -> but you can do it right next to your microservices code

2830.5 -> with some sort what I'll generically call middleware.

2834.67 -> Now, all the major frameworks

2836.26 -> have some concept of middleware,

2838.48 -> whether it's node Express or go-chi

2841.92 -> or the Spring framework or Message Handlers in .NET.

2844.96 -> Everybody's got some concept of the request

2848.56 -> is coming into my microservice,

2850.72 -> I've got all the context of that request,

2853.75 -> I can interact with it and do stuff with it,

2856.78 -> I can run the main functionality of my microservice

2860.38 -> and then I get a handle on that response

2862.72 -> before it goes out the door.

2866.41 -> Again, these two concepts work great EC2 and in containers,

2873.91 -> but they don't work real well with Lambda,

2876.67 -> because you're not in charge of the life cycle of Lambda.

2880.51 -> Your Lambda function gets lit up

2882.1 -> and torn down by the Lambda service.

2886.75 -> So you don't have a way to hook into what happens

2889.54 -> before it gets lit up or after it gets torn down.

2895.93 -> But you know that there must have been some way to do it.

2898.45 -> How does X-ray work if there's no way to understand

2902.32 -> what's happening before the Lambda function lights up

2905.8 -> and after it comes down?

2907.75 -> So clearly, AWS had a way to do it,

2911.02 -> and so we've now exposed that functionality

2913.45 -> in a set of APIs that we call Lambda extensions.

2916.93 -> So there are now APIs to give you hooks

2919.72 -> into the life cycle of your Lambda function

2922.69 -> where you're told your function's about to light up

2926.41 -> and you're told your function is done

2928.06 -> and I'm about to return back to the client.

2930.73 -> So this is a really powerful way now

2932.77 -> to implement middleware techniques in Lambda functions.

2937.33 -> Lambda extensions are bundled up and packaged

2939.88 -> and delivered just like Lambda layers are.

2943.33 -> So they count against the number of Lambda layers

2946.93 -> that you can have for a function

2949.09 -> and they impact, of course,

2950.77 -> the code size of your Lambda functions,

2954.01 -> but there's a lot of neat ideas that you could now imagine

2958.75 -> as you're thinking about the logging

2960.55 -> and the metrics and some of this other stuff

2962.8 -> that you could now implement using middleware.

2969.01 -> So what are our takeaways?

2974.14 -> Multi-tenancy is really gonna impact

2975.88 -> all the different parts of your microservices architecture,

2978.55 -> so we have to be thinking about it.

2981.16 -> We want to simplify the fact that we have to think about it

2984.07 -> by using encapsulation to hide away some of the details

2986.95 -> and provide these reusable constructs

2988.93 -> to our product development teams.

2992.29 -> If you don't take anything else away from today,

2994.63 -> please think about using runtime security policies

2999.16 -> as you access AWS resources.

3001.89 -> It's a very powerful solution that we have

3004.8 -> for multi-tenant workloads on AWS.

3010.17 -> Depending on what you're running your microservice on,

3012.84 -> you might have some different options available to you

3016.11 -> in terms of how to reuse these encapsulations.

3021.9 -> And think about how you could squirrel away

3025.56 -> some of these things in middleware,

3028.8 -> so that your microservices code

3030.6 -> doesn't even need to know that it's happening.

3036.84 -> We have a ton of SaaS sessions this year at re:Invent

3040.98 -> More breakouts like this one, workshops,

3045.15 -> Chalk Talks, which are a little bit more

3046.77 -> of an intimate experience where you get

3049.41 -> to interact more with the presenter

3053.97 -> and the SaaS Factory has a ton of content and functionality

3058.95 -> available to help you accelerate on your SaaS journey.

3064.74 -> So I encourage you to check some of that out.

3070.44 -> So thank you, that's what I have for you today.

3072.24 -> Hopefully, everybody learned something.

3073.769 -> (audience clapping)

Source: https://www.youtube.com/watch?v=NpThwz0z_D0