AWS On Air ft. Amazon CloudWatch Evidently | AWS Events

Aug 16, 2023

AWS On Air ft. Amazon CloudWatch Evidently | AWS Events

AWS On Air Hosts Darko Mesaros, Sr. Developer Advocate, Sohan Maheshwar, Sr. Developer Adovate, Adel Lahlou, Sr. Software Developer, and Frank McQuillan, Principal Technical Product Manager demonstrate Amazon CloudWatch Evidently is a new capability which helps application developers validate new features across the full application stack.

For more information, visit:
https://aws.amazon.com/about-aws/what…

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#AWS #AmazonWebServices #CloudComputing

Content

0.71 -> (upbeat music)

10.43 -> And we're back.

11.38 -> Hello, hello, hello, we have Frank and Adel here with us.

16.858 -> So welcome.

19.01 -> Before we start about the things we gonna be talking

22.29 -> about today, please do introduce yourselves.

25.2 -> Frank, where do you join us from?

26.51 -> What do you do?

27.393 -> And why should we care?

31.55 -> Well, good afternoon.

32.59 -> Good evening everyone.

33.64 -> My name is Frank McQuillan.

35.57 -> I am joining you from the Bay Area.

38.46 -> I would love to be at re:Invent, but I'm in the Bay Area.

42.07 -> Happy to be joining virtually,

44.01 -> and I am a product manager on the CloudWatch team

48.43 -> and pleased to be working with Adel.

50.66 -> Adel who are you?

52.743 -> Hey, thanks for having me today.

55.48 -> My name is Adel Lahlou.

56.55 -> I am an engineer working on the CloudWatch team.

61 -> I'm based out of the Bay Area.

62.61 -> I am lucky enough to be here in Las Vegas during re:Invent.

66.27 -> So I'm happy to join you all.

69.65 -> Thank you, thank you.

70.972 -> A lot of folks here are joining us

72.144 -> from their hotel rooms,

73.65 -> but we have Frank joining us from home.

75.68 -> So very good, very good.

77.5 -> So tell us, what are we launching today?

80.61 -> What are we talking about today?

82.99 -> Well, today we are talking

84.2 -> about Amazon CloudWatch Evidently,

87.27 -> which is a new feature of CloudWatches

89.72 -> digital experience monitoring capability.

92.95 -> Just launched a day,

94.47 -> generally available out there right now, as we speak.

98.79 -> Awesome, what kind of problems this Evidently solve?

103.74 -> Well, it's designed to help

106.12 -> application developers validate new features before release.

110.74 -> So it addresses questions like,

113.57 -> is this feature better?

115.34 -> How can I roll out this feature carefully?

118.28 -> So Evidently allows for experimentation on new features,

122.21 -> and it provides quantitative insights

125.97 -> based on those experiments with easy to follow

129.558 -> easy to understand results.

132.32 -> And it also allows for launching features

135.03 -> in a controlled manner.

136.74 -> So launching features in a safer manner during which time

142.1 -> you can monitor metrics and only dial up traffic

145.43 -> when it is safe to do so.

148.94 -> And so my understanding is that kinda

151.01 -> monitors for these like features you've launched,

153.49 -> and what does it actually monitor?

154.8 -> Does it look like metrics and elementary?

159.34 -> What does it, what does it actually monitor?

162.4 -> Well, it can monitor both business

164.72 -> and performance kinds of metrics.

167.03 -> So in the area of business metrics,

170.04 -> this could be anything that you care about with respect

172.81 -> to your application.

174.14 -> It could be checkouts,

175.42 -> it could be revenue could be click through rate.

178.374 -> Whatever's important for your business.

182.15 -> And however that's manifested in the introduction

185.42 -> of your customers with your application.

188.64 -> And the other important side is performance,

191.653 -> because we wanna make sure that our customers can build

196.291 -> applications that are performing and they can monitor

199.77 -> those during experiments and launches.

202.366 -> And so we just had a session

204.24 -> on CloudWatch RUM for real user metrics.

209.1 -> You can actually use RUM metrics directly

212.89 -> within Evidently in your launches and experiments.

217.26 -> Awesome, I always thought like when you deploy

220.36 -> your code, your code is not just,

224.21 -> is it performing good?

225.37 -> Is the CPU utilization,

228.71 -> memory utilization this or that?

230.56 -> It is also is it actually doing something

232.72 -> so Evidently gonna actually help you

234.15 -> as your code change perform.

236.53 -> Does it still work?

238.819 -> Is there a checkout issue with people

240.448 -> and all of those things?

242.83 -> So wonderful.

244.063 -> And

244.896 -> how did the customers you do this before,

246.5 -> before Evidently it was a thing?

251.458 -> There are a variety of ways that customers

253.41 -> kinda handled this problem.

256.45 -> One is to do code deployments for every feature.

258.85 -> So every time you have a new feature,

260.38 -> you deploy your code to production after your usual

263.21 -> kind of dev tests cycle.

268.018 -> And customers could also use configuration files,

272.29 -> which kind of gets you part way there.

274.93 -> And as well as having a variety of in-built methods,

278.59 -> that they've generated within their organization,

281.01 -> that they use in order to do this feature release cycle.

283.64 -> But I think the key thing is that without automation

287.88 -> and without capability like Evidently,

290.55 -> it means less frequent releases.

294.259 -> It means slower feature development cycle

297.2 -> that is not moving at the speed of business.

299.43 -> And it also means less experimentation

301.39 -> because experimentation is hard

302.75 -> to set up and run and interpret.

308.32 -> And let's talk about the type of applications

311.06 -> that Evidently can help with evidently.

314.788 -> You spoke, you touched upon a few of them.

318.01 -> Can we afford that maybe?

319.88 -> Sure, I would say a wide range of features,

323.84 -> both large and small,

326.48 -> for example, it could be a new user experience.

328.95 -> Let's say that you're an e-commerce enterprise

332.59 -> and you have something as small as a checkout button.

335.54 -> What does that checkout button look like?

337.869 -> What size is it?

338.702 -> What color is it?

340.156 -> If you have a new checkout flow that is more streamlined,

343.95 -> that's one example.

345.48 -> So let me use your experiences.

347.21 -> Another one is in the area of machine learning.

350.01 -> So this is not training the machine learning models,

353.26 -> but once you have a trained machine, very mild,

355.89 -> so that you felt unsafe maker, how does it compare?

358.95 -> How does one model compare with the other,

362.32 -> for example, in the area

363.46 -> of recommendations for new products,

366.76 -> for your customers?

368.67 -> And then a third category

370.16 -> would be on service side implementation.

372.69 -> So let's say that you want to add a caching layer in front

376.446 -> of your database for accelerated performance.

379.96 -> That's a new feature, a new server-side

382.49 -> feature you wanna roll out.

384.05 -> You may wanna do experiments

385.18 -> with that in controlled launch.

388.43 -> There is the question here from Randall coming in.

390.24 -> So regarding the SageMaker and Evidently,

392.37 -> can Personalize and Evidently be used together in this case?

395.93 -> Like personalized as a feature, as a service.

400.14 -> So that's a great question.

402.39 -> Actually one of the folks within Personalize,

407.76 -> is actually in the process of developing

410.66 -> a reference architecture,

413.1 -> which shows how that you can,

414.52 -> how you can use Personalized and Evidently together.

418.39 -> So as usual, we're always interested

421.99 -> in new integrations between different services within AWS.

428.22 -> And this is an excellent one

429.5 -> because Personalized is a great,

433.4 -> venue for testing to see

435.54 -> how fact up your personalization methods are.

440.18 -> You entered and you mentioned extra experimentation.

441.93 -> So how would experimentation in this case work?

446.02 -> Experimenting is very important

447.37 -> when it comes to deploying software,

449.16 -> you wanna make sure that you can try out things

451.46 -> and maybe those things

452.36 -> are gonna bring you a lot of good things.

454.07 -> So how would it work in this case

456.181 -> when we use Evidently?

459.729 -> So that's a key point Darko

461.34 -> is that you may have a great idea for a feature.

464.61 -> You may know your customer base very well.

468.15 -> There's really no guarantee.

469.32 -> So testing is really key.

470.72 -> So I, for example, we work with customers

473.26 -> who have a culture of experimentation

478.24 -> that do literally hundreds of experiments per year.

481.87 -> So the way that it works within Evidently

484.55 -> is that it allows you to do extra experimentation

488.72 -> without having statistical knowledge.

491.539 -> So experiments are very easy to set up.

493.57 -> They're very to run and the results

496.75 -> themselves are presented in a clear way.

498.96 -> So if you're doing an experiment between this feature

501.82 -> and this other potential treatment

503.35 -> that you're interested in,

504.84 -> it will say this feature is better.

507.72 -> This feature is not better

508.64 -> or perhaps there is no statistical difference

511.46 -> between these features.

514.23 -> I think this all sounds amazing.

516.74 -> All, a lot of things are going on here.

519.87 -> I would love that Adel here shows us this entire thing.

523.37 -> I understand that the Adel is gonna try to demo

527.472 -> it this entire new feature.

528.305 -> And we will definitely have questions as we go along.

531.96 -> Adel has a potentially a sporty internet connections.

535.63 -> Thank you hotel rooms, but we'll try to go through it.

537.97 -> So I don't know if you can just go ahead and just click the

540.97 -> Share Your Screen thing here and we should bring it up.

545.67 -> And in the meanwhile, folks, if you have any questions,

547.42 -> put them in chat, we'll try and get to as many as we can.

552.29 -> We love the questions.

554.094 -> Well, maybe Frank can address the question

556.36 -> we have here from Randall.

560.25 -> How does this compare/contrast overlap

562.027 -> with database AWS AppConfig feature flags?

566.25 -> Is this like, is AppConfig thing more a backend focused

569.55 -> while Evidently is evidently more front-end focused?

575 -> I love the make by the way.

575.965 -> All right that's a good question.

580.63 -> So the way to look at it is that yeah AppConfig

584.3 -> is config focused but you can also do feature

587.68 -> flags within AppConfig.

589.79 -> However, if you actually want to do more

593.662 -> around the area of monitoring metrics,

597.05 -> when you're doing launches,

599.23 -> if you're interested in integration

600.83 -> with real user metrics like CloudWatch RUM,

604.1 -> and if you're interested in say experimentation

607.22 -> running AB experiments,

610.01 -> then that's where you can go to Evidently.

612.76 -> So AppConfig is great product

615.76 -> focused on configurations and feature flags.

619.18 -> Evidently then sort of does additional work.

623.295 -> Has additional capability in the area of experimentation

626.81 -> and launch control with metrics monitoring.

631.74 -> Excellent, thank you.

632.75 -> Adel we have you back here.

633.97 -> So I'm gonna bring up your screen on screen right now.

637.49 -> Tell us what you're gonna show us and well, show us.

642.31 -> So today we're gonna go through a quick demo application.

646.56 -> I'm gonna start with an overview of the application

649.18 -> we're working with today

650.24 -> then I'm gonna hop right into the CloudWatch console

652.36 -> and go introduce the Evidently console.

656.27 -> And we're going to instrument a new feature

659.05 -> in about the next, hopefully five to 10 minutes.

662.05 -> So onscreen, I have a bookstore running on AWS,

667.76 -> which is something we're very familiar

669.92 -> and a it's the homepage,

672.09 -> and a homepage is a very important

674.81 -> for any web application.

676.97 -> Setting it up with the right assets,

679.26 -> messaging, and navigation,

681.51 -> help your customers find products,

683.89 -> faster, easier, and keeps them more engaged.

688.01 -> So our goal is to look at this application

690.41 -> and identify the opportunities

693.15 -> to power features with Evidently.

696.38 -> On first look, a few things come to mind.

699.53 -> So you can see, we have highlighted categories here,

703.1 -> which is something you would wanna power

704.95 -> with Evidently because it would allow

708.402 -> you to test and change it during different holidays.

711.4 -> Another thing you might want to power with Evidently,

713.85 -> is your search product search model.

718.305 -> Frank mentioned it earlier that it would be great

721.6 -> to use Evidently to switch between different models,

725.09 -> say that you have a stable product search model,

727.73 -> and you have a couple of candidates,

729.66 -> you can use a combination of SageMaker

731.747 -> and Evidently here to switch between the models

734.16 -> and find out which one performs

736.37 -> better and get those insights into your business metrics

739.68 -> and performance metrics that Evidently can track.

744.02 -> The feature we're gonna be focusing on today

746.58 -> is a promotion banner.

748.14 -> We're going to add a promotion banner

750.31 -> to this homepage at the top,

752.7 -> and a promotion banner is great

754.19 -> because it's something you can measure

755.53 -> the performance of without being in production.

758.57 -> That's the only way you're gonna know how your customers

760.68 -> engage with it.

762.27 -> So we're gonna hop over to the CloudWatch console.

764.75 -> You'll find Evidently on the left side of the CloudWatch

767.88 -> console under application monitoring

770.32 -> right next to the new service

772.947 -> product from which I think you just did a feature on.

776.68 -> And the first thing you'll see are projects and projects

779.41 -> are basically a container for different resources

782.16 -> for the rest of the Evidently resources.

784.19 -> And they're gonna be related to one of your applications.

787.12 -> When you click into a project,

789.3 -> you'll be greeted with a list of three

791.1 -> more types of resources,

792.81 -> features, launches, and experiments.

796.177 -> In our bookstore, you can kind of see

798.271 -> that we already have a few features working.

799.71 -> We have the highlighted categories that I mentioned.

801.7 -> We have the product search model that I mentioned as well,

804.13 -> as well as the recommendations feature.

807.4 -> Before our goal is to add the promo banner feature,

811.55 -> but real quick, I want to start with the end goal.

816.25 -> By visiting an already competed experiment

819.13 -> that we have four highlighted categories.

822.19 -> So our goal is to find out how different promo banners

827.37 -> perform and this is what we're aiming for.

830.3 -> So here you can see...

833.27 -> Give it a moment here.

836.3 -> That we have a experiment

840.11 -> here about highlighted categories

842.32 -> at the start of summer 2021,

844.7 -> and you'll instantly be greeted by a set of graphs

850.18 -> in Evidently you can track multiple metrics

854.33 -> and the types of metrics you can track are not limited.

858.38 -> So here we have one on revenue, one on user activity.

864.02 -> And so what you'll see when you're using Evidently

867.34 -> is you'll get to see that performance

869.2 -> of the different variations broken up over time.

873.27 -> And so here you can see that in this highlighted categories

876.38 -> experiment we were testing two different types of categories

879.38 -> at the top of our bookstore.

882.14 -> You'll see here we were testing a top selling categories

887.06 -> versus a summer activities categories.

888.92 -> So seeing whether your top selling of all time

891.44 -> versus your seasonal will outperform

894.17 -> which one will perform better.

895.78 -> And you can kind of see that very consistently top selling

899.75 -> is doing well, but going beyond average values,

903.82 -> you'll get additional insights here,

905.82 -> including statistical significance,

907.61 -> as well as our confidence here.

911.07 -> So here you'll see a few different insights

921.26 -> that you get in addition to regular CloudWatch metrics,

924.19 -> including improvement, confidence intervals,

926.49 -> as well as technical significance.

928.04 -> Now, if you're not familiar with all these terms,

930.79 -> there's always info tabs on the side

933.01 -> to help you figure out what that means.

934.71 -> And we also, as Frank mentioned earlier,

937.34 -> we'll label each results with just a clear term,

941.2 -> according to very standard practice,

942.82 -> whether it's worse, better performing, et cetera.

945.4 -> So here we can see that summer activities

948.33 -> did not outperform our top selling category.

953.26 -> And that was statistically significant.

955.56 -> And we can see this pretty clearly from this trend here,

958.45 -> where we do see that top selling did trend

961.9 -> towards our summer activities over time,

964.24 -> but it really never summer activities

967.26 -> never really had an impact on revenue.

972.21 -> So that's our goal today.

973.79 -> So we're gonna head back to our project

976.99 -> and we're gonna get started with adding a feature.

979.5 -> And so the feature we're gonna be adding today

981.7 -> is a promotional banner.

984.36 -> And so this promotional banner,

987.21 -> when you create a feature in Evidently

988.78 -> you represent them with a value,

991.6 -> and since we're doing a promotional banner,

994.05 -> we're gonna use images because we can just plug in.

997.53 -> We can just plug in image URLs

999.96 -> and get it to render on our website pretty quickly.

1002.56 -> So I'm gonna add a default variation

1005.08 -> just for having no banner, no promotions going out at all.

1008.82 -> I'm gonna add another one.

1010.16 -> We're gonna do like a 20% off one.

1016.23 -> And we're going to also add another one

1018.48 -> that's like buy one, get one.

1021.16 -> So we're doing different kinds

1023.131 -> of things here we're gonna be using, right?

1024.99 -> Correct, and I'm just gonna grab these URLs very quickly.

1033.07 -> So you're creating these different variations

1035.113 -> and we're gonna test against all three

1037.29 -> of those once we do the experiment, right?

1039.3 -> Correct, so we're gonna,

1040.31 -> we're gonna throw these in the experiment.

1042.31 -> So as you create a feature,

1045 -> you'll get a snippet here on how to set up that feature

1047.93 -> inside your code.

1048.763 -> So evidently does two parts,

1050.72 -> it helps you take the decision logic

1055.38 -> out of your application,

1056.56 -> and we'll help you decide which feature,

1057.812 -> which variation of your feature is active to serve per user,

1062.02 -> as well as doing the recording,

1064.14 -> the performance that Frank mentioned earlier.

1067.02 -> You can see how to instrument it here in JavaScript,

1069.58 -> and you can see a sample results here.

1072.142 -> So you can see that it'll return a string value here

1075.36 -> if I switched the default here,

1076.79 -> you'll see that it'll be returning that specific URL.

1080.71 -> Which languages do you support?

1082.894 -> That's maybe the first question here.

1086.057 -> We support C++, C#, .Net.

1091.58 -> There are lists JavaScript, Java here on the sample codes

1096.01 -> we do have Java and JavaScript are ready for customers

1098.77 -> to copy and paste into their application.

1100.85 -> Awesome.

1102.09 -> So I'm gonna go ahead and create this feature.

1104.743 -> I'll go back to the bookstore.

1106.44 -> You'll see that there's no banner yet.

1110.28 -> Just an FYI,

1111.68 -> I didn't want to show me logging in and out.

1113.39 -> I have a little override here that helps me switch between

1115.185 -> accounts so I can show you that different

1117.68 -> things are happening.

1118.82 -> So I can see here that there is nothing

1123.285 -> there's still no promo banner.

1125.17 -> I'm gonna be act as a developer right now,

1127.79 -> and I'm gonna go and let myself

1130.686 -> see one of the variations.

1132.25 -> So if I go into the feature,

1133.47 -> I can add what's called an override,

1134.92 -> and this is really helpful for your developers

1137.21 -> to put something into production

1138.55 -> and see how it works in production.

1140.51 -> Just to do a kind of quick sandy check themselves.

1144.07 -> So I'm just gonna go ahead and add this feature,

1148.5 -> and then we're gonna hop over to the bookstore.

1150.8 -> We're gonna reload, and you can see that right away,

1153.48 -> I get the new variation.

1155.54 -> I see now that I have a promo banner,

1157.3 -> I can see that there's a buy one, get one.

1159.6 -> I'm just gonna hop back in here,

1161.84 -> do another edit and switch to the other variation

1166.25 -> to see the 20 off the 20% off variation.

1170.01 -> Again, I hop over here, reload.

1173.37 -> I see that I made a mistake here and I put

1177.3 -> a 10% off a variation.

1181.82 -> So I'm just gonna hop back into the feature

1184.34 -> really quickly fix that mistake

1186.08 -> before we start our experiment.

1189.7 -> Copy this one.

1193.29 -> Now all this data feeds back into CloudWatch.

1197.087 -> And we're able to see this information

1199.19 -> on that kind of experiment dashboard.

1201.8 -> Correct, correct.

1202.82 -> So that dashboard that we saw is also that data

1206.02 -> is also available in CloudWatch metrics

1208.1 -> so that you can create alarms.

1209.7 -> And so here,

1215.131 -> we'll see that the 20% off is done correctly.

1217.38 -> And you can also, we can do a quick validation

1219.56 -> here that there aren't any other users

1222.88 -> who can see that promo banner.

1225.36 -> Correct, it's only

1226.193 -> for you because you set up that the identifier.

1227.88 -> So this only applies the specific user in this case.

1231.078 -> Correct.

1232.46 -> Okay. Gotcha.

1234 -> So now that we have a feature up and running,

1236.04 -> we can hop over to create that experiment.

1238.94 -> So our goal is to get an experiment

1241.27 -> running for the promotional banner.

1242.76 -> That's similar to the highlighted categories

1244.77 -> we will create experiment real quick.

1247.41 -> And it's wonderful that you can actually get

1249.96 -> all these metrics and all these experiments.

1251.69 -> 'Cause, well, we live in a data driven world right now.

1255.307 -> That is awesome.

1256.14 -> So let's have a look how you can actually create

1257.61 -> this experiment then we're gonna have to wrap up

1259.21 -> this entire thing.

1260.043 -> So creating the experiment

1261.72 -> right now, okay got it.

1265.677 -> So we get through, here we go through the audience.

1267.91 -> We want to include just these two variations.

1270.17 -> We're gonna split the traffic equally between the two.

1273.21 -> And here is the power of Evidently is that you can define

1276.318 -> any type of metric that you want.

1278.88 -> Let's say that I'm doing revenue,

1281.53 -> when you send events to Evidently,

1283.72 -> you're able to set a few key parameters

1288.22 -> that allow you to filter for any type of metrics.

1292.22 -> So if you see here if you're familiar with EventBridge,

1294.96 -> you'll notice that this is the EventBridge syntax.

1298.27 -> We add the revenue metric here and we go ahead

1301.23 -> and create our experiment.

1311.196 -> It's creating the experiment right now,

1313.8 -> once this is done,

1314.742 -> it's gonna create the exact same dashboard

1316.05 -> as we've seen before on the one that's works.

1318.84 -> Correct, so all we do now is we just start

1321.67 -> this experiment, and so as your customer starts to,

1325.445 -> we set an end time for the experiment.

1328.01 -> And when we, as a customer visits our website,

1331 -> we'll get the same dashboard that we saw in the other.

1334.67 -> Wonderful, and this experiment in a sense,

1336.36 -> it will actually distribute this load between two different

1338.58 -> variations and we can see which one works better.

1340.84 -> So experimentation is actually key

1342.79 -> to making your product better, right?

1345 -> So you need to try things you need to,

1346.786 -> you need to play with things, break things,

1349.1 -> but very importantly, when you experiment,

1351.53 -> you actually need to see what's happening right then.

1353.99 -> And Evidently evidently helps you do that.

1356.89 -> So thank you.

1359.43 -> Thank you, Frank and Adele, thank you very much

1361.3 -> for showing us Evidently,

1363.09 -> which evidently has the best name.

1366.8 -> I'm sorry.

1367.71 -> I bet that's not the last time

1369.023 -> you're gonna make that joke.

1370.51 -> Yeah I know.

1371.74 -> Evidently is evidently the best name here

1373.59 -> So thank you to whoever made up, made up that name.

1377.14 -> So wonderful.

1379.322 -> So I think, I think this is going to be a wonderful launch,

1383 -> especially for developers and folks

1385.11 -> trying to make their things so much better.

1387.18 -> So excellent.

1388.55 -> With that being said,

1390.95 -> we are gonna jump to a very loud music segment right now,

1394.57 -> I guess, which we're gonna,

1398.03 -> Sohan Maheshwar play something.

1399.58 -> So Frank, Adel thank you very much.

1401.554 -> Thank you.

1402.68 -> Thank you.

1403.513 -> (upbeat music)

Source: https://www.youtube.com/watch?v=ydX7lRNKAOo