AWS re:Invent 2022 - Better decisions with no-code ML using SageMaker Canvas, feat. Samsung (AIM207)

Aug 16, 2023

AWS re:Invent 2022 - Better decisions with no-code ML using SageMaker Canvas, feat. Samsung (AIM207)

Organizations everywhere use ML to accurately predict outcomes and make faster business decisions. However, this often requires preparing, building, training, and deploying ML models. Amazon SageMaker Canvas expands access to ML by giving business analysts a visual point-and-click interface to generate accurate ML predictions on their own without requiring any ML experience or writing a single line of code. In this session, learn how you can use SageMaker Canvas to access and combine data from a variety of sources, clean data, build ML models to generate predictions with a single click, and share models across your organization to improve productivity.

Learn more about AWS re:Invent at https://go.aws/3ikK4dD.

Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4

ABOUT AWS
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#reInvent2022 #AWSreInvent2022 #AWSEvents

Content

0.18 -> - Hello, can you hear me?

2.01 -> Yes?

3.09 -> Okay, great.

4.35 -> Welcome.

5.43 -> Wonderful have you all with us this afternoon.

8.61 -> Thank you for joining us

9.63 -> and hope you're having a wonderful week

11.49 -> at AWS re:Invent.

13.2 -> Our favorite week of the year.

16.53 -> Welcome to Making Better Decisions with No-code ML.

20.52 -> Before I get started,

22.17 -> Danny Smith, my colleague asked, you know,

24.27 -> some of you with a show of hands

25.53 -> who is a data scientist, a line of business users,

28.171 -> or somebody who's with a data background.

31.14 -> And it was a mix.

32.55 -> So it's great

33.39 -> because that's what we want.

34.56 -> We want to talk to you

36.27 -> about how you can make better business decisions using

40.5 -> what we call as No-code ML.

42.51 -> And today we are gonna dive deep into that.

44.61 -> My name is Shyam Srinivasan.

46.44 -> I'm a product manager.

48.48 -> We're handling Amazon SageMaker Canvas.

51.18 -> The No-code ML service which AWS offers.

54.39 -> I'm gonna be joined shortly by my esteemed co-speakers.

57.39 -> Danny Smith and our honored speaker

61.17 -> from all the way from Korea, Derrick Lee,

63.81 -> who's gonna talk us through the Samsung story.

66.93 -> So let's get started.

72.09 -> So if you listen to Swami in his keynote this morning

76.47 -> at the data and ML keynote.

78.63 -> One of the aspects he talked about data and ML

80.91 -> was democratization of machine learning.

84.93 -> Now what this session is all about

87.15 -> is gonna dive deep onto what that actually means.

91.02 -> What do we mean by ML democratization?

94.08 -> How do we create value from machine learning?

97.92 -> We're gonna follow that up with a demonstration.

102.03 -> A picture or a demo is worth a thousand words.

104.76 -> So no slide can do justification

106.98 -> unless you see the actual product.

109.32 -> So Danny is going to give us an end-to-end demo

112.14 -> of what Canvas is and what we mean

114.84 -> by end-to-end ML value creation.

118.65 -> We'll follow that up with Derek telling us

122.01 -> about the Samsung story.

123.24 -> It's a fascinating story and you're gonna love it.

126.6 -> And then we'll wrap it up with some Q & A.

130.74 -> So let's take a step back.

133.74 -> Not too long ago, about 10 to 12 years.

137.31 -> 2010.

138.96 -> Machine learning was getting out of research labs.

143.94 -> AI and ML have been around for decades.

147.6 -> In fact, Amazon has been using AI and ML

151.71 -> much before AWS is born and we continue to do so.

156.03 -> So around that time in 2010 machine learning

159.48 -> was gaining foothold in the business,

162.48 -> in an enterprise, and it looked as if it

165.42 -> was holding promise to solve business problems.

169.59 -> Technologies were built.

171.412 -> AWS was at the forefront of this journey.

176.46 -> For those of you who've seen us through the years

179.16 -> we have launched a number of solutions services focusing

183.51 -> on machine learning.

185.13 -> Making us the leader today.

187.98 -> We wanted to ease that ML adoption

190.53 -> into the business and not really restrict

193.14 -> into research labs.

195.78 -> But of course the folks who are using ML

199.05 -> were still the ML practitioners

201.06 -> and some of you in the audience today.

203.67 -> Be it a data scientist and ML engineer or a data engineer.

209.34 -> But going back to ML democratization.

212.79 -> It makes sense only when we take ML mainstream.

218.07 -> When a business analyst, a business user,

220.02 -> who has no background in coding or in technology,

225.78 -> if she or he can use machine learning

229.53 -> then we call it a success.

233.85 -> But the business analyst needs to be unblocked.

237.87 -> A typical business analyst loves data.

240.51 -> They know their data but they will not

243 -> have the ML expertise.

245.58 -> And up scaling is difficult.

247.14 -> Overnight you can't become a data scientist as we know.

251.01 -> The second part is about transparency.

253.32 -> The business needs transparency.

255.39 -> You need validation from the experts

257.64 -> but it has to be transparent and collaborative.

261.9 -> So No-code is the future.

264.39 -> In fact, Gartner in a report last year

268.32 -> predicted that just in about two years 80%

272.61 -> of technology, of services and solutions,

275.07 -> are gonna be built by non-tech professionals.

280.77 -> And the key part is about seamless collaboration

284.34 -> which I talked about earlier.

286.08 -> You need to have the business teams

287.97 -> and the data teams get out of their silos

291.39 -> and seamlessly collaborate

292.98 -> without artificially being forced

296.25 -> to learn machine learning skills overnight.

300.03 -> So welcome to Amazon SageMaker Canvas.

304.02 -> We launched Amazon SageMaker Canvas at this very stage.

307.44 -> Last year at re:Invent 2021.

309.9 -> So it's a one year old baby.

312 -> Canvas is a No-code visual machine learning tool

317.34 -> which is actually a workspace.

319.47 -> Canvas gives you the entire journey of taking your data

324.24 -> and generating predictions to solve business problems

326.94 -> in a single workspace.

328.56 -> Can import data from different sources.

330.96 -> Understand and explore the data.

332.85 -> Make the data work for you.

335.4 -> And then with a click of a button.

337.5 -> Ask Canvas to build a model for you.

340.17 -> The complete data science nuances

343.23 -> are hidden from you, are abstracted from you,

345.12 -> if you're a business analyst.

348.24 -> Canvas uses auto mail on the,

350.01 -> under the hood, to build a model,

352.86 -> and to give you the most accurate prediction.

356.31 -> For those of you are interested you can still go deep

358.92 -> into knowing what those model metrics are.

361.98 -> For some of you are interested in things like F1 scores

364.8 -> or you just wanna know the feature importance

366.6 -> of a particular feature in your data

369.3 -> that is impacting the model metric.

372.87 -> And then you can share the same model

375 -> with your data science teams for validation or for review.

378.99 -> And last but not the least, like any other AWS service,

382.74 -> it's a pay as you go model.

384.3 -> You pay only when you use Canvas.

386.64 -> And no license fees.

389.85 -> We've been busy.

390.72 -> This is a crowded slide and intentionally so.

394.05 -> It's been one year.

395.79 -> We have been busy launching different capabilities

399.39 -> in Canvas.

401.01 -> Thanks to feedback from customers like you.

403.44 -> Customers have kept us honest.

405.3 -> Have told us what they would like to see.

407.4 -> So be it in areas like exploratory data analysis.

411.21 -> Be it in easy onboarding.

412.8 -> Or even for the administrator

414.48 -> to enable easy provisioning for the Canvas user.

418.65 -> And we have just begun.

419.91 -> It's an exciting road ahead for us.

423.9 -> So you may ask, "This all good

426.06 -> but where can I use Canvas?

427.86 -> What is the use case"?

429.45 -> Well what I have here on the slide

431.28 -> is just the tip of the iceberg.

432.78 -> I've just scratched the surface.

435.39 -> I talked to some of you in the audience before we started

437.76 -> that you are coming from a finance background.

439.95 -> So if you want to talk about credit risk scoring.

443.91 -> If you want talk about fraud detection.

446.88 -> On the sales side you wanna talk about propensity

448.86 -> to buy or you want to avoid showing your customers

452.94 -> and keep them loyal.

454.8 -> The operations and logistics.

456.45 -> You'll see about, you know,

457.65 -> if you want to predict delivery times.

460.5 -> You wanna talk about how to keep your machines up to date.

464.58 -> Canvas can make that happen for you.

468.06 -> So as I said, you have data

470.13 -> which is a foundation for machine learning.

472.62 -> Ask questions.

473.97 -> Make the data work for you.

476.25 -> So Canvas is as simple as you select a target column.

479.37 -> If you have a table of data.

480.96 -> Rows and columns.

481.793 -> Select a column.

482.626 -> And see what the impact of the target column would be.

486.03 -> How is it explained by the other features?

490.41 -> So it's as simple as bringing in a spreadsheet

494.28 -> and creating a model and generating predictions.

497.28 -> But don't take my word.

499.05 -> Let's hear from Danny as you see the demo.

501.72 -> Welcome Danny.

503.515 -> (crowd applauds)

507.27 -> - Thank you Shyam.

508.89 -> Thank you everybody.

517.14 -> All right, I want one slide first

519.27 -> because I wanna explain the use case.

521.19 -> So this use case is a manufacturing use case.

524.97 -> So imagine if I was a manufacturing quality engineer

530.52 -> and what I'm trying to do is understand

533.28 -> how the end of the line quality tests.

536.25 -> So when people make things

538.11 -> they test them before they send them to you.

541.26 -> So they make sure they work.

543.15 -> And this manufacturing line has a lot of sensors

546.24 -> and other information on it.

548.25 -> And so the use case is

550.26 -> what we wanna do is we wanna predict end-of-line quality

554.55 -> or at least explain it by those other variables

558.42 -> that we have, the other sensor readings,

561.18 -> so that we can understand how to improve

564.09 -> but we can also spot problems ahead of time.

567.06 -> So that's the use case.

572.07 -> And this is Canvas.

573.36 -> So the first thing I would say is if you go into Canvas

575.94 -> I would highly recommend you press the help button

580.29 -> because the help button

581.16 -> has a built in guided experience for you.

585.21 -> It'll teach you what you need to do.

587.037 -> And so the demo that I'm gonna run today

589.92 -> is actually built into the tool.

591.57 -> And so just as an example.

594.36 -> Just press it and it'll say this is how you import data.

598.59 -> Go to the data sets page.

600.48 -> And then click on import.

603.39 -> Pick the data you have.

605.82 -> You get the idea.

607.02 -> So I'll, so never, never forget that there's help

610.59 -> inside the application.

612.72 -> So let's go back to our use case.

614.67 -> So the first thing we have to do is get data in.

618.69 -> So it's actually pretty straightforward.

623.43 -> What I did

625.2 -> was I just uploaded data.

627.36 -> Right so here's the data I wanted.

629.88 -> I pulled it in.

631.841 -> We can take a peek at it.

633.997 -> Is this the data that I think we want?

636.45 -> Yes it is.

637.283 -> We've got some test information.

639.9 -> We've got some center readings.

642.12 -> We've got this X and Y offset

644.55 -> which is a position from where it's,

647.58 -> from where a component is placed on the,

650.79 -> on another component.

651.9 -> So X and Y offset is how far off from the ideal it is.

655.5 -> And then of course we've got this end-of-line test.

659.04 -> Right so we have some fails and we scroll down.

662.4 -> Here's some passes.

663.66 -> So that's the data that I pulled in.

666.69 -> Now I could have also pulled it in from S3

670.08 -> which is Amazon storage locations.

674.04 -> Or if I had access to a database

676.77 -> I could pull in information.

679.29 -> So as an example.

685.44 -> I'll tell you what, like.

686.273 -> Let me make this a little bit bigger

687.84 -> so that you can see it.

690.823 -> Here we go.

693.87 -> So if I had a few tables and wanted

697.17 -> to just kind of pull them in together,

699.57 -> Canvas would automatically figure out

701.13 -> how they're related to each other and join 'em.

703.65 -> If I wanted to I could look at SQL

705.75 -> but I certainly don't wanna look at SQL.

710.82 -> So you can bring data in.

713.61 -> All right, let's get back to the story.

715.71 -> So then I started building some models to analyze this data.

721.56 -> So let me catch you up

723.15 -> 'cause I've already started taking a look.

725.61 -> So first thing I did

729.06 -> was I selected the data.

731.07 -> So here's the data that I showed you just a second ago.

734.37 -> Sorry.

736.35 -> And then I came in here and I said end-of-line the test.

740.55 -> EOL test is the target column I want to predict.

743.85 -> And as soon as I did that Canvas started doing

746.31 -> a bunch of things.

747.45 -> So Canvas looks at the data.

750.87 -> And it's like, okay.

751.703 -> What kind of data type it is?

752.85 -> Binary.

753.87 -> Does that have missing or mismatched values?

756.36 -> What are the unique, you know.

758.31 -> How many unique values are there?

759.78 -> What are some average values?

762.27 -> It also, when you put in end-of-line test,

764.46 -> it also

766.47 -> analyzes that data

767.97 -> and it says, "Oh I see you have pass and fail."

771.9 -> It's automatically gonna recommend

774.9 -> a two category prediction.

777.09 -> So there's lots of different things Canvas can do.

779.19 -> It can do what a data scientist

781.11 -> would call a binary classifier.

783.24 -> So that's a prediction for two categories.

786.09 -> There's also multi class classifier.

788.37 -> Or a three or more multi, you know, multiple prediction,

791.79 -> multiple value predictions.

793.89 -> If the target column is numeric

796.62 -> it'll recommend regressions and other numeric models.

800.79 -> Even time series.

802.77 -> So for time series forecasting.

804.72 -> So in this case I have a classifier.

807.3 -> I wanna see if I can predict pass or fail.

810.3 -> So it suggested that's the model type I want to use.

814.44 -> So we can also do other things to get our data ready.

819.9 -> I always like to look at our data.

822.81 -> So here's kind of some visualizations of the data itself.

825.69 -> So you see that some of these things,

827.19 -> this test one is actually a binary.

830.43 -> So is the gate one.

834.18 -> If there are things that need to be fixed.

836.88 -> So first of all you notice

838.14 -> that it says validate your data here.

840.15 -> Canvas is already looking at the data, sorry.

845.19 -> Running through validations.

846.54 -> I'm just having it, you know, go and check

848.97 -> and make sure there's nothing in there.

851.25 -> But let's say there was.

852.24 -> So we could say if we had missing values.

855.45 -> Missing values sometimes are hard to deal with.

857.7 -> We could remove rows that have missing values as an example.

860.79 -> So reading six.

863.73 -> If there are any missing we could just remove values.

867.3 -> Same with replacing.

869.73 -> You know, if we wanted to say okay reading six.

875.73 -> If it has outliers,

877.05 -> in other words really strange random occurrences of data,

881.37 -> we could actually kind of scope that down

883.26 -> and say I want to bring it into the normal range.

886.56 -> In this case a standard deviation.

888.42 -> So if it's

892.68 -> three standard deviations

893.97 -> I want you to just bring it in

896.07 -> to three standard deviations

897.36 -> so it doesn't throw off the data.

899.46 -> So you can do a lot of data preparation.

901.5 -> Data fixing.

902.55 -> One of the most interesting things is functions.

905.16 -> So functions is just like what you're used to

908.52 -> in a spreadsheet.

909.75 -> If I wanted to.

910.68 -> So let's take a look at this reading six.

912.87 -> So reading six seems to have a strange distribution

916.38 -> in the data.

917.85 -> Right but maybe,

921 -> for human understanding,

923.31 -> maybe what we want to do is say

924.84 -> if reading six is greater than.

928.08 -> Hmm.

931.05 -> 130.

933.39 -> Call it high.

935.37 -> And if reading six

938.7 -> is less than.

943.29 -> 85.

946.35 -> Call it low.

948.81 -> Otherwise.

955.62 -> So typing alone, you know,

957.24 -> it's hard to type and talk at the same time.

959.22 -> You get the idea.

964.68 -> Maybe we call that R six pin

966.63 -> and then we can just process the data

969.33 -> and let it take a look.

971.19 -> You know, see if we want to accept that or not.

973.83 -> Right, so Canvas can spin through that.

976.95 -> So.

980.43 -> I've already done all this so we don't need to do it.

983.13 -> But.

985.23 -> We probably as a user

987.27 -> are probably curious about our data, right.

989.37 -> I certainly am.

990.6 -> So right now I'm trying to figure out if any of this data

993.27 -> that you see can actually explain end-of-line tests.

997.737 -> And so maybe before I do any modeling

1000.32 -> I just want to take a look at the data.

1001.94 -> And so probably the first thing I would look at

1003.89 -> is a correlation matrix.

1005.39 -> So correlation matrix shows

1007.28 -> how data is related to each other.

1009.65 -> Right and so here's my end-of-line test.

1012.59 -> And so end-of-line test is related a hundred percent

1015.14 -> with end-of-line test.

1017.57 -> That's logical.

1019.04 -> But it looks like this X offset

1021.74 -> has a high relationship.

1023.72 -> So this might be useful.

1025.52 -> That value may be useful to predict.

1028.79 -> Now there's some other things in here too.

1030.2 -> So like reading one

1032.15 -> and reading five

1034.76 -> are a hundred percent correlated with each other.

1036.98 -> Probably don't need both of those in the model.

1040.58 -> But if we wanted to take a look at that,

1042.59 -> right so reading one, reading five.

1045.08 -> I can come in I can take a look at a scatter plot.

1052.13 -> Maybe run it against the entire data set

1057.47 -> versus just a sample.

1059.33 -> Okay.

1060.71 -> That's exactly the same information.

1063.26 -> Right, you can do other things right

1065.09 -> with these visualizations.

1066.74 -> And so I remember this X offset.

1069.47 -> So how does X offset relate to end-of-line tests?

1072.92 -> So this is a box plot.

1074.12 -> So a box plot's really useful.

1076.25 -> That line in the middle kind of shows the average.

1079.13 -> The box shows the 25th quartile, 25th percentile,

1082.79 -> the 75th percentile.

1084.11 -> So the, like, the middle quartile ranges.

1087.92 -> And then those whiskers show the long tail

1090.89 -> of outliers beyond those.

1093.41 -> And so if you think about this

1094.85 -> as does the data that has, that's passed,

1100.25 -> does it behave differently than the data,

1103.173 -> then the items that failed?

1104.93 -> And so I would say yes.

1107.57 -> So this is probably gonna be a pretty good variable.

1111.56 -> But now I'm impatient.

1113.18 -> Time is ticking.

1114.2 -> It's only a short demo.

1115.73 -> So what I want to do is start building some models.

1118.55 -> So the first thing I would suggest

1123.17 -> to train some data is to just press preview.

1125.18 -> Preview is really quick.

1127.28 -> Right, it's just a single pass

1129.41 -> but we don't charge you for it.

1132.59 -> So it's useful

1134.57 -> to kind of see if we want to submit all the data

1137.54 -> or if we wanna scope down the data.

1139.52 -> So here's an example I ran just a second ago.

1142.31 -> So we could, this model, it ran a quick one pass.

1146.45 -> 93% accuracy.

1147.83 -> It looks like we can explain end-of-line test really well.

1151.67 -> Here are the columns that make the most sense.

1154.94 -> So you see down here.

1157.91 -> Gate two, gate one test.

1160.94 -> These aren't really explaining much

1163.04 -> or contributing to the prediction.

1165.74 -> So maybe I just get rid of those.

1168.62 -> Now when I'm getting rid of these,

1171.23 -> couple of things are going on.

1172.97 -> The total cell count is going down.

1175.7 -> So when we actually want to train a model.

1179.33 -> The pricing's based on the cell count.

1181.52 -> Cell count is like rows times columns.

1184.1 -> Everybody knows what a, you know, spreadsheet is.

1187.19 -> So this is a great way of reducing your cost

1190.16 -> by just kind of narrowing down the scope

1193.25 -> without giving up much.

1194.57 -> So the question is...

1195.403 -> And so we keep track of that.

1196.58 -> So we're dropping test gate one, gate two, reading one.

1200.3 -> We keep track of that with the recipe

1201.83 -> but does it help us any?

1203.51 -> Or hurt us?

1204.98 -> So we can just run it again.

1207.8 -> See what happens.

1211.64 -> It's always seems longer

1212.63 -> when I'm standing up here in front of you.

1216.29 -> And we can see what the result is.

1218.45 -> Well okay so here's the result.

1220.04 -> So accuracy is about the same.

1224.27 -> So we can get rid of these columns

1226.1 -> without really impacting accuracy.

1228.86 -> So now I'm probably ready to kind of continue

1231.41 -> and actually build a machine learning model

1235.22 -> that takes advantage of the auto ML.

1237.77 -> So the auto ML does some things.

1239.54 -> So let me just kind of show you that for a second.

1241.4 -> So there's two types of auto ML model building.

1245.93 -> One of them is called Standard Build.

1248.39 -> Standard build is the best of auto ML.

1252.77 -> It does a bunch of data pre-processing,

1254.72 -> it cleans up missing values, it standardizes the data,

1258.92 -> it does all this pre-processing,

1260.3 -> then it looks at algorithms,

1262.1 -> and it's like okay let me look at these,

1264.08 -> all these different algorithms.

1265.46 -> And then it does what's called hyperparameter tuning.

1268.07 -> If you don't know what hyperparameter tuning is

1270.02 -> you remember on old radios

1272.24 -> where you had to turn the knobs

1273.68 -> and you could pick up the radio signal.

1276.44 -> It's only for older people.

1278.93 -> Hyperparameter tuning is just tuning

1280.97 -> all those input parameters to see if

1282.68 -> there's a better reception, a better model.

1286.01 -> So it'll do all those.

1287.03 -> It'll do a hundred iterations of that

1288.62 -> and then it'll show you the best.

1291.56 -> That's two hours.

1293.45 -> I don't know if I'm ready for the two hours.

1295.4 -> So I'm gonna go for a quick model

1297.86 -> which is like two to 15 minutes.

1299.87 -> So it does a subset of that best practice.

1304.04 -> But it's faster.

1306.17 -> So I'm and

1308.42 -> just so you guys know I'm not gonna make you sit here

1310.61 -> and wait for 15 minutes.

1312.26 -> I'm actually

1315.2 -> gonna show you one I already ran

1319.46 -> in a different version.

1321.17 -> It's the magic of cooking demos.

1323.72 -> So let's pick up the stories.

1326.63 -> So here we are.

1327.463 -> I pressed this button.

1328.296 -> I press the quick build button.

1329.84 -> Now let's see what kind of results we get.

1333.8 -> So the first thing I want to do is.

1336.32 -> Should I trust this model?

1338.96 -> That 94% sounds really good.

1341.45 -> But is that trustworthy?

1343.34 -> So best practice machine learning.

1345.38 -> What it does behind the scenes is it takes the data

1348.77 -> and it takes like 80% of the data and it builds a model.

1352.49 -> That's the training data set.

1354.47 -> And the other 20% is the test data set.

1358.85 -> After the model's built we take that test data set,

1361.1 -> we feed it back in, and see if the model

1364.73 -> maybe learned generally.

1366.98 -> On data, you know, so that it's good on data

1369.74 -> it hasn't seen yet.

1371.03 -> Or maybe it doesn't do a good job.

1373.04 -> And so this is what this shows.

1374.9 -> So of the 1,300 rows in this model we held out 263.

1381.65 -> Of those, and I'm interested in a fail,

1384.83 -> of those 102 were actual failures.

1388.88 -> And we predicted, this model predicted 101.

1392.45 -> However, there was a few false positives, right.

1396.29 -> We got it wrong a few ways.

1398.51 -> So this is what this shows.

1400.52 -> Now if you have a little citizen data science in you

1404.21 -> you can kind of take a look at the details

1406.55 -> and you can say, "Okay, for the fail.

1409.31 -> If we want to predict fail, right.

1411.83 -> We have an accuracy in 94%,

1415.43 -> but we have some false positives, some false negatives."

1419.18 -> That balance is represented by this F1 score.

1423.47 -> So this is a pretty good model.

1426.02 -> I would feel like I could trust this model.

1430.31 -> So if I can trust it let's do something with it.

1433.28 -> So we could go straight to prediction

1435.05 -> and run some new data through and get some predictions

1437.03 -> but I'm not gonna do that yet.

1438.29 -> I'm gonna tell you about that in a minute.

1439.91 -> What I want to do is see if there's more information

1442.97 -> that I can learn just on the behavior of the model.

1446.75 -> And this one's really interesting.

1448.04 -> So if we look at this variable.

1450.98 -> X offset.

1452.24 -> So the physical reality of this model is

1454.73 -> we're taking a component

1456.53 -> and we're putting it down on another piece of equipment.

1460.82 -> And if it's perfect, X offset and Y offset are zero zero.

1467.33 -> 'Cause this is offset from the ideal.

1470.63 -> Now nothing's perfect in life especially

1472.46 -> in my manufacturing process.

1474.32 -> So we're gonna,

1475.153 -> so this is what this, you know, X offset means.

1478.4 -> We put it down.

1479.57 -> So what's really interesting to me

1481.61 -> as a quality engineer is this relationship.

1485.48 -> So here's impact of prediction

1489.29 -> for failure and then we show the value of X offset.

1494.36 -> So as you can see if it gets above about 10

1498.59 -> and by the time it hits maybe 18 or 19

1503.09 -> it's really causing problems with my quality result.

1507.59 -> Now I don't even have to do a prediction.

1510.92 -> I can pick up the phone and call the line

1514.04 -> and say, "Hey, you need to make really sure

1517.13 -> that this thing's adjusted."

1518.93 -> And I can just take action.

1520.115 -> Or, you know, they'll go look at it

1521.9 -> and we fix the problem.

1524.36 -> But let's run some new data through instead

1526.79 -> because this is pretty useful for predicting the future.

1529.76 -> So if I go to...

1531.8 -> I can do predictions two different ways.

1533.81 -> One is a batch and so here's

1538.64 -> new data that, you know,

1539.9 -> new manufacturing that's come off the line.

1542.15 -> What we don't have is the test

1544.22 -> 'cause we wanna predict it.

1546.65 -> So if I select that.

1549.41 -> And I'm just gonna run it through

1551.33 -> and, you know, it's 35

1553.43 -> so it'll run in a few seconds here.

1557.15 -> And then we can take a look.

1559.13 -> What we get

1560.63 -> is not only the prediction,

1561.92 -> so this one's probably gonna fail, but a probability.

1565.55 -> So this, the model is saying

1567.32 -> this thing is 99% probability

1570.08 -> that this thing's gonna fail.

1572.15 -> So I would be very interested in this

1575.96 -> if I'm the production line manager.

1579.02 -> Okay.

1581.06 -> There's another way we can do predictions.

1586.28 -> And our customers call this what if.

1589.1 -> And so let's say that we have certain values

1591.92 -> and we want to kind of see what a change in that value is.

1594.83 -> So you remember I said if X offset.

1597.41 -> You know you saw the curve, right?

1599.09 -> So if we get up to 17 what do you think's gonna happen?

1604.447 -> - [Audience Member] Fail.

1606.89 -> - It's a demo of course.

1609.17 -> So yes we did get a fail.

1611 -> What's interesting though

1611.96 -> is we also get the probability.

1615.44 -> Right so if it's at 17, 87% chance of fail.

1622.43 -> With everything else being the same

1624.14 -> if it's at 16 it's only a 40% chance of it failing.

1630.26 -> So you can get a sense of where the sensitive areas are.

1634.58 -> And so this is why our customers love it.

1637.55 -> So I'm almost done.

1639.26 -> I've got one more thing to show you.

1641.75 -> Let's say that I wanted to actually

1643.55 -> have my ML engineering group take a look at this.

1648.26 -> I want data scientists to review this

1650.06 -> to make sure that they're comfortable with the auto ML

1653.18 -> and then more importantly, I want them to take it

1655.61 -> and productionalize it so that I can embed this

1658.52 -> in an automated system on the line.

1660.77 -> So how do I do that?

1662 -> Well.

1664.61 -> The first thing we do

1667.16 -> is we run that other kind of modeling that I told you.

1671.99 -> Standard built.

1673.16 -> I don't want to ask a data scientist

1674.72 -> to look at, you know, the quick model.

1676.61 -> I want 'em to look at the best practice model.

1679.22 -> So I ran.

1680.51 -> I spent two hours running it.

1685.46 -> I got a more nuanced result.

1686.96 -> I actually got a better result as well.

1690.62 -> And now I can share it.

1693.08 -> So I share this.

1695.9 -> And now the data scientist can kind of come in.

1699.05 -> And if I send that link to 'em

1700.37 -> they can see it in SageMaker Studio.

1703.13 -> SageMaker Studio is the interface

1705.41 -> that a data scientist uses

1707.12 -> to take advantage of the SageMaker managed service.

1710.75 -> And so here is the Canvas model

1713.93 -> from the data scientist viewpoint.

1716.15 -> So in this case, if there were manual feature engineering

1720.17 -> it would show here.

1721.19 -> I didn't actually do anything in this model.

1722.99 -> I just submitted it all.

1724.85 -> You know, let them sort it out.

1726.74 -> But behind the scenes I told you

1729.05 -> it does a lot of pre-processing and things like this.

1731.75 -> So you can kind of,

1732.86 -> the data scientist can kind of come in

1734.54 -> and scroll through and look at all the analysis

1736.46 -> we did on the data.

1737.807 -> And so here's an example of where, like, reading one

1740.27 -> and reading five are the same value.

1742.46 -> Since I didn't get rid of one of 'em

1744.86 -> we'd just show the data scientists.

1746.36 -> They'd be aware of this.

1748.4 -> What else can we look at?

1750.08 -> Well we can look at all of those hundred iterations.

1753.8 -> Data scientists calls those trials in an experiment.

1756.86 -> So we can see them all.

1758.78 -> We can even go into a notebook

1760.4 -> and see how they were generated.

1763.1 -> So I can scroll down.

1765.83 -> A bit.

1766.663 -> So here's an example.

1767.496 -> This is one that was looking at an XG boost algorithm.

1770.57 -> This is a tree based algorithm.

1772.61 -> And so what do we do?

1773.57 -> We converted some numeric features using an imputer

1777.317 -> and we converted categorical features using an encoder

1781.37 -> and looks like we use some Robust PCA

1784.22 -> followed by some standard scaling.

1787.22 -> And here's the code.

1789.71 -> If you're not a data scientist

1790.94 -> this is not maybe the most compelling thing in the world,

1793.43 -> but if you are, it's really interesting.

1796.25 -> Right, so it's full transparency.

1797.9 -> Full visibility.

1800.03 -> We can also look at the best model.

1802.7 -> And the best model.

1803.69 -> We can look at explain ability

1806.48 -> at the depth that a data scientist wants to.

1809.12 -> So data scientists.

1810.2 -> You know all those little hyper parameters I was talking

1812.51 -> to you about where they tuned them?

1814.13 -> So you can kind of come down and see those.

1815.9 -> So here's

1817.73 -> an example.

1818.78 -> Like here's our learning, right.

1820.04 -> Hyperparameter.

1821.36 -> Right, this was the tweet,

1822.38 -> this is the tuning that gave us the best result.

1824.57 -> They can look at all this stuff.

1828.35 -> They can look at performance the way they want to.

1830.96 -> And I think rather than scroll through this

1833.15 -> I think this is a really cool feature.

1834.71 -> It's also self-documenting.

1836.78 -> So we can document everything the auto ML did and learned.

1843.29 -> So data scientists can, you know, have that.

1846.38 -> And then finally the last thing I wanna show you is

1848.783 -> if a data scientist wants to take any of these artifacts

1852.68 -> and productionalize them

1854.36 -> using the standard SageMaker functionality,

1857.12 -> they have access to the artifacts themselves.

1859.31 -> So the model artifacts not just the model

1861.8 -> but so input data where we split the training and test data

1866.24 -> where we then ran it through a transformation logic

1868.82 -> that we did.

1870.14 -> If we wanted to see the feature engineering code itself

1872.84 -> we could take a look at that.

1876.95 -> People always ask me this

1878.21 -> so I'm just cutting off some questions

1880.07 -> during the Q & A section.

1883.25 -> So they're like, "What's the feature engineering code"?

1887.15 -> Well there it is.

1887.983 -> Right, so you've got full access

1890.96 -> to everything you need as a data scientist.

1894.98 -> All right, so let's, let's go back.

1898.73 -> So what I did was I selected data.

1904.25 -> I told that I wanted to predict

1905.84 -> or explain end-of-line test results.

1909.23 -> Canvas suggested a model type for me.

1913.01 -> I explored some data.

1916.16 -> I ran a preview model.

1918.08 -> Maybe scoped the data down.

1920.09 -> I submitted it for formal training.

1923.03 -> I was able to see that I could trust the data.

1927.23 -> I could get some interesting insights from the data itself.

1931.55 -> Maybe go act immediately.

1934.04 -> I could run predictions.

1937.13 -> And I could share.

1941.09 -> Interestingly, just like all the things in the help

1944.87 -> that I told you about.

1946.43 -> So again, if you don't forget anything else.

1950.12 -> Go into the help and help will guide you.

1952.85 -> But hopefully you got to see

1954.26 -> how I could make better business decisions

1957.71 -> by analyzing my data with SageMaker Canvas.

1960.41 -> And then one last thing.

1961.73 -> Since I'm done with it I'm gonna log out

1964.52 -> so that I stop the session charges.

1967.16 -> All right, so thank you for that part.

1970.49 -> My part is done.

1971.51 -> That I have a really exciting job I have to.

1974.9 -> I get the privilege and pleasure of introducing Derrick Lee

1979.85 -> from South Korea.

1981.86 -> He works for Samsung.

1983.33 -> And he's gonna tell you how his group uses Canvas.

1987.74 -> So Derrick please.

1989.36 -> Come on up.

1990.547 -> (crowd applauds)

1993 -> - Thank you very much.

1997.19 -> Thank you Danny.

1998.24 -> Yeah.

1999.23 -> Thank you very much.

2002.47 -> Yeah, hello everyone.

2004.3 -> It is a great pleasure to meet you guys here in Las Vegas.

2007.69 -> Today I will present

2009.22 -> how Samsung Electronics is collaborating with AWS

2013.278 -> with a topic Journey to AI Machine Learning

2015.91 -> as business user.

2019.09 -> Briefly introducing myself.

2020.89 -> I have been working at Samsung Electronics

2023.23 -> for over 10 years.

2025.15 -> It's been a while.

2026.59 -> Right.

2027.55 -> And the reason why I'm showing you my wedding picture

2030.34 -> here though.

2031.173 -> I'm not dressed like it today

2032.5 -> because I wanted to show

2033.73 -> how I am very excited about today's session

2037.21 -> with a very high content model like my wedding day.

2040.48 -> So here's the agenda that I'm going to present today.

2044.11 -> I will introduce quickly my team first

2047.44 -> and I would like to share the concerns we had

2050.98 -> and why we adopted AI machine learning services.

2054.25 -> SageMaker Canvas.

2055.78 -> And I would like to share journey

2056.95 -> with AWS team and next plan.

2061.24 -> So before diving deep into our journey with AWS

2064.67 -> I would like to introduce my team first.

2067.66 -> The market intelligence group

2069.97 -> of Samsung Electronics Memory Marketing Division.

2074.56 -> Our team focuses on predicting the memory market demand

2079.27 -> based on domain knowledge.

2083.2 -> Based on domain knowledge.

2084.82 -> But you know when you hear a word about Sage Forecast,

2089.26 -> Demand Forecast, you might think of it

2091.45 -> as a sales forecasting.

2092.71 -> But the different thing is

2094.6 -> our team predicts the overall memory market demand.

2098.56 -> In particular,

2100.09 -> our team is a business user who has no IT background,

2104.56 -> no experience of writing code,

2106.57 -> no experience of AI machine learning services before.

2111.58 -> So let's get started.

2113.41 -> What is good demand forecasting?

2116.26 -> I just want to start with this question first.

2119.08 -> And our team has deeply deliberated

2121.75 -> about this question.

2124.51 -> First.

2126.19 -> Yeah, the convenient way to demand forecast

2129.4 -> is based on your supervisor requirement.

2133.06 -> You know, this kind of demand forecast

2135.22 -> is usually reflect supervisors experience

2138.4 -> or influence such as making newer forecasts.

2142.12 -> If your supervisor say, "Oh it seems to be a little high."

2145.66 -> Yeah this is a really convenient way

2147.4 -> but it's not a good demand forecast.

2152.35 -> Originally I thought that the demand forecasting

2155.83 -> with the low ad rate based on current data

2159.85 -> and domain knowledge was the dissent demand forecast.

2164.05 -> However, I soon realized that

2166.45 -> this is not the best demand forecast

2168.37 -> as if we didn't know the forecast

2171.49 -> would still stay valid in the future.

2174.76 -> Then what is the best demand forecast?

2179.95 -> Yes, the best demand forecast

2182.2 -> is the demand forecast makes the future.

2185.44 -> Wouldn't it be the one

2188.02 -> in which we can make the demand forecast

2190.75 -> in the direction we want?

2193.24 -> Wouldn't it be the one we can trust

2195.07 -> and implement and executing and materializing one by one?

2199.51 -> That's really amazing, right?

2202.72 -> Then how would I score

2204.7 -> the past demand forecast Samsung Electronics memory made?

2210.1 -> Yes.

2210.933 -> Our general to give 100 points.

2213.49 -> So please give me and my team round of applause.

2217.286 -> (crowd applauds) Thank you so much.

2218.86 -> Yeah.

2220.63 -> Yes, yes, yes.

2222.04 -> Our seniors have done well in the past.

2225.22 -> And I think the memory division

2226.75 -> has been in the top for more than 30 years

2230.95 -> and still doing well.

2232.93 -> But is it okay to stick

2236.2 -> to the existing way?

2242.47 -> Though it may have been okay in the past

2244.81 -> but it's different now.

2247.3 -> The environment that surrounded us in the past

2251.98 -> was less complex than now.

2254.74 -> You know, in particular single device for example.

2258.61 -> The PC was the more than the half the way

2262.06 -> and the future was predictable based on simple information.

2267.31 -> But now there are a lot of new applications

2271.63 -> and various devices such as PC, server, mobile, IoT cloud,

2277.24 -> and the environment factors

2278.86 -> such as the pandemic, logistic costs,

2281.68 -> which poses greater impacts to the future.

2285.79 -> So the demand forecast has become more important

2291.34 -> which, you know, more important than than before.

2294.64 -> So we calculated we should change the methodology

2298.54 -> of demand forecast to cope with this complex environment.

2305.05 -> Yeah I think you guys are worried

2306.52 -> about the same in your field.

2308.68 -> Because...

2309.88 -> So looking back, the methodology, the demand forecast

2314.203 -> that we have done in the past.

2316.06 -> First the customer VOC.

2318.4 -> You know customers demand is really important

2321.25 -> but also highly volatile.

2323.5 -> And the result of customer strategy.

2327.16 -> So it is really difficult make an important decision

2331.39 -> by solely listening to the customers views only.

2335.53 -> Second thing is you cannot simply make decision relying

2339.88 -> on the external researchers forecast.

2342.97 -> And also, simple integration is good technique

2345.55 -> but it is really hard to detect the inflection point.

2349.75 -> And also, it is really hard to reflect the new factors

2353.71 -> of the future.

2357.01 -> So that's why we have tried

2359.5 -> to advance the demand forecast methodology

2362.8 -> by discovering new and sophisticated ways

2366.01 -> of observing these complex environments

2369.52 -> such as the technological change, the uncertain the future,

2374.23 -> and so on.

2375.73 -> Also, we have tried to combine the experience

2380.62 -> and our domain knowledge with data analysis

2384.07 -> to gain insight and solutions to our problems.

2388.21 -> So we looked for the type of services

2392.2 -> that business users like me

2394.51 -> could easily use and get output.

2399.34 -> So it may sound unreal

2402.34 -> but to me the news on SageMaker Canvas

2405.67 -> was similar to a movie.

2408.07 -> I immediately contacted to the AWS Korea team

2411.4 -> after encountering these places.

2414.19 -> AI machine learning.

2416.02 -> We done machine learning expert as shown here.

2419.56 -> Yeah, of course I had some skepticism too back then.

2423.43 -> So looking back on my steps to today's presentations

2427.45 -> we had a kickoff meeting with AWS Korea team in April.

2431.65 -> You know as I mentioned, we are a business users

2434.68 -> so we need the training to use them.

2438.43 -> So through our data lab program we learned

2441.73 -> how to use the SageMaker Canvas

2444.19 -> and we learned how to, you know, pre-processing the data.

2449.62 -> And now I'm presenting journey to you.

2455.44 -> As expected, the whole process did not go smoothly.

2461.59 -> You know, most of you guys usually know what S3 is

2467.35 -> but I didn't.

2468.34 -> I never used cloud

2471.22 -> used AI and machine learning services before.

2474.37 -> So you know, I tried.

2476.71 -> I start try to understand what S3 was first.

2480.1 -> Even my boss, you know, Tommy Kwon,

2482.65 -> he asked me what's the S3 meaning.

2485.68 -> Storage, storage, storaging?

2487.3 -> No simple storage service.

2488.77 -> So with more questions answered

2492.19 -> I gradually become acquainted with AWS more and more.

2497.32 -> And second thing is

2499.69 -> I thought that if I just unloaded the data

2502.33 -> to the AI machine learning service

2504.31 -> the AI machine learning would predict the future.

2507.49 -> But I soon realized that

2510.58 -> we need the necessary data preparation,

2514.42 -> formatting and structuring were required.

2518.74 -> At first, even if I get the output,

2523.679 -> the the result, I didn't get the good result.

2526.66 -> So I had challenging judging whether

2529.96 -> to believe in the unconvincing result.

2535.75 -> Despite the difficulties, our team serve challenges

2538.66 -> one by one with AWS team.

2542.178 -> We started with a very small project first.

2545.98 -> We collected the data with Excel.

2548.98 -> Our very familiar tool.

2551.05 -> And we preceded the data

2554.2 -> through a tool which is called Wrangler.

2558.25 -> Wrangler was available with a click in the console.

2561.88 -> And especially the resend for function

2563.8 -> was very impressive as it makes our data richer

2568.3 -> and helped us, the company, the data,

2570.22 -> by seeing the distribution of the value.

2573.61 -> After that I unloaded it to the SageMaker Canvas

2577.63 -> with well organized data

2580.09 -> that matched required data structure.

2584.32 -> What I want to emphasize here.

2587.02 -> All of this were simply done with a click in the console.

2592.66 -> So the business users like me could easily use the tool.

2600.52 -> For now I will explain the steps

2603.7 -> for using SageMaker Canvas.

2605.92 -> My first impression was Canvas was easy and user friendly.

2611.68 -> You know, even my first line manager, Lena Lee,

2615.55 -> she took one day training program and

2618.67 -> after this she mastered how to use it.

2622.51 -> Don't be surprised.

2624.07 -> You'll find it very simple.

2626.35 -> And also you can just start right away

2628.78 -> by selecting your data on your PC or S3 account.

2634.36 -> And once data uploaded it

2637.03 -> the appropriate model is automatically recommended.

2641.08 -> So I mostly use the time series forecasting model.

2644.71 -> And when I started the training,

2646.72 -> which is called Build here,

2648.19 -> and then for 10 minutes to several hours.

2654.64 -> And when the research came out

2657.49 -> we were able to analyze them.

2660.37 -> It was especially great that you could immediately check

2663.28 -> which related data affected the result.

2667.66 -> With this you can acquire better training result

2671.44 -> by adding or deleting the related data.

2676.12 -> Isn't it Very simple?

2677.8 -> Right?

2680.89 -> Yeah the two graphs shows the actual predicted result.

2685.99 -> The graph above is the result

2688.87 -> of the forecasting with historical data only.

2693.34 -> The graph below is the result of the forecasting

2697.57 -> with related data.

2699.61 -> Comparing two graphs.

2701.41 -> Which graphs do you think is more accurate?

2705.4 -> Yes.

2706.78 -> So you know, I wanted to see the quarterly demand

2712.45 -> for the next eight quarters by PC set shipment.

2716.86 -> So when I added the related data

2720.52 -> that is influenced the future demand,

2723.16 -> the focused accuracy increased dramatically

2727.93 -> and then the graph gave us the upper bound

2730.09 -> and lower bound rather than showing a specific value.

2734.56 -> So we are more convinced to combine the result

2739.24 -> with our domain knowledge.

2744.37 -> You know after using SageMaker Canvas

2746.56 -> we had the experience of the positive changes.

2750.16 -> You know, business user like me

2754.33 -> have been able to use AI and machine learning tools

2757.63 -> and get help to make data driven decision for the future.

2763.45 -> Yeah, of course, focused accuracy is getting better,

2766.69 -> but you know,

2770.44 -> our team has some still very important task left.

2776.59 -> I think most of you guys are also worried about,

2778.84 -> in your field, about the demand forecasting.

2782.98 -> How would you explain it?

2786.4 -> So when your supervisor, or your boss, your manager asked,

2791.14 -> or time management you asked,

2792.707 -> "Why AI imagine only forecast like this"?

2795.22 -> That is really difficult to explain, you know.

2797.98 -> So currently explainable AI is very important now.

2802.33 -> And the reliable and actionable

2805.06 -> is also very difficult problem.

2807.82 -> So and how can we make better demand forecast next time?

2814.87 -> You know, I think we are not predicting the future.

2817.81 -> We are analyzing the data.

2820.09 -> We are going to develop it further

2823.63 -> and build a bit again in this model

2826.27 -> and develop it further more.

2828.34 -> You know, the difference between us

2830.47 -> and the fortune tellers

2831.88 -> is fortune tellers prediction accuracy is based on luck.

2836.56 -> Each all about luck.

2838.27 -> Meanwhile, using data and AI machine learning like this

2842.65 -> we can continuously advance the forecast accuracy

2848.53 -> over time even though there may be some

2852.19 -> and more errors initially.

2856.87 -> Accordingly, our team desires to be evangelist

2861.01 -> of this movement and spread this culture

2864.73 -> within our team beyond the small project we

2868.39 -> are currently working on.

2870.79 -> So let's work together.

2872.53 -> And to design the future

2876.01 -> that beyond the pivoting, the demand outlook

2878.89 -> I just discussed about.

2881.05 -> Thank you so much for your time to listen.

2883.42 -> And I hope you enjoyed the rest of the re:Invent session.

2886.21 -> Thank you very much. (crowd applauds)

2891.4 -> Yeah.

2892.54 -> Sure.

2893.373 -> Yeah, thank you.

2894.39 -> - Thank you.

2896.357 -> Thank you Derrick for sharing your story.

2898.66 -> What a wonderful story.

2899.68 -> I mean it's always gratifying to hear from our customers.

2903.52 -> Talk through the journey,

2904.57 -> the challenges, and the success.

2906.76 -> Thank you again Derrick for coming all the way

2908.38 -> from South Korea.

2909.97 -> We truly appreciate it.

2912.37 -> So let's bring it all together.

2914.68 -> Right.

2916.75 -> We talked about democratization of machine learning.

2920.2 -> So the keys.

2921.4 -> No-code is the future.

2923.26 -> You really don't need to know coding

2926.038 -> to use machine learning and achieve your outcomes.

2931.99 -> Give the analysts the best practice without code.

2937.51 -> And finally, about seamless collaboration.

2939.82 -> You saw in the demo how we built a model

2943.39 -> and we could share it with the data science team

2946.09 -> for them to go under the hood,

2949.21 -> look through the details if needed,

2951.19 -> because the goal is for the analysts

2953.71 -> and the ML practitioners operating as a single team.

2958.9 -> So we'd love for you to explore this further.

2962.32 -> We have a couple of workshops tomorrow for Canvas.

2965.53 -> So you could actually get your hands on.

2967.72 -> So please do

2969.82 -> try Canvas tomorrow.

2971.41 -> We have two sessions of workshops tomorrow

2974.26 -> at the Caesar forum.

2975.7 -> We also have a lab tomorrow at the Venetian.

2978.91 -> We'd love for you to get hands on with Canvas.

2984.85 -> If you wanna do it at home.

2987.31 -> We have a course on Coursera

2989.5 -> which we launched earlier this year.

2991.75 -> It's a course about practical decision making.

2994.66 -> It gives you the fundamentals of data science.

2998.32 -> It gives you a chance to try Canvas for free.

3001.38 -> So please do check it out.

3003.39 -> It's been written by AWS professionals

3006.45 -> and I would definitely encourage you

3009.06 -> to get your hands on this course.

3011.94 -> For those of you who like to play with code.

3015.48 -> We have more sessions.

3017.04 -> Some of the low code services like SageMaker Data Angler

3021.42 -> for data prep, SageMaker autopilot for Auto ML,

3025.59 -> and SageMaker Jumpstart for pre-team models.

3029.01 -> We have a bunch of sessions through this week.

3031.62 -> So please check it out.

3035.94 -> And finally, could always go to the Canvas website

3038.94 -> where all this information is available

3041.61 -> of how Canvas works, resources, blog posts, videos and more.

3048.72 -> But last but not the least, we wanna hear from you.

3051.75 -> Canvas is built based on feedback from customers like you.

3054.87 -> So please let us know your feedback,

3056.94 -> not only for the session,

3058.68 -> but also what you would like us to see.

3060.6 -> How can we help you

3062.52 -> use Canvas to achieve your goals?

3067.47 -> Thank you again.

3068.49 -> and we truly appreciate you joining us today.

3071.01 -> Thank you. (crowd applauds)

Source: https://www.youtube.com/watch?v=vH5xc8C3OX4