AWS re:Invent 2020: Sharing code at arm’s length with AWS CodeArtifact
AWS re:Invent 2020: Sharing code at arm’s length with AWS CodeArtifact
Companies that use software package management systems can facilitate sharing the code across teams. However, they need various mechanisms to ensure the code being shared is reliable. In this session, learn how to use AWS CodeArtifact to safely and effectively share software artifacts across teams and explore pipeline processes for quality assurance.
Learn more about re:Invent 2020 at http://bit.ly/3c4NSdY
Subscribe:
More AWS videos http://bit.ly/2O3zS75
More AWS events videos http://bit.ly/316g9t4
#AWS #AWSEvents
Content
1.199 -> hi i'm john standish i'm a solutions
2.96 -> architect with aws
4.48 -> and i was a developer before i came here
7.04 -> so
7.759 -> working with packages was something i
9.44 -> was very familiar with
11.44 -> especially working with package managers
14.719 -> companies often use uh package managers
17.84 -> to control and restrict possibly the
20.24 -> security of the packages from
21.439 -> third-party
22.64 -> um package managers they may have local
25.599 -> shared packages such as auth
27.199 -> encryption and one of standardized on
29.279 -> versions
31.039 -> so let's walk through a customer
33.12 -> stackery in this case
34.559 -> and the benefits they got out of using
36.64 -> code artifact
38.32 -> stackery had issues with a
42.559 -> open source package they were using uh
44.399 -> it had a connection pooling issue in it
46.48 -> when they were trying to connect to a
47.52 -> database
48.719 -> this caused the application to have
50.079 -> those intermittent connections and
51.6 -> connection failures
53.12 -> so what they end up doing is they
56.399 -> took a version of the package modified
58.879 -> it and checked it into code artifact
61.68 -> this way they didn't have to wait uh for
63.6 -> the upstream package author
65.519 -> to go ahead and fix that
69.52 -> so what is code artifact well code
71.36 -> artifact is a managed artifact
73.2 -> repository
75.119 -> let's walk through some of the core
76.96 -> concepts of code artifact
80.56 -> so code artifact you have a domain and a
82.799 -> domain is an
83.759 -> aggregation of multiple repositories
88.08 -> and repositories are a they contain a
91.36 -> set of packages
92.64 -> each which map to a set of assets
97.2 -> those packages are a bundle of software
100.64 -> together with metadata to require to
102.88 -> resolve dependencies and install
104.88 -> software
106.399 -> and inside those you have assets
109.84 -> and assets are individual files stored
112.24 -> in code artifact
113.2 -> associated with a package version such
116.479 -> as
116.799 -> an npm tgz file maven palm
120.479 -> jar files nuget new spec files
123.84 -> and the like
127.2 -> code artifact supports multiple
129.36 -> different package managers
131.039 -> with java you have maven and gradle so
133.12 -> any java based language
134.56 -> such as kotlin scala or closure is also
136.959 -> supported
138.64 -> javascript you have npm and yarn you can
141.44 -> use
142.319 -> with python you have pip and twine you
144.72 -> also have nugetsupport4.net
147.599 -> and there's many more languages to come
149.28 -> and i encourage you to reach out to your
150.959 -> team or to the community about which
152.4 -> languages you want to see supported
156.48 -> now let's walk through an end-to-end
158.959 -> demo
159.76 -> of code artifact in use we're going to
162.879 -> provision our resources
164.16 -> so our domain and our repository
167.2 -> we're then going to take a a project
171.28 -> that is currently not having a package
173.76 -> that we're going to build
175.36 -> push into code artifact consume that
177.44 -> package and fix that project
179.76 -> so let's walk through that
185.28 -> code artifact you can find it in the aws
187.2 -> console by going to services
188.8 -> it's down here in the developer tools
190.959 -> under code under code artifact
194.239 -> or you can type it in here and select
197.68 -> the service so let's go there we're
199.04 -> going to select our service
202 -> and if you look in here we don't have
203.36 -> anything provisioned we don't have any
205.44 -> domains
207.04 -> nor do we have any repositories so let's
210 -> go ahead and provision those we're going
211.12 -> to go to cloudformation
215.12 -> we'll go ahead and click create stack
218.4 -> and we're going to upload our
220.4 -> cloudformation template
221.599 -> which we'll go into in a little bit in
223.68 -> the details of that cloudformation
226 -> template let's give it a name reinvent
229.04 -> in this case
229.68 -> for the stack and hit next
234.48 -> and go down click next
238.319 -> and we're going to provision that stack
243.519 -> and we're going to watch that stack it
245.36 -> should be done any moment
248.4 -> and now that our repositories and
249.68 -> domains are created
252.08 -> let let's go into code artifact inside
255.04 -> the console
255.76 -> and see what's been provisioned
261.84 -> so if you look in here we have two
263.12 -> repositories that have been created
265.04 -> my team repository and external
266.84 -> repository
268.24 -> so if you click on my team repository
270.56 -> and go to the details you're going to
272 -> see
272.32 -> there's an upstream connection to my
274.32 -> external repository
277.6 -> if we click on that there's a connection
281.12 -> to an external repository a public one
283.68 -> this is nuget.org
285.759 -> and we'll go into detail later on what
288.639 -> external uptrend connections are
291.28 -> let's go over here into visual studio
294.32 -> and we're going to look at a package or
295.68 -> a project here
298.32 -> that currently does not have reference
301.68 -> to code artifact sample that package
303.44 -> does not exist
305.36 -> so we look up here you can see in our
308.16 -> project
309.199 -> in our c sharp file it will not compile
312.72 -> we don't have a reference to the package
314.32 -> we do not have
315.759 -> the classes that are being outputted for
317.36 -> logging as you can see here it doesn't
319.199 -> exist
322.24 -> so let's go over to our shared project
325.199 -> which has
325.919 -> a reference to log4net this is a wrap
328 -> around log for net
331.6 -> and let's go ahead and install log4net
334.479 -> as you can see
335.6 -> it hasn't been brought in so we'll go
338.56 -> over to the project and we'll hit manage
340 -> new get packages
343.199 -> we'll go ahead type in log for net
348.08 -> and we'll see it one thing i want to
349.68 -> point out is this is using code
351.52 -> artifact as a package source so if you
353.84 -> look here we're not using the public
355.52 -> reference to it we're using
357.36 -> only code artifact to pull the packages
359.68 -> through
361.759 -> so i'll go ahead and hit cancel i'll go
363.919 -> ahead and install
365.039 -> so we'll install this package from
366.88 -> log4net
369.919 -> and installs and you'll see all these
371.28 -> errors go away
376.08 -> now let's go ahead and compile this make
378.56 -> sure everything's working
383.44 -> and we're good to go
387.28 -> so now we want to be able to push this
388.88 -> package up we're actually going to
390.319 -> publish this package so we're going to
391.68 -> use
392.84 -> netpak and dotnet nuget push
396.4 -> this nativecommand.net
400.639 -> but first what we need to do is we need
402 -> to log in to
404.08 -> code artifact and get an authentication
406 -> token to push
407.12 -> so i'm going to go up a directory into
409.28 -> the support directory and
410.639 -> run login ps1 that's just a wrapper
412.96 -> around the aws cli to get an
414.96 -> authentication token
417.84 -> as you can see here it successfully
419.44 -> grabbed an authentication token that's
420.88 -> good for 12 hours
423.36 -> i'm going to go back and then i'm going
425.84 -> to
426.8 -> publish this package up
436.8 -> and as you can see it's successfully
438.16 -> pushed the package using.net pack
440.88 -> and nuget push and it's pushed up to our
442.88 -> team repository
445.12 -> so now let's go back to our project our
448.08 -> main project
449.759 -> and pull down that package now that's
452.08 -> been pushed to code artifact
454.96 -> go back to manage nuget packages
458.24 -> our code artifact our source is set to
459.919 -> code artifact
463.039 -> and i'm going to type in shared since
464.4 -> that's the name of the package that we
465.599 -> pushed up
468.4 -> and there it appears and we're going to
470.08 -> hit install
475.36 -> so now that it's been installed let's go
477.599 -> back you'll see the arrows go away
480.8 -> and let's go ahead and run this
487.28 -> there you go as you can see we're using
490.16 -> the log4net wrapper
492.4 -> and everything is working let's go and
494.479 -> close this let's go back to code
495.919 -> artifact
497.12 -> let's show what lives inside the
499.039 -> repositories after we pushed up
504.16 -> so go my external repository there's
505.68 -> nothing there right now
508.08 -> if i go to my team repository what
510 -> you'll see is you'll see the share
511.12 -> package that we pushed
513.599 -> and i'm going to navigate into that
514.88 -> you'll see the version we have here
518.24 -> you'll see the assets are in here it's a
520 -> new bit package i was actually pushed up
522.08 -> and the dependency on log for net and
524.959 -> everything resides within there
534.72 -> so now that you have seen us provision
537.92 -> repositories and domains how do you
540.72 -> manage these
541.76 -> how can you control the sprawl now that
543.36 -> you can have n
544.8 -> number of repository you have multiple
546.839 -> repositories
549.36 -> and often you had one or two or three
552.48 -> repositories
553.76 -> and as more packages became available
555.6 -> and more repositories were needed
557.76 -> for segmentation it became increasingly
559.76 -> difficult to organize trace discover
561.6 -> and govern the sprawl so how would you
563.6 -> do that well
566.72 -> what we provisioned in our demo was a
568.56 -> shared repository
569.839 -> that shared repository had a connection
572.8 -> an external connection
574.56 -> to nuga.org now an external connection
577.519 -> allows the packages within the public
580.24 -> repository
581.279 -> here nuget or npm whatever you choose
585.519 -> to be referenced and pulled through to
588.48 -> the shared repository
590.56 -> also when you pull through a package it
592.24 -> will keep a copy in code artifact for
594.64 -> that version
595.839 -> so that if something should happen to
597.2 -> the upstream repository that you see
599.279 -> here
600.64 -> you have a copy and you can go on go
603.04 -> along your way
606.8 -> we also provisioned a team repository
608.64 -> but you could also provision different
610.16 -> other kinds of repositories
612 -> such as a sandbox or even a functional
614.56 -> repository now a functional repository
616.8 -> is one that's split by a
620 -> domain in domain driven design or a
622 -> service or another functional boundary
625.12 -> shared could contain packages such as
628.32 -> authentication logging or any other
631.76 -> commonly used packages within your
633.68 -> organization
636.079 -> now say you wanted to pull packages from
638.56 -> the shared repository you wanted to have
639.92 -> visibility into those
641.68 -> from your team repository well that's
643.92 -> where you would establish an upstream
645.2 -> connection an upstream connection
647.279 -> enables the package manager client to
649.36 -> access packages that
650.56 -> are contained in more than one
652.64 -> repository
654.24 -> via a single repository endpoint
658.079 -> now say you wanted to iterate
661.2 -> on those packages and you would do that
663.68 -> with your sandbox repository
665.12 -> well by having that you can create an
666.8 -> upstream connection
668.24 -> from your team repository connect it to
670.48 -> your sandbox repository
672.32 -> and now you have reference to the
673.6 -> packages that are in team and
675.68 -> shared likewise when you push up
679.12 -> into the team repository when you
680.64 -> promote
682.56 -> all the packages and references are
684.24 -> maintained
686.079 -> and not broken
690.079 -> and we provisioned this using cloud
691.44 -> formation and let's walk through
693.76 -> what the confirmation was that we used
695.76 -> to provision this
698.079 -> so we created a domain and we gave it a
700.24 -> name this is example domain
703.6 -> and we have my shared repository in my
706.079 -> shared repository
707.76 -> it had an external connection in this
709.36 -> case we changed now to
710.8 -> npm.js we could use nuget.org or even
714.839 -> central
717.2 -> and then we also provisioned a team
718.959 -> repository that had an upstream
720.48 -> connection
721.519 -> to the shared repository
724.639 -> and this is how the connection is made
726.16 -> using that git attribute now
730.56 -> if you were to use a build system to
733.68 -> push up
734.8 -> which most of us do
738.16 -> how would we do this well you can use an
740.079 -> existing ci system you could use also
742 -> code build
743.04 -> or any other tool you use to build your
746 -> packages
749.279 -> so what we do is we use the awcli
752.48 -> we use code artifact login and you
753.92 -> specify the tool in this case would be
755.76 -> npm
757.12 -> we specify the domain and here it's my
760.24 -> org
760.959 -> and the account id that's domain owner
764.399 -> this allows you to have cross account
767.279 -> usage
767.92 -> for logging in we then specify the
771.2 -> repository that we want to log into
773.519 -> now what this is doing is it's going to
774.88 -> give us an authentication token
776.639 -> that we can then use in our native
778.32 -> package managers our packet manager
779.839 -> clients
781.68 -> next run our npm command in this case
784.32 -> npm run build we'll set the version of
786.639 -> the package
787.6 -> and we'll push it now that we've pushed
791.2 -> the package up
792.32 -> say you wanted to promote from our
793.44 -> sandbox to our team repository
796.16 -> well you can do that using the copy
797.92 -> package version
799.36 -> the copy package version you specify the
802.24 -> domain
803.04 -> and domain owner the same way you did
806.24 -> with the login but you specify the
809.44 -> source and destination repositories
811.6 -> here we're going from my team repo to a
814 -> shared repo
816.16 -> specify the package the format
819.279 -> and then the version now what this will
822.079 -> do
822.88 -> is it will not copy a
825.92 -> instance of the package instead it will
828.8 -> copy
829.6 -> a reference to it what this means is
832.079 -> that the storage itself is deduplicated
834.24 -> underneath
836.72 -> you don't double pay for having two
838.88 -> references to a package there's only one
840.839 -> instance
842.8 -> and what this allows you to do is near
844.639 -> instantly promote a package from one
846.8 -> repository to another
851.279 -> now that you've been able to build and
854.48 -> promote
855.36 -> and provision your packages with code
857.92 -> artifact
859.199 -> how would you consume this with your
860.56 -> team
863.44 -> so as developers we want to be able to
865.12 -> pull those packages using our native
867.36 -> tools
868.32 -> and there's a couple ways of doing it
869.68 -> one was through the abcli that you saw
871.6 -> there
874.24 -> now if you forgot what the command is to
876.639 -> run it you can go to the console
878.32 -> and hit view connection properties
882.32 -> it will bring up this modal here and you
883.76 -> can select your package manager here
885.199 -> we'll select npm
887.279 -> and then it will give you the snippet of
888.399 -> the tool here for you
890.639 -> you'll take that put that in your cli
892.88 -> run it
894.32 -> and you're good to go then you can use
897.44 -> any of your native tooling you have
899.199 -> today you can use npm
902.399 -> or yarn we mentioned earlier for
903.839 -> javascript doing an npm install
905.76 -> with python you could use pip or twine
909.04 -> with java we talked about maven and
910.56 -> gradle you can use all those and with
912.24 -> dotnet you can use
914.24 -> sdli or even nuget now underneath what's
917.519 -> happening with the npm one
919.12 -> is it's modifying the mpmrc file it's
921.519 -> putting in
922.16 -> the registry url and the authentication
924.639 -> token for you in
926.399 -> net it's modifying the new get conf file
929.12 -> and in python
930.16 -> it's modifying the python file for you
933.36 -> and also works in a virtual environment
939.279 -> now a common restriction of existing
942.56 -> artifact repositories was you had to
944.16 -> abide by either an api
946 -> or a plugin and it was made harder and
949.68 -> harder to extend it
952 -> with code artifact you have access to
953.6 -> extend this using cloudwatch events to
955.68 -> fit your organization's needs
958.8 -> so let's talk about some advanced use
960.32 -> cases here's an example
962.72 -> of a promotion pipeline with multiple
965.6 -> stages in it
966.72 -> where you have a team repository a
968.48 -> staging repository
969.759 -> and a production repository now
973.279 -> developer goes ahead they check in their
975.279 -> code and here in code commit or any
977.36 -> repository that triggers a build and
979.759 -> publishes into
981.12 -> our sandbox repository that sandbox
984.399 -> repository fires an event that says
986.72 -> a package has been created or updated
989.199 -> that will then trigger a flow
991.04 -> in this case it's configuring air firing
994.16 -> a step function flow
996.16 -> that step function flow goes through
998.639 -> multiple steps in the team repository
1000.88 -> stage it could be doing some basic ui
1003.68 -> testing
1004.8 -> maybe some compliance checks maybe some
1007.12 -> security scans
1009.199 -> if that's successful it pushes into the
1012.079 -> team repository
1013.36 -> and then goes on to the next stage and
1015.519 -> as it gets to production it gets
1016.72 -> continuously more strict on times it's
1018.32 -> testing
1019.68 -> another flavor of this could be if you
1021.519 -> had a quarantine
1023.36 -> repository that ran a workflow to
1026.64 -> check for licenses do some some code
1029.28 -> scanning
1030.16 -> that would then eventually promote that
1031.6 -> into a shared repository that's another
1033.839 -> flow you could use
1035.839 -> another extension you could possibly do
1038 -> is dependency based notifications
1041.76 -> often when you're developing and you're
1044 -> using package managers
1045.199 -> there's versions assigned to it and
1047.76 -> there's new packages that are created
1050 -> well how do you get a notification if
1052.24 -> the package
1053.36 -> you're using changes maybe a team has
1056.96 -> updated something fixed a bug added a
1060.08 -> new feature
1061.36 -> and something of that nature how would
1062.72 -> you get notification well in this case
1064.72 -> we've done the same subscription with a
1066.16 -> cloudwatch event for the package
1068.799 -> and then we have the repositories with
1071.84 -> the packages
1072.64 -> split into a dynamo table that has
1075.28 -> references
1076.08 -> to which versions and packages are used
1078.64 -> for a particular repository
1080.48 -> or actually for a particular project we
1083.52 -> then
1084.16 -> have a list of slack channels with those
1086.32 -> repositories as references
1088.88 -> we connect in those based on the version
1090.799 -> that changes we fire into the channel
1092.88 -> a message say hey this package has
1094.24 -> changed and you know how to complete
1096 -> visibility to that
1096.96 -> so if you were trying to get around a
1098.559 -> bug or there's something happening
1100.88 -> you have this ability okay i don't need
1102.64 -> to continue on this i'd have you meet
1106.84 -> visibility
1109.52 -> so let's walk through real quick what we
1112 -> just covered
1113.039 -> and how using code artifact
1116.4 -> is different than the self-managed ways
1117.84 -> that many may be doing today
1120.559 -> with a self-managed you're required to
1122.24 -> set up operate
1123.6 -> and manage the scaling aspects of it the
1125.44 -> code artifact you only have to worry
1127.28 -> about the domain the repositories
1128.96 -> is security so security being resource
1131.52 -> policies themselves
1136 -> also have limited durability and
1137.44 -> security because you had to control that
1139.039 -> yourself
1139.919 -> you had to set up the storage arrays you
1142.48 -> had to set up the encryption at rest
1144.64 -> any of that things of that nature with
1147.039 -> code artifact
1148.08 -> you inherit the durability and security
1151.039 -> the
1151.52 -> durability for the storage comes because
1154.32 -> code artifact is backed by s3 with at
1156.32 -> the 119's of durability
1158.08 -> you can establish a zero trust policy
1160.08 -> using resource policies and iam
1162.799 -> and it's encrypted at rest by default or
1165.919 -> you could bring in your own encryption
1167.28 -> keys using your own cmk
1170.72 -> you also had upfront fees for your
1173.36 -> self-managements you had to
1175.12 -> purchase a enterprise version and that
1178.08 -> required
1178.72 -> a sizable investment with code artifact
1181.12 -> you only pay for what you use you pay by
1183.28 -> the request
1184 -> and you pay by the storage and remember
1186.799 -> multiple package versions are
1188.16 -> deduplicated
1189.12 -> so you only pay for that one instance of
1190.88 -> the package
1192.88 -> and then there was limited extensibility
1194.88 -> extensibility only by the api
1196.96 -> or the plug-in for the existing
1198.64 -> self-managed repositories
1200.72 -> now you can extend using cloudwatch
1202.4 -> events to fit your company's needs
1207.52 -> we also talked about native tool
1208.72 -> integration with npm
1210.799 -> with nuget with python and with java
1214.159 -> we talked about cloud cloud formation
1215.6 -> support and how you can quickly spin up
1217.12 -> domains and repositories
1219.28 -> and how you could extend that using
1220.32 -> cloud watch events
1222.32 -> all the code that we used is available
1224.48 -> here in this github link
1226.159 -> and so you can get started today i just
1228.96 -> want to say thank you
1229.919 -> and i hope you have a wonderful re
1231.2 -> invent please fill out the survey
1234.48 -> thank you
Source: https://www.youtube.com/watch?v=vEwv2cBS-MQ