Amazon CodeCatalyst’s packages support – a glimpse at what’s to come for artifact management

In this post you will get a short introduction and my personal assessment on how the new Packages component in CodeCatalyst can help you to set up your complete SDLC in Amazon CodeCatalyst.

Only npm supported – a early launch to show what we can expect going forward and to get feedback from users?

With the initial launch Amazon CodeCatalyst allows to manage npm package repositories inside CodeCatalyst. When seing this for the first time I was a bit concerned that this decision – to initially launch with only npm support – will not help a lot, as I expect that organizations would need to store other type of artifacts (jar files, containers, python paackages, …). But I think that at least for Cloud Native Projects that use typescript this will solve the problem of storing artifacts and accessing them natively within CodeCatalyst. Let’s look at how they can be used today.

Using packages repositories in CodeCatalyst

Today you can set up package repositories in CodeCatalyst and connect to upstream (public or private) repositories. You can also change the sort order and the upstream order of the package repositories. The documentation covers the different possibilities to set up repositories very well. You can access the package repositories from your local machine by setting up the connection locally

npm config set registry https://packages.region.codecatalyst.aws/npm/space-name/proj-name/repo-name/

After setting up multiple repositories you can set the order of retrieval and usage. You can also set it up as a “pass-through” repository and allow access to public repositories.

Using packages in CodeCatalyst

Packages in CodeCatalyst are integrated with workflows. You can read or store packages with native actions or by setting up the npm repository manually.

You can also read and publish packages from other systems by using Personal Access Tokens (PAT). The documentation outlines currently supported client commands.

The CodeCatalyst workflows will by default use the set up repositories that you have set up within CodeCatalyst.

What I think we need next in packages

With the launch of the packages component in CodeCatalyst AWS has added a definately missing functionality in CodeCatalyst. Users are not able to store artifacts and share them within a space which makes it possible to create, re-use and deploy immutable artifacts natively within the tool.

This is a much needed functionality, but as I already mentioned, I believe that limiting this to npm packages only limits the use cases for it. I would have expected npmpython and docker support at launch and was a bit disappointed that this was not included.

I do also believe that the “packages” functionality should be better integrated and allow further configuration options, especially when loading/reading packages from upstream repositories – e.g. being able to limit to only certain licences or ensuring that packages included are still “supported” (and regularly updated) or security-scanned.

These type of functionalities would have made the new option “meaningful” and would have empowered developers to build better software too.

I can imagine that we will see pythonjava and dockersupport pretty soon due to these reasons and it would be great to also support “internal” repositories as “upstream” repositories.

How do YOU think that this new functionality will help you to adapt CodeCatalyst?
Please drop me a message on socials or an E-Mail!

Views: 562

Amazon CodeCatalyst’s integration with the IAM identity center allows SSO with your own IdP

In this post you will learn how the new integration of the IAM Identity center with Amazon CodeCatalyst allows to use your own IdP as single source of truth of users for CodeCatalyst spaces. We will look at how you can connect a space to an IdP and on how this can be used to set up certain permissions based on your IdP roles.

Single Sign On (SSO) using IAM Identity Center empowers CodeCatalyst users to use their own IdP to provision user accounts in CodeCatalyst

With the new integration for Amazon CodeCatalyst into the AWS Identity Center organizations can now connect to their own existing IdP (AzureAD, Ping, Okta, …) going through the AWS Identity Center.

Benefits of the new integration

With this new integration users and organizations go to the Space settings in CodeCatalyst and choose to integrate with an existing instance of the IAM Identity Center.

This also connects potentialy to any other IdP that uses the IAM Itentity Center as a proxy. Administrators will only need to manage roles and permissions in on central place. The roles from the IdP can then be mapped to certain team roles in Amazon CodeCatalyst.

What I don’ t like about the new integration

There are two things that I didn’t like when looking at this integration:

Why do we need to go through the IAM Identity Center and cannot allow direct SAML or OIDC integration for a specific space? With the “need” to have an IAM Identity center organizations also need to directly connect to an existing AWS Account to manage connectivity – and this once again brings back a “tooling” account that initially I was hoping to replace with CodeCatalyst.

The second thing that I missed was the possibility to automatically map existing roles to groups in Amazon CodeCatalyst – today this is a – one time – manually effort that needs to be conducted. If you need to do that only once for your organization that isn’t a problem, but if you decide to have multiple spaces for your organization and need to perform this multiple times, then this can be very time consuming.

Summary

I believe that this change will have a very big impact on the adoption for CodeCatalyst as administrators and security teams in organizations will make it easier to allow the usage now that there is a central place to manage identities. And with the possibility to actually try out CodeCatalyst in an enterprise world, I hope that we will also see more adoption of CodeCatalyst and with that also a more streamlined and better communicated roadmap of CodeCatalyst.
If you would like to learn more on how to activate this, please refer to the documentation.

Views: 189

Custom BluePrints in CodeCatalyst – templated projects that empower you to build better software

In this post you will learn how to build a Custom BluePrints in Amazon CodeCatalyst. You will get to know how Custom BluePrints allow you to generate a consistent setup of your projects and CI/CD workflows and how you can leverage this to empower your teams to be compliant with rules and deployment best practices. AWS has announced this today at his annual user conference AWS re:invent 2023 in Las Vegas.

Custom BluePrints – what they do, why they are there and how you can use them

Custom BluePrints can be seen as “project templates” that you can build and offer to all of your users of a space. As a user of CodeCatalyst you should already be aware of the general concept of a blueprint: It’s a templated project that you can start your software project on.

Custom BluePrints take this concept to a new level: Now you can build BluePrints for your projects!
And that’s not it: a project can be depended/generated by multiple BluePrints and a project generated from BluePrints can also become a BluePrint! yey!
You do not start from zero and you are now also abled to add a BluePrint to an existing project.

Think about what this means for you and your organization: You can set up certain default workflows, permissions, dependencies, environments, etc. and apply that to all of your new projects by default, without the need to “touch” anything.

And that’s not “it” – there is more: CodeCatalyst also allows you to “update” projects that have been build from a Custom BluePrint!

Think about the scenerio where you ahve over 20 microservices that all follow the same CI/CD pattern, including governance rules and dependencies. They all have the same dependencies and pre-requisites (e.g. AWS account permissions, environment setups, etc.). Now one of the dependencies has a critical, security relevant defect and you urgently need to upgrade all of these projects…

CodeCatalyst Custom BluePrints enables you to do exactly that with just a few clicks and pull request approvals!

I hope you are curious to look at this new functionality in details now, keep reading!

The technology behind blue prints – the power of Projen and Code Generation

Custom Blue Prints are powered by Projen under the hood – to learn more about Projen, please look here. Similar to a projen.rc.ts you will need to create a blueprint.ts file in the src folder of your BluePrint project. There you can then define the rules and automations which will be applied when the BluePrint is used as a starting point for a new project. Currently the Custom BluePrint SDK allows you to define Wizard configurations, Workflows, Environments, Source Repositories, Pre-Configured Dev Environments, … Using Projen as an underying technology the team is able to re-generate the code for your project and create a pull request on your behalf for the source repository of your project i there is changes. And this is great!

Building your first BluePrint

Building your first BluePrint is easy!

Create your BluePrint project

It starts with creating a new Custom BluePrint project in the CodeCatalyst UI using the BluePrint builder. This one can be accessed from the Space “Settings” page. After going through the wizard, this will create a new project for you in CodeCatalyst. In this project, you can now either checkout the source repository into your local IDE or you can use the DevEnvironments to directly get you started to work on the project.

Building and Editing Custom BluePrints can be a HIGH RISK

As you are reading this and as the functionality has only just been announced, it might be early – but it is never too early to warn about this:

If you edit a Custom BluePrint and potentially introduce security problems – maybe even unsecure dependencies – you might expose everyone that uses your BluePrint to challenges.

Better set up additional protections for all of your BluePrint projects!

Edit the sources of your BluePrint Project

Now that you your project is set up, you can start building your own components and parts of your project.

Be cautious to edit package.json in your BluePrint project – you can, but it might break some of the integrations. The typescript project is set up for you to be able to preview and publish your BluePrint.

The main “sources” of your BluePrint and the definitions will be in the src/blueprint.ts file. Initially, the project comes with a simple wizard set up with only a few parameters. It copies only the contents of the static-assets directory when being executed.

I’ve not been able yet to try out a lot of the functionalities and possibilities of the SDK, but still I was able to create a custom BluePrint that can be used to deploy a Flutter Web application with a Serverless Backend.

What I found out is that it will be a huge effort to set up the BluePrint in combination with a Wizard. This is not a 4 hour task, we’re rather talking about a week to get started. Further details on this – see the “Testing” section. This might also explain why this functionality is part of the new Enterprise tier for Amazon CodeCatalyst.

Please read the available options of package.json, but to get you started: use npm run blueprint:synth or yarn blueprint:synth to generate the BluePrint locally.

This will also execute any local unit tests that you have build for your BluePrint.

Testing your Blue Print locally

Using npm run blueprint:synth or yarn blueprint:synth will generate the BluePrint bundle locally and execute unit tests. It will generate a version of your project created by the BluePrint in the folder synth/synth.[options-name]/proposed-bundle/. This will use the default options you provided.

It is also possible to simulate the wizard adding the --cache parameter.

You could also build Unit Tests or SnapShots tests and integrate them into this lifecycle step of the build procedure.

Publishing your BluePrint to the Space Catalog

Use npm run blueprint:preview or yarn blueprint:preview to upload a “preview” version of your BluePrint to your CodeCatalyst space. This will make the Custom BluePrint available but not make it available in the catalog yet, so users cannot use it.

It also automatically increases the version number for your BluePrint.

Testing your Blue Print through the UI

After publishing your BluePrint preview from your local environment, you will now see this preview version within your Space Settings. From there, you are able to create a project out of the Preview version of your BluePrint.

Here you will need to test all options of the wizard and in addition to that verify that the customizations that you added have been correctly generated after you went through the wizard.

I have made the experience that a whole testing cycle can take 3-5 minutes – so I really recommend you to look at local unit tests.

After you have finished your tests, you can add a specific version of your BluePrint to the BluePrint catalog. This is currenty only be possible through the UI in the Space Settings.

Creating (one or many) projects from your BluePrint

Now you are ready to start creating projects from your BluePrint!

Go to the normal “Create Project” screen – after using “Create from Blueprint” you will now have the option sect “Space Blueprints” which is where you will find your newly created BluePrint.

This should now generate your project from your BluePrint and the structure should be as you wanted it to be. 🙂

You can also add additional BluePrints to an existing project – this allows you to incorporate multiple BluePrints in a single project, a combination of multiple “default” project settings.

Please be aware that this will remove manually added or created resources that are part of the BluePrint that you are adding – it will overwrite manual changes that “interfere” with the BluePrints code.

Upgrading your BluePrint

Upgrading your works similar to creating it – upload a new “Preview” version, add the new version to the catalog.

Upgrading your projects to a new BluePrint version

You can currently update the used Blue Print vesion for a project by going to the project settings, to the Blue Prints section there and then by loading up one BluePrint. Choosing to upgrade to a different or newer version will generate the proposed changes and make them visible to you before you apply them.

What’s next for Custom BluePrints?

Oh wow, this is one of the new features that I am most excited about for CodeCatalyst! There is so much potential! I have to explore further the possibility to have multiple BluePrints attached to a project, the possibility to remove a BluePrint from a project, and all of the other things that I didn’t touch yet.

Here’s stuff that I would love to get for Custom BluePrints:

  • auto-apply updates to ALL projects that are from the same BluePrint automatically
  • share Custom BluePrints with other Spaces
  • a Custom BluePrints market place where developers (like me) can put “their” BluePrints in and organizations can “buy” a BluePrint from

I also believe that the team needs to work a bit more on the developer experience when building Custom BluePrints, e.g. by pre-suggesting automated integrated tests (integration or snapshot tests). Publishing a new version to the catalog should also visualize the changes that are going to be added to other projects before you actually publish the new version.

I am a bit disappointed that the “free tier” does not give you the possibility to try out this functionality by being able to add one or two Custom Blue Prints to your space, but maybe this is something that will be possible as soon as the Blueprint SDK is stable.

What do you think of Custom BluePrints? What are your main wishes? Please reach out to me and let me know!

Views: 1270

Hey, I’m going to San Francisco and Vegas to meet friends at re:Invent 2023

In this post I’m writing about how getting to re:Invent 2023 was and about how the last year has made a big impact on my personality and my efforts in the AWS community.

Thinking it couldn’t get better last year, 2023 changed me once more and leveled me up

While going to re:Invent 2022 I wrote down my thoughts and went home from re:Invent expecting things could not get better for me. But this year has prooven my expectations wrong, as the last 10 months have been life-changing in a lot of different aspects.

Blogging & Starting a YouTube channel – winning 1.5 hackathons and starting a project

When I reached my home town after last years re:Invent I knew that this community is what makes me happy and gives me back so much that I also would like to give something back. And so, I formed a few ideas in my head and started a few initiatives that I would like to share with you today:

Blogging & Writing

I have been blogging both on dev.to and on my personal blog – writing about topics and things that I do or did on a regular basis, trying to share experiences of day-to-day things and building experiences.

The most important thing for me he is to always be transarent and authentic – speaking out on things that I liked and challenges that I’ve faced.

For you reading this – thank you, for being part of my journey, for giving me feedback and for being you! Please let me know if you have any feedback on what I should change or do better.

Kicking off a YouTube channel

The last three years of my career have been focused on building secure and reliable CI/CD pipelines! But hey, there is so much more to learn. As part of my travel back to my home town after re:Invent, I had the thought to share my experiences and the experiences of other builders on a “podcast-like” YouTube channel which is today known as @cicdonaws.

Learning on how to produce a YouTube video to how to set up a good lightning and sound, preparing a script and creating better thumbnails – a lot of learnings, experiences and hours went into the videos that I have been producing.

This only kicked off because of all of the great builders that joined me on the show and shared their projects, learnings and experiences with me. Thank you all, every single one, for your time, dedication and experiences. Keep up sharing your knowledge, as this is how we can all grow!

Winning 1.5 hackathons and making a small project something big

Back in march Lily announced a, Community Builders-internal hackathon where it was possible to win some prices – and I thought I had a great idea on what to build that actually fit into the “rules” of the hackathon. And this is how the project started that has since then used up countless of hours of my after-work evenings and weekends. Millions of Slack messages have gone from Germany to US, from US and Germany to Africa and back – we kicked of a real project with our Speakers Directory that is finally ready for prime time! After becoming second in the first Hackathon, we also submitted the same project with a few tweaks for a 2nd hackathon powered by Hashnode and Amplify and this also brought us “honourable mentions” in the list of top 10.

We’re ready now to get more of you to add your talks and events! Please sign up and give us feedback, we are very proud of what we have produced so far and eager to get all of you on board!

Thanks to Danielle, Julian, Matt, Raphal, Baimam for actively working with me on this fun project – and thanks to everyone else that has supported us through 2023 to make this a bigger thing!

Organizing a community day – the EU (more technical) edition of re:Invent

Another piece last year has been organizing the AWS Community Day DACH that happened in Munich this year – it was fascinating to be part of the team that brought more than 500 AWS nerds together, including over 25 AWS Heroes and 30 Community Builers!

Thank you all for attending – and thanks for the sponsors for making this possible!

We’re up for doing this again next year – further details to be announced soon! Looking forward to see you there!

Hey, hiking before re:Invent is what you need!

And then, there is this “secret thing”: Ever since I went to re:Invent I used the sunday before re:Invent to go out “hiking” somewhere close to Las Vegas. The first two visits together with colleagues were three persons – a pretty cool small group.

Last year, they did not go with me to re:Invent, so I asked Community Builders to join me – at the end, we were nine persons from I think 8 different countries hiking together from 10 am till 5 pm – we had a lot of fun, so I decided: Let’s do this again!

Somehow, this became a bigger thing – for this years hike, we have over 50 Community Builders, Heros and UG Leaders signed up. The group became so big that the AWS Community team is helping us this year by sponsoring the transportation.

Thank you all, that you are supporting this thing – and thanks to everyone participating, I am sure this will be an amazing experience for everyone!

How and Why what I do for the community changed

Since June I can name myself an AWS Devtools Hero – which honors me a lot but also drove a bit on how and what I do for the AWS community. My work has become a little bit less “public” – a lot of my hours that I have “free” went into talking to AWS Service Teams, into mentoring other builders and helping them grow. This has lead to less new videos on YouTube and less blog posts – but instead, I’ve helped others to grow and to invest into the community – supporting the “re-start” of the AWS UG Frankfurt, the re-start of the User Groups Mainz, Karlsruhe and Heilbronn and overall making other builders “grow”. This makes me happy. Thank you all, that you are part of my journey.

Making new friends

This year has also shown to me how important relationships are and how much you can make friends by supporting each other. Friends, I’ve never met in person (Matt), friends I’ve finally met in September (Ran) and others like Markus, Thorsten, Philipp and Raphael that are part of most of my days and that I write 100s of Slack messages per week.

This is what community means! Making friends, learning, growing. Thank you, for being you!

What’s next for me?

Changing my role at work

I’ve recently moved to the global Platform Architecture team here at FICO and that gives me a lot of new possibility and learning opportunities.

The platform that FICO is building is an important one and helps organizations around the globe to take better decisions – I’m proud to be part of the team and looking forward to make this bigger than it is today.

Community Day DACH 2024

We’re already planning next year. Stay tuned for some announcements…soon!

Supporting and mentoring builders around the globe

I’m up for helping other builders grow – if you’re intersted to collaborate with me or to learn from me, reach out to me and we can talk!

If you have any CI/CD or CodeCatalyst specific topics that you would like to talk about on my channel – reach out to me!

If you need feedback, help or advice on anything (e.g. a blog post or anything else) – let me know and reach out!

Looking forward to an amazing re:Invent 2023 with a lot of friends in Las Vegas!

Have a great week – don’t be a stranger and say HI if you see me around!

Views: 444

Experiencing GenAI using PartyRock – the best applications I’ve seen until now

As a few of you might have seen, AWS has today launched PartyRock – an Amazon Bedrock Playground that can be used to generate and build applications using GenAI technologies.

PartyRock is an educational tool for providing any builder with low-friction access to learn through experimentation in a foundation model playground built on Amazon Bedrock. It is not a product or service in the traditional AWS definition and should never be referred to as such. The preferred descriptor is playground, though in most cases tool is also acceptable.

AWS

German engineering – Party from Berlin

I’ve been fortunate to be able to know people that are behind this launch and I’m excited for it, not only because a bunch of the engineering team members are part of the AWS Development Center in Germany but also because I’ve tried to not touch anything related to GenAI until this tool was made available.

After being able to look at the tool and playing around with it, I can see a lot of benefits of using Generative AI going forward and I see a lot of the value that this technology can bring us in addition to “simple ChatGPT like” Chatbots.

Build your first GenAI App with PartyRock

It’s too simple. Click on the “Generate app” button, add a prompt for Amazon Bedrock and within seconds you have a working application that uses GenAI under the hood!

Here’s an example of a prompt that I have used:

And the outcome of that: https://partyrock.aws/u/lockhead/rpZ8z9kG5/re%3AInvent-2023-DevTools-and-CICD-Attendees-Guide

Pretty cool, isn’t it? Even tho the underlying model does not have access to the session catalog (which is a pitty), I liked the outcome that you can look at in this snapshot.

Holding to it’s promises it allows you to experience with GenAI

As the introduction of the AWS team says, PartyRock really makes Bedrock accessible and gives everyone a great possibility to “try out” how the different models behave with different prompts.
I can only encourage you to try it out and make your own experiences with it. It’s worth your time!
Being part of the AWS Community (in this case, being a Hero or a Community Builder) gave us the advantage of a few hours to try this out before the official release…and this gives me the chance to already NOW present you a few cool use cases that other builders have created with this tool 🙂

PartyRock Apps & Use cases that have made me smile or bring value

It’s amazing to see how creative AWS Community Builders and Heroes are 🙂
Here a bunch of the apps that I’ve looked at and played around with and I think are worth sharing:

NameAuthorDescription
Content MarketingAWS DevTools Hero Thorsten HögerThis simple app allows you to use GenAI to create tailored Marketing Materials and post snippets for social media base on a Service Input.
Quiz GeneratorAWS Community Builder Dixit R JainGenerates questions for a specific topic that you can use in a quiz – the original idea is from Dixit, I’ve linked my version which is “remixed” and adds a language option
Event Countdown AppAWS DevTools Hero Johannes KochA simple event countdown calculator that shows you the time until a specific event starts.
Gameday Team Name GeneratorAWS Community Hero Markus OstertagGenerate your teamname for your next AWS Gameday – maybe at re:Invent in Las Vegas?
Generate Google Calendar and iCalAWS DevTools Hero Johannes KochGenerates an iCal and a Google Calendar link for your next event
Social Media AssistantAWS Community Builder Matteo DepascaleGenerates Tweets and LinkedIn posts to make your life easier
re:Invent Steps & Calory calculatorAWS DevTools Hero Johannes KochHelps you to calculate steps and calories burned when walking at re:Invent 2023 (hint: DON’T DO IT!) 🙂
TrumpifierAWS Community Builder Damien GallagherNo comments, but it was funny.
Conversation StarterAWS Community Hero Markus OstertagYou have challenges starting a conversation with your PeerTalk expert? This app will help you find the right entry words.
AWS Certification RecommenderAWS Community Builder Ganesh SwaminathanRecommends you a specific certification to do given your personal background
Bill & Ted Quote GeneratorAWS Community Builder Jenn BergstromGenerates quote from Bill & Ted based on your mood and situation
AirlineDestinationAdvisorAWS Community Hero Anders BjørnestadGet travel tips and tricks for a destination

Did you find an exciting app that I should include? Reach out to me and let me know 🙂

Where do we take it from here? How PartyRock helps and what I would love to get

PartyRock is a great starting point for experimenting with GenAI!

To take this to the next level there is a few things I’d love to get:

  1. Make your PartyRock app “yours”
    • Deploy to my AWS Account Button
    • Export to CDK / IaC Option
    • Export to CodeCatalyst Project
  2. Additional UI options
    • Radio Boxes, ComboBoxes
    • Make the applications “user aware”
    • Other generation options than text/image

Especially the first point would help developers to take action after creating the app – you could directly use the generated app and use this on your own AWS Account and this will help you understand how Bedrock fits into your existing AWS architecture.

What do you think of it? I’d love to hear your feedback and thoughts!

Time until re:Invent 2023:

Views: 618

A real project with CodeCatalyst – Our hackathon gave us a good insight into what works and what doesn’t

As Matt already wrote in his project introduction post we have been using Amazon CodeCatalyst to handle all of our project activities. This gives a very good indication into where CodeCatalyst is already “ready for prime time” and where further adjustments need to be made in order to develop the service into a fully fledge, all-or-nothing, integrated DevOps service.
The one pushing for using CodeCatalyst was myself, as I wanted to try out the new Amazon service that recently went “GA” after being announced at re:invent 2023 in a bigger team and in a real project. It was great fun – and we learned a lot, too!
Let’s look at some details.

How we planned our tasks using “issues”

Our journey with CodeCatalyst started with a – very agile and intense – planning session using the “issues” component of CodeCatalyst. Given the nature of an hackathon, we created roughly the first ten issues in the system. Three of us were “new users” for CodeCatalyst, but they were able to quickly interact with the issues component – by default, this is a simple Kanban board and has a well structured user interface. One thing we directly missed was the ability to copy/paste images into comments within the issues…and to attach files to the issues themselves. Linking issues to one another is also only hardly possible right now.
We used a simple workflow and did a regular planning (twice per week) and created/updated/closed more than 50 issues (of partly very small scope) throught the course of less than four weeks. Overall, the available functionalities were good enough for our project and supported our project.

Collaborating source code – working with Git repositories and pull requests

The four of us are (Community) Builders from the bottom and all we need to be happy is a Git repository that we can push code to 🙂
Of course this is not currect, but CodeCatalyst allows you to host you own, private Git repositories. You can have multiple repositories per project. In our case, we had decided to go with a mono-repo approach and so we strickly worked in a single repository. Working with the source code repository was “just fine” and working as expected. We also decided to merge as early and as often as possible to our “main” branch. The team however struggled in working with setting up Pull Requests and working in the UI when reviewing pull requests.
The most problematic things: UI response time, working with comments on pull requests (e.g. threaded comments) and creating pull requests. The whole UI does not yet feel as “structured” as in other competitors (e.g. Github) – examples: PR title/description proposals when creating a PR, link in terminal/commandline that allows your to create a PR.
Another thing we missed is “git blame” online – not to blame each other but to understand what changed and broke our “hacked” “hackathon” source code 🙂
Cool: the markdown rendering for files named *.md. Missing here: including local images is not possible.

CodeCatalyst usage increased a lot in this project!

Working with Workflows – Github actions for the rescue!

We started off building our workflows with natively available actions in Amazon CodeCatalyst – using the “cdk deploy” action to deploy out infrastructure and the “s3 deploy” action to deploy our front end – but quickly shifted away from that and switched over to using Github Actions within the workflow to allow building our Flutter application as a CDK asset. As our Continuous Integration pieces matured within our npm implementation, there was less reason to go back to the native actions as the CI part was handled through npm.
Our current workflow and CI/CD pipeline:



The main preference here is to allow a continuous deployment to our production environment. This goes through an intial “sandbox” deployment, promotes to a “test” environment and then deploys to “production”. The pipeline is prepared to execute integration tests, but they are not yet implemented – which is the nature of a _hack_athon 🙂
This would definately one of the next things to change – adding real integration tests and also adding some security verifications.

Our workflow is, as I mentioned, pretty basic – here an excerpt of it until the deployment to our sandbox environment:

BuildAndTestCDKApp:
    Identifier: aws/github-actions-runner@v1
    Compute:
      Type: EC2
    Inputs:
      Sources:
        - WorkflowSource
      Variables:
        - Name: AWS_REGION
          Value: us-east-1
        - Name: NODE_ENV
          Value: sandbox
    Outputs:
      AutoDiscoverReports:
        Enabled: true
        ReportNamePrefix: rpt
        IncludePaths:
          - ./coverage/lcov.info
    Configuration:
      Steps:
        - name: Flutter Build Web
          uses: subosito/flutter-action@v2.8.0
          with:
            channel: stable
        - name: Java Install
          uses: actions/setup-java@v2
          with:
            distribution: zulu
            java-version: "11"
        - name: Setup Android SDK
          uses: android-actions/setup-android@v2
        - uses: actions/setup-node@v3
          with:
            node-version: 16
        - run: npm ci
        - run: npm run
        - run: npm run synth
    Environment:
      Connections:
        - Role: CodeCatalystWorkflowDevelopmentRole-AWSCommunityBuilders
          Name: community-builders-sandbox
      Name: speakers_sandbox
  DeploySandboxCDKApp:
    DependsOn:
      - BuildAndTestCDKApp
    Actions:
      DeployBase:
        Identifier: aws/github-actions-runner@v1
        Compute:
          Type: EC2
        Inputs:
          Sources:
            - WorkflowSource
          Variables:
            - Name: AWS_REGION
              Value: us-east-1
            - Name: NODE_ENV
              Value: sandbox
        Configuration:
          Steps:
            - name: Flutter Build Web
              uses: subosito/flutter-action@v2.8.0
              with:
                channel: stable
            - name: Java Install
              uses: actions/setup-java@v2
              with:
                distribution: zulu
                java-version: "11"
            - name: Setup Android SDK
              uses: android-actions/setup-android@v2
            - uses: actions/setup-node@v3
              with:
                node-version: 16
            - run: npm ci
            - run: npm run deploy
        Environment:
          Connections:
            - Role: CodeCatalystWorkflowDevelopmentRole-AWSCommunityBuilders
              Name: community-builders-sandbox
          Name: speakers_sandbox
  IntegrationTestSandbox:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Configuration:
      Steps:
        - Run: ls -al
    Compute:
      Type: Lambda
    DependsOn:
      - DeploySandboxCDKApp

As you can see, a lot of the CD (and also the “deployment”) runs within the corresponding task definitions in package.json.

No “Open Source” or “Readonly” view

We would really love to open-source our whole projet, to give you the possibility to get involved and learn from what we build, but unfortunately Amazon CodeCatalyst does currrently not offer an Open Source or a “Readonly” view and forces you to create an account for CodeCatalyst – these are things that make it difficult for us as we cannot link to our sources.
If you are an AWS Community Builder, please reach out to either of us in Slack and we can give you access to the project.

What did we really miss in the workflow component

  • manual approvals before promotion to our “production” environment
  • Notifications from workflows – could have been done through this implementation but we did not take time to implement that
  • Integration of actions deployed through “npm – cdk deploy” with making the changes in infrastructure visible
  • importing existing pipelines or render CDK pipelines to CodeCatalyst workflows The “killer feature” of the workflows, at least as far as we see it, is the possibility to use Github actions as part of the workflow. With only minor adjustmends an existing Github workflow can be transformed to a CodeCatalyst workflow. If either of you know about an existing tool that converts existing Github workflows to CodeCatalyst workflows in an automated way – please let me know.

The hidden gem: Development Environments

One of the “hidden gems” of CodeCatalyst are the Dev Environments – also known as Development Enviroments. Essentially, its a “service in the service” similar to Gitpod where you can host your IDE or the environment that you develop on. In our project we all didn’t use it as the architecture was small and we only had this single project. Also, we were developing mainly on our main machines without switching between machines too often. That’s why we all did not consider really using the DevEnvironments for this project. Last but not least, Flutter is not natively supported on the Dev Environments which would have forced us to prepare a devfile that includes the Flutter dependencies – and thats something that would have taken too much time out of the project.
Still, the De Environments within CodeCatalyst are hidden gems that I believe should get more traction than they are getting until now.

A summary of what we think works and what doesn’t – and our biggest wishes

For a hackathon like this (3-6 weeks working time) CodeCatalyst seems to be a very good choice for a project that starts with a “new” or “fresh” codebase. The tool has most of the minimal requirements implemented to allow simple project management and good integration with AWS.
It also allowed us to quickly get started and to collaborate on our sources.

We really missed workflow notifications as well as manual approvals in the workflows.
As the tool matures I personally hope that more and more AWS internal teams will be using CodeCatalyst to develop, plan and deploy out their internal AWS services and with that the “flow” in the User Interface will definately be improved, too – as today there are some hickups in a real developers workflow.

And, last but not least – we would love to share this project with you, but given CodeCatalyst does not allow to “Open Source” a project, that’s not possible 🙂

Maybe something for the next hackathon?

If you would like to get involved in this project and help us to shape and build our AWS Speakers directory to make it easier for AWS User Group Leaders to find speakers – please reach out to either of us four about collaborating!

Who to reach out to?
DanielleMattJulian or myself.

Views: 1693

Amplify SDK for Flutter – Developer experience and challenges in a hackathon

Background of this post

This blog post is part of a a series of posts that explains details and technical challenges that we (Danielle, Matt, Julian and myself) faced during an hackathon that was focused on the Transformers Huggingface tools – please read Matt’s article for additional information and details.
In this post we are focusing on the experiences we made in the project using the Amplify SDK for Flutter.

Project Setup – architecture dependencies for project set up

Initially our project is targetting to be available to all AWS User Group Leaders on the web from any browser. As we mature the project, at least I am hoping to be able to also publish this application as an Android and iOS application using cross platform capabilities.
Here is a birds eyes view to our UI architecture:


Why did we choose Flutter as UI?

Flutter is a trending cross-platform development toolkit, altough it is mainly sponsored by Google there is a vibrant open source community. I personally have had some exposure to Flutter in the past and I liked the developer experience and the short iteration cycles. Also the AWS Amplify Team has been investing into making their SDK available to developers, so we thought this is a good possibility to try out our use case and implement the Web app in Flutter using the Amplify SDK for Flutter to connect to the backend resources which we were planning to implement using the AWS CDK. With this approach, we also want to draw some attention on the fact that the Amplify SDKs can be used without an Amplify owned backend. This is cool and opens up a lot of possibilities – but also challenges as we will see later. I convinced Danielle, Matt and Julian to also use Flutter for our frontend. They also saw this as a good opportunity to learn a bit of Flutter by themselves.

Amplify SDK for Flutter – Developer experience and challenges

The Amplify SDK for Flutter recently announced the “GA” for Web and Desktop which is a milestone for the team to reach. As we outlined in Matt’s blog post, we are using Typescript in the backend. Amplify Flutter has native support for AppSync/GraphQL – we needed to connect the Flutter App to existing AppSync endpoints.
The AppSync schema however was written manually. No, we needed to use “amplify codegen” to generate the Dart models for the GraphQL types – but we also needed to write a type model in Typescript to be able to work with the same objects on the backend.
This turned out to be more difficult than expected: the [amplify codegen](https://docs.amplify.aws/cli/graphql/client-code-generation/#shared-schema-modified-elsewhere-eg-console-or-team-workflows) functionality is available for Flutter, too, but it was difficult to get this to work.
We ended up creating a new Flutter application using amplify init and then copy/pasted our schema(s) into the expected location. Then we needed to manually copy the generated models into our project.
Oh, I neqarly forgot: When using amplify codegen you need to ensure that your schema is anotated correctly, types e.g. need to have an @model annotation – but if you have this annotation in the schema when trying to deploy to AppSync, that deployment failed…so we needed to also manually adjust the schema before we were able to execute amplify codegen.

Using the Amplify SDK for Flutter

In our use case, all of the backend infrastructure is created using the AWS CDK. We are not using Amplify to create backend resources – and this use case has – until recently – not really been promoted by the Amplify team. Thanks to one of my last blog posts around the same topic, this new documentation page has been added which simplifies the setup of your amplifyconfiguration.dart – but we were missing the same documentation for the “Authentication” library. One again, we workarounded this problem by using a temporary Amplify Flutter project. That allowed us to copy/paste the configuration and adjust it to our needs.

Environment aware connections

In our current setup, we have at least three environments to test and promote our application. In order to be able to execute and test the Flutter application on all environments without code changes, we needed to make the maplifyconfiguration.dart environment aware:

class EnvironmentConfig {
    static const WEB_URL = String.fromEnvironment('WEB_URL');
    static const API_URL = String.fromEnvironment('API_URL');
    static const CLIENT_ID = String.fromEnvironment('CLIENT_ID');
    static const POOL_ID = String.fromEnvironment('POOL_ID');
}

These environment variables are then used in our configuration object:

const amplifyconfig = """{
"UserAgent": "aws-amplify-cli/2.0",
    "Version": "1.0",
    "api": {
        "plugins": {
            "awsAPIPlugin": {
                "frontend": {
                    "endpointType": "GraphQL",
                    "endpoint": "${EnvironmentConfig.API_URL}",
                    "region": "us-east-1",
                    "authorizationType": "AMAZON_COGNITO_USER_POOLS"
                }
            }
        }
    },
    "auth": {
        "plugins": {
            "awsCognitoAuthPlugin": {
                "UserAgent": "aws-amplify-cli/0.1.0",
                "Version": "0.1.0",
                "IdentityManager": {
                    "Default": {}
                },
                "CognitoUserPool": {
                    "Default": {
                        "PoolId": "${EnvironmentConfig.POOL_ID}",
                        "AppClientId": "${EnvironmentConfig.CLIENT_ID}",
                        "Region": "us-east-1"
                    }
                },
                "Auth": {
                    "Default": {
                        "authenticationFlowType": "USER_PASSWORD_AUTH",
                        "socialProviders": [],
                        "usernameAttributes": [
                            "EMAIL"
                        ],
                        "signupAttributes": [
                            "BIRTHDATE",
                            "EMAIL",
                            "FAMILY_NAME",
                            "NAME",
                            "NICKNAME",
                            "PREFERRED_USERNAME",
                            "WEBSITE",
                            "ZONEINFO"
                        ],
                        "passwordProtectionSettings": {
                            "passwordPolicyMinLength": 8,
                            "passwordPolicyCharacters": []
                        },
                        "mfaConfiguration": "OFF",
                        "mfaTypes": [
                            "SMS"
                        ],
                        "verificationMechanisms": [
                            "EMAIL"
                        ]
                    }
                }
            }
        }
    }
}""";

Within our CI/CD workflow we then pass the correct values for the variables and during the flutter build web they are backed into the application.

After solving these challenges, using the Amplify Flutter library worked without noteworthy problems or hickups.

Using the Amplify Flutter Library to authenticate a user with Cognito

The documentation for Amplify Flutter is really good and we decided to also use the Authenticator Widget – at the end, this project was born through a hackathon – and we did not have much time to implement the authentication flow ourselves.

In our main.dart we needed to include the Amplify configuration:

Future<void> _configureAmplify() async {
  final api = AmplifyAPI(modelProvider: ModelProvider.instance);
  await Amplify.addPlugin(api);

  // Add any Amplify plugins you want to use
  final authPlugin = AmplifyAuthCognito();
  await Amplify.addPlugin(authPlugin);

  try {
    await Amplify.configure(amplifyconfig);
  } on AmplifyAlreadyConfiguredException {
    safePrint(
        'Tried to reconfigure Amplify; this can occur when your app restarts on Android.');
  }
}

and then we only needed to adjust the build method to distinguish betwen AuthenticatedViews and “normal” views:

Widget build(BuildContext context) {


   final GoRouter router = GoRouter(
      routes: <RouteBase>[
        GoRoute(
          path: '/',
          builder: (BuildContext context, GoRouterState state) {
            return HomeScreen();
          },
        ),
        GoRoute(
          path: '/profile',
          builder: (BuildContext context, GoRouterState state) {
            return AuthenticatedView(
                child: ProfileScreen(),
            );
          },
        ),
        GoRoute(
          path: '/imprint',
          builder: (BuildContext context, GoRouterState state) {
            return ImprintScreen();
          },
        ),
        GoRoute(
          path: '/privacy',
          builder: (BuildContext context, GoRouterState state) {
            return PrivacyScreen();
          },

        ),
        GoRoute(
            path: '/in',
            builder: (BuildContext context, GoRouterState state) {
              return AuthenticatedView(
                  child: LoggedInHomepage(title: 'AWS Speakers Directory'));
            },
        ),
      ],
    );

    return Authenticator(
      initialStep: AuthenticatorStep.signIn,
      signUpForm: SignUpForm.custom(
        fields: [
          SignUpFormField.username(),
          SignUpFormField.password(),
          SignUpFormField.passwordConfirmation(),
          SignUpFormField.email(),
          SignUpFormField.custom(
              title: "First name",
              attributeKey: CognitoUserAttributeKey.givenName),
          SignUpFormField.custom(
              title: "Last name",
              attributeKey: CognitoUserAttributeKey.familyName),
          SignUpFormField.custom(
              title: "City",
              attributeKey: CognitoUserAttributeKey.custom("city")),
          SignUpFormField.custom(
              title: "Country",
              attributeKey: CognitoUserAttributeKey.custom("country")),
          SignUpFormField.birthdate(),
          SignUpFormField.gender(),
        ],
      ),
      child: MaterialApp.router(
        title: 'AWS Speakers Directory',
        routerConfig: router,
      ),
    );
  }

We would have loved to open source our code, but unfortunately that option is currently not available in Amazon CodeCatalst.
If you’re an experienced Flutter developer and don’t like the code you see above – that’s fine, the four of us have been learning Flutter iwth this project – approach us and tell us how to do this better! 😉

Using the Amplify Flutter Library to execute AppSync / GraphQL queries

Accessing AppSync was also really easy by following the Amplify Flutter documentation. Here is our code for retrieving events from the backend:

Future<List<EventRow?>> _listEvents() async {
    try {
      String graphQLDocument = '''query GetEvents {
      listEvents {
        pk
        sk
        title
        description
        tags
        length
      }
    }''';

      var operation = Amplify.API
          .query(request: GraphQLRequest<String>(document: graphQLDocument));
      List<EventRow> events = [];
      var response = await operation.response;
      var data = response.data;
      if (data != null) {
        Map<String, dynamic> userMap = jsonDecode(data);
        print('Query result: ' + data.toString());
        List<dynamic> matches =
            userMap["listEvents"] != null ? userMap["listEvents"] : [];
        matches.forEach((element) {
          if (element != null) {
            if (element["id"] == null) {
              element["id"] = "rnd-id";
            }
            var event = Event.fromJson(element);
            events.add(EventRow(event, context));
          }
        });
        return events;
      }
    } on ApiException catch (e) {
      print('Query failed: $e');
    }

    return <EventRow?>[];
  }

This uses the previously generated models. All of the queries are authenticated using the Cognito Authentication – and this is the identity that we can also use in the backend to authorize access.
The Amplify documentation is pretty good, hence I will not be adding more details into this post.
Please ask if you have any specific questions.

Wishes for the Amplify SDK for Flutter

We already mentioned most of them: better amplify codegen support (including the option to generate both Flutter and Typescript models at the same time), better documentation or support for setting up the Amplify configuration completely without an Amplify backend.
Another thing we did not talk about here but Julian mentions in his blog post are the AppSync [merged APIs] which are currently not supported – thus we needed to execute amplify codegen for each of our microservices schemas and then for the merged APi – and then manually needed to bring the models into a usable structure (e.g. copy/pasting from the different ModelProviders).

What’s next for our project

So, what’s next? After the hackathon we are thinking about making this a “kind of” OpenSource, collaborative project. Here we are looking for contributors – please contact us if you are interested.
Besides that, Matt covers a godd bunch of roadmap items already – and we definately need someone with more Flutter experience to review our UI code to e.g. introduce the BloC pattern for cleaner programming, adding some styling and to expand existing functionalities.
And we still need to play the “cross-platform” card – we need a build step to generate iOS and Android versions of our application and need to make sure that the apps land in the Appstore(s)!
Please reach out to us if you would like to get involved!

Who to reach out to?
DanielleMattJulian or myself.

Views: 446

Amazon Codecatalyst reaches “GA” status and becomes available for general use

The new service announced by Amazon in Las Vegas at re:Invent 2022 which is an integrated DevOps service to empower development teams to develop and deliver software faster finally reaches the “general availability” status. As I have previously outlined, this achievement is very important for Amazon and the CodeCatalyst team. Congratulations to the team for reaching this goal, which I can imagine is not an easy step for this product. The tool touches a lot of very sensitive parts of a software project and I can imagine the security standards being really high. 

A hugh achievement – thank you everyone in the team for investing into CodeCatalyst and for listening as closely to the customer feedback as you are!

What changes did get implemented for GA?

As part of the GA release we see a lot of minor improvements in the User Interface and color changes. In the last weeks, we have seen a few “bigger” changes – like the possibility to use Dev Environments for Github based projects. We also got “graviton based” execution environments for CI/CD workflows which, according to AWS, should reduce our costs.

It is still hard to track down all of the changes in CodeCatalyst, as there is – to my knowledge – no public or semi-public roadmap. This is one of the things that I’d love to see, as for an integrated service that is at the core of the Developer Experience for teams, any minor change can either improve or destroy the “usage experience”. If you as a team invest into adoption a new tool like CodeCatalyst, they will need to know how changes in workflow, features or user interface can influence their day-to-day activities. Let’s see, maybe the team can share “something” like a “changelog” with us (or even an RFC process like Amplify or AppSync)?

Reached “GA” – so who can start using it now?

As of today CodeCatalyst is only availble in US regions and this means that it can be adopted mainly by US enterprise customers. CodeCatalyst already gives you the possibility to set up different Spaces for your account and within a space you can manage multiple projects. So in theory, CodeCatalyst is “ready to be used” by everyone. 

Practically speaking, it is easier to adapt the service for new projects than for existing projects , as there is no real “import” functionality. Yes, you can integrate existing Github projects, but that only integrates the source code. Unfortunately that does not make all of the “cool” things available right from the start of integrating the source: existing workflows (CI/CD pipelines) are lost and need to be re-build, issues/tickets are not imported into CodeCatalyst (tho they can be made available through the JIRA integration). 

I have been regularly using CodeCatalyst (both for imported and “new” projects) – and I really think that the tool already works very well. 

The “killer feature” that I see for new projects are the “blue prints” which essentially get you started within minutes, e.g. to deploy a SPA application, or to have a “true” CI/CD pipeline for a full stack application following the DPRA

Right now I would recommend using CodeCatalyst for any new project that you start to start building out your workflows and best practices.

So what do I still need to recommend CodeCatalyst for existing projects?

There are a few things that I have already been writing about:

  • “Import” of existing CI/CD workflows (e.g. Github actions, CDK Pipelines or CodePipelines)
  • Fully import projects
    • existing issues from Github or JIRA
    • Git-based projects including the history
  • Tighter security settings and permissions
    • Fine granular roles to allow or forbid access to specific parts of a project
    • Options to allow or forbid execution of workflows (or to deployments)
  • Additional workflow options
    • Manual approvals are very high on my wish list
    • Integration of other AWS services natively

A question for the readers: What do YOU think that you need to adopt CodeCatalyst?

A big question for the CodeCatalyst team – HOW MANY AWS TEAMS ARE USING CODECATALYST FOR PRODUCTION DEPLOYMENTS TODAY?

Where do I see the potential for CodeCatalyst?

CodeCatalyst is a big bet by AWS. There is a big potential that can really improve the life of development teams and these are the main things that I believe that can out-grow other existing solutions:

  • Integration of AWS Services / deployments metrics
    • the true integration with AWS APIs
    • Integration into “post-deployment” verifications (e.g. auto roll-back after failed CloudWatch metrics)
  • “At-hand” developer support to improve efficiency
    • with CodeWhisperer (who recently reached GA) AWS already aims to support developers during the development phase, but with CodeCatalyst AWS can take this to the next level:
    • AI support during Pull Request Reviews (or automated approvals for PRs – e.g. by including CodeGuru, etc., automated merges, etc.)
    • AI support during workflow executions (when to approve, when to deloy, when to promote, etc.)
    • With improvement proposals for workflows if the “AI model” recognizes patterns (in issue workflows or CI/CD workflows)
  • With automated improvements for existing projects based on blue prints
    • Best practices change – and so blue prints change – and if the CodeCatalyst team can automatically apply them for existing projects, customers will benefit from it

And last but not least:

I trully believe that every software project should start with a CI/CD pipeline – and with the Blue Prints including the CI/CD workflow that follows DPRA and other AWS best practices, we can trully make this possible: Empower developers to deliver their software projects in minutes right after starting their project.

Do you see the potential in CodeCatalyst? If you do not see any potential in the tool – why not?

Views: 1723

Pipeline strategies for a mono-repo – experiences with our Football Match Center projects in CodeCatalyst

Both Christian and I have been writing about our “Football Match Center” project – and as part of this project we obviously also needed a CI/CD (Continuous Integration and Continuous Deployment) pipeline. Our aim was to be able to integrate changes that we do regularly and see commits to the main branch being directly and automatically deployed to our environments.

I will first try to define some pre-requisites and then talk about learnings and experiences. 

What is a mono-repo

A mono-repo is an abbreviation of a “mono repository” which I understand as being a single repository, where different microservices or components are stored and saved in the same git repository. This can be various different services, infrastructure or user interface components or backend services.

A mono-repo has special requirements when building the CI/CD pipeline.

Expectations for our CI/CD pipeline

For our CI/CD pipeline we wanted to be able push changes to production quickly and be able to iterate fast. We wanted to achieve 100% automation for everything required for our project. As we have been writing, we completely develop this project using Amazon CodeCatalyst and thus the pipeline also should be build using the Workflows in CodeCatalyst.

Going forward we want to ensure that the pipeline also includes all CI/CD best practices as well as security scans and automated integration or end to end tests.

How to structure your pipelines

In this article we will purely focus on the CI/CD pipeline for your “main” or “trunk” branch – the production branch that will be used to deploy your software or product to the production environment.

We will not consider pipelines that should be executed on feature branches or on pull request creation.

The “one-pipeline-to-rule-them-all” approach

In this approach all services are deployed within the same pipeline. This means that there is only a single pipeline for the “main” branch. All services that are independed rom each other can deployed in parallel, services that have a dependency need to be deployed one after another. Dependencies or information from one to another service can be pushed through the pipeline using environment variables.

This can lead to longer deployment/execution timelines but ensures that “one commit” to this “main” branch is always deployed completely after a commit. If tests are included in the pipeline, they will need to cover all aspects of the application.

The “context-specific” or “component-specific” approach

Different components or contextes get a different pipeline – which means that e.g. the backend-services are deployed in one pipeline and the frontend-services in a different pipeline. 

In this approach, you automate the deployments for components and need to ensure that, if there are dependencies between the components, the pipeline verifies the dependencies. If one component requires information from another one you need to pass these dependencies using other options.

This can lead to faster iteration cycles for specific components but increases the complexity of the pipeline dependencies. You can also do not directly see if a specific commit has been deployed for all components or not.

The “one-pipeline-for-each-service” approach

This is the most decoupled option for building a CI/CD pipeline. Each service (lambda function, backend, microservice) gets its own pipeline. For each service, you can implement service specific steps as part of the pipeline. 

One of the main requirement for this is that the services are fully decoupled, otherwise managing dependencies can get very difficult. However, this allows a very fast iteration and development cycle for each microservice as the pipeline execution for each service is usually very fast.

The pipeline needs to verify the dependencies for each service as it executes the deployment.

Football Match Center – our experiences with building our CI/CD pipeline in Amazon CodeCatalyst

For our project we decided to start with a “mono-repo” – in our case today, we have a CDK application (written in Typescript) that describes the required infrastructure and includes Lambda functions (where required) and a user interface which is written in Flutter.

From a deployment perspective, the CDK application needs to be deployed on AWS and the Flutter application then needs to be deployed on a S3 bucket to serve as a Single Page Application (SPA) behind Cloudfront. Obviously this deployment/upload has the pre-requisite of the S3 bucket to be already available.

How we started

We started, very classic, with the “one-pipeline-to-rule-them-all” approach. We had one single pipeline that was used to deploy all services that are part of the infrastructure.

This pipeline started with “cdk synth” using the “CDK deploy” action in CodeCatalyst and then had other steps that depended on the first one – to executing the “flutter build” and later the “UI deploy” (using the S3 deploy action).

In this first version, the CDK deploy step had variables/output with the name of the S3 bucket and the CloudFront distribution ID passing it it to the next step where the output of “flutter build” was then uploaded and the CloudFront distribution invalidation request was triggered.

In this approach a commit to the “main” branch always triggered the same pipeline and this pipeline deployed the complete application.

We also used only natively available CodeCatalyst actions for deployment – “cdk deploy” and “build”. For the Flutter action we used a Github Action for flutter.

Experiences and pipeline adjustments

With this approach we had the problem that the Flutter build step took ~8 minutes and blocked a new iteration of changes in the CDK application or the lambda function. Thus, this slowed down our development cycle.

In addition to that we found out that there was no possibility to influence the CDK version with the CDK deploy action – but we wanted to be able to use the version defined in our Projen project – to be able to deploy to development environments from our local with the same version as from the CI/CD pipeline.

Both of these findings and experiences brought us to implement some changes to the pipeline:

  • We separated the UI build from the CDK build
  • We moved away from using “cdk deploy” and replaced it with a “build” step – to be able to trigger “projen” as part of the pipeline

So now we have two pipelines:

  1. CDK deployment
    • Triggered on changes to the “cdk-app/*” folder
      • Executing CDK synth, build and deploy steps – but not using the “cdk deploy” action but a normal build step instead
      • We adjusted the CDK app to include Cloudformation exports that exports the S3 bucket name and the Cloudfront distribution ID
  2. Ui deployment
    • Triggered on changes to the “ui/*” folder
    • Reads the values for the S3 bucket and the CloudFront distribution ID from the CloudFormation exports using the AWS cli
    • Executing the Flutter build steps and the S3 deploy action

These changes reduced in faster iterations for the development cycle of the CDK app and allowed decoupling the backend from the UI part. We were also able to fix the CDK version to the version we have selected in Projen.

In our project we have chosen the “context-specific” approach for the pipeline.

My recommendations for building CI/CD pipelines for a mono-repo

Our CI/CD pipeline is not perfect yet and we’re yet to add some important things to our pipeline.

From the experiences we have made I am still not convinced that our “context-spefic” approach is the right path.

As of writing this post in early April 2023 I’m inclined to move towards a model where we combine the “context specific” and the “one-pipeline-to-rule-them-all” approach: context-specific for “lower”, non production environments and then a single pipeline that does the promotion to our production environment.

Today we do not yet have a production environment, so we did not answer that question yet 🙂

How do you solve this challenge around building CI/CD pipelines for mono-repos?

Views: 687

What can we expect for the General Availability (GA) of Amazon CodeCatalyst?

At re:Invent 2022, as usualy, different new AWS Services or functionalities have been announced in Preview. Now, at the beginning of April 2023, a few of them have already reached the “General Availability” (GA) status – Application Composer (in early march), Latice (in late march). My favourite new service, Amazon CodeCatalyst, has not yet reached this goal – but I have a feeling that now is the right time to think about what and when we can expect this status.

You wonder what CodeCatalyst is? Watch this video on my YouTube channel or read my two initial posts about it.

Why is reaching the “GA” milestone so important?

Before starting with my assumptions on what we can expect for GA, lets clarify why reaching this milestone is so important. Being “in Preview” can mean a lot of different things. In a lot of organizations this usually translate to “limited availability”, a service not being available in all regions or not being reliable or scalable. For other organizations, it means that specific aspects of the product can be immature or not reliable. It can also mean that bigger API change are yet to be implemented or missing security guardrails. 

In general, this can be seen as a “beta” offering which is not appropiate to use for productive workloads.

Because of these reasons and maybe others, a lot of organizations (especially US based) do not allow using or adapting services that are in “Preview”.

For all of my experiences, tests, videos and projects I was so far able to only be on the free tier. And I assume that this will also be the truth for most of my readers: You can get a long way using the Free Tier that Amazon CodeCatalyst offers today.

So thats another big reason for AWS to push this service out of “Preview”: It gives organizations that are forbidden to use the service in “Preview” the possibility to start using and adopting the service – and with that Amazon can start earning money with the service which unil now might be difficult.

And as we know, AWS tries to “work backwards” from client requirements and the early usage of CodeCatalyst will drive further investments into the service.

What to expect for GA of CodeCatalyst?

Simple: Nothing big – most probably only regional rollout.

I do personally not expect any major new features for the service as the team has been constantely releasing new features and functionalities to the service on a regular cadance. There was simply not more time to work on bigger features while preparing the “General Availability” (GA).

What the CodeCatalyst already has delivered until today…

Let’s look at what has been added to CodeCatalyst since its official release in december 2022:

  • Additional Reporting auto-discovery
  • Change Tracking – the possibility to see which changes have been deployed to a certain environment
  • Additional Workflow native actions and improvements, E.g.
    • a problem with the CDK action to be able to define the “workpath” of a CDK app
    • Additional native actions
  • Linked issues to Pull Requests – you are now able to link issues to a pull request
  • UX improvements
    • Log files wider accessible in UI – at the beginning you where not able to make the log view larger, now this is possible
    • Page title adjustments
  • New Blue Prints (like the “Textract” one)
  • Development environments for Github based projects

This is not a complete list, but the things that I personally noticed and that I liked to see.

So…when is “the date”?

Hard to guess, but I would expect “soon”. Ideally right before a month starts, which will make the billing cycle easier 🙂 

So I would guess “end of april” which would bring the service right in time for the Berlin Summit (3rd of May).

Next steps for CodeCatalyst

In my last posts I have already been communicating my thoughts and features that I would love to see. But what will AWS implement?

Given reaching the “GA” status opens the way to “enterprise clients” I would expect that one of the first features will be Single-Sign-On functionalities, maybe with an integration to Okta, Ping, Azure Active Directory or other already existing IdPs.

In addition to that I believe that the User Interface needs to get some tweaks to streamline the navigation and workflow – that’s something that I personally experience every day: not knowing when and where to click to get on the rigth place. Also I think that additional service integrations will be added – e.g. StepFunctions or SNS, maybe SQS – see also my post about sending notifications from workflows.

And then there is one last thing which has been getting only limited attention so far: APIs and CLI integrations that can be used – so I would expect a major update there.

I’m really looking forward to see CodeCatalyst reaching GA – I’ve had various conversations with the team in the last months and I know that they have a true vision to make CodeCatalyst successfull as a trully AWS integrated and fully functional DevOps tool.

Are there features you are missing? Please let me know and I will forward them to the team.

Views: 771