A few weeks ago AWS CodeCommit became a deprecated service on AWS. This means, customers cannot create new repositories anymore – refer to this announcement for all details: Blog for CodeCommit
There are obviously a lot of alternatives to CodeCommit (Github, Gitlab, …) but if you need a “self-hosted” Git repository in your own AWS account this can become a little bit harder to provision.
If you’re looking for a “self-hosted” Git server, a bunch of tools come up:
…and obviously also others or just a “git” server running on EC2
As I wanted to be able to deploy something that works “out of the box” I looked at how to provision one of these alternatives on my own AWS account.
Gitness
Gitness is an open source development platform packed with the power of code hosting and automated DevOps pipelines. It’s biggest mantainer is Harness. It includes “way more” than just Git – eg Gitspaces, Pipelines, etc.
Deployment of Gitness
If you want to deploy Gitness, the docs point you at running it locally using Docker, or by deploying it on EC2 or k8s.
My aim was to “only” make the “Git” component available and because of that I’ve chosen Amazon ECS with EFS as storage to provision Gitness.
Open Source Code for the deployment of Gitness
I’ve set up a project on Github where all of the code examples are available for you to look at:
In this post you are going to get to know the new service that AWS announced at re:Invent 2022 in Las Vegas. With this new service AWS brings the Code* tools (CodeStar, CodeBuild, CodePipelines, etc.) into one user interface and opens up the usage to new audiences.
Amazon CodeCatalyst is a service that helps you to manage your project code in a central place and allows integrating it with existing CI/CD tools. It also simplifies the cross account functionalities of CI/CD best practices. With the market place functionality it opens up the tool for further expansion through third party contributors.
Functionality highlights
Organizations and projects
CodeCatalyst allows the set up of an organization and projects within that organization. This allows a structured set up of your projects and allows you to segregate access. What I have not been able to verify is if existing organizations of AWS Organizations are automatically imported and available in the service. This would be beneficial for larger organizations that want to get started with CI/CD on AWS.
Integration with your “builder ID”
Amazon CodeCatalyst is integrated with AWS your builder id – and that means that you can have a unique, personal account. This account is not tied or attached to an existing AWS account. The integration with the AWS accounts it done through cross account permissions that are created automatically for you. This concept has various benefits:
You (as a developer) can have access to the required CI/CD structure and code management without the requirement to have an IAM user or access to the AWS console to the specific AWS account that you are aiming to deploy to
You (as a developer) can have access to different “organizations” and “projects” within the service – which will make it easier for 3rd party teams working on AWS projects (e.g. consultancy engagements)
You can allow the CodeCatalyst workflows to only be able to deploy into certain accounts and the list of accounts that you can connect to from a specific project or organization can be modified on demand
Unified user interface and workflows
The new user interface of CodeCatalyst allows to look at the complete lifecycle of your product – form development to deployment – within the same window. You can easily set up a project, connect or set up a repository and then create a CI/CD pipeline (or workflow) for it. You will also be able to visualize the workflow of your CI/CD pipeline during creation or editing.
Integration of Github Actions
With the integration of existing Github actions and the possibility to re-use already available automation steps within your workflow AWS opens the door to re-use and migrate a lot of already existing functionality. This is going to be very beneficial and will help to drive the adoption of the service. The availability of “native” actions as part of CodeCatalyst is still limited and I assume with the Marketplace new actions will be made available quickly and the list of available actions will grow rapidly.
Project Templates – CI/CD pipelines
I believe that this is the most important functionality of the service. Allowing to quickly set up new applications, workflows or projects on the service empowers developers to get started efficiently. I’m eager to see additional templates going forward. As you can include both infrastructure and application code, but also the required workflow in a template (=that’s your deployment pipeline), I hope that with this functionality we will be able to empower developers to re-use better CI/CD pipelines that enforce a DevSecOps mindset and include all of the “best practices”. I would love to see all of the reference implementations of the “DPRA” (Deployment Pipeline Reference Implementation) implemented as templates in this service.
A good, automated CI/CD pipeline helps your developers to focus on creating business value – the more you can “re-use” or drive through standards and templates, there more efficient your teams will be.
Additional wishes
As I have not explored all areas of CodeCatalyst yet, some of my “wishes” might already be implemented and I did not yet discover them – here’s a few things that I’d love to see but haven’t found yet. Please reach out to me if this is functionality that is available somewhere and I’ve missed to find it 😊
Re-applying project templates
I would love to be able to re-apply a project template to an existing project, especially for the workflows. This allows two things: 1) if you’ve changed the workflow and it doesn’t work anymore, you can quickly go back to the working version. 2) if the template changes, you can re-apply the changes to a project without the need to access every project manually.
In combination, some times a project changes or evolves and you might want to move it from a “backend” project-template to a “full-stack” project. This can today only be achieve manually but not in an automated fashion.
Better “mobile app” support
As you’ve maybe read, I’m working on a “Flutter” application – and because of that I would really love to see a possibility in CodeCatalyst that better supports the “cross platform” app functionality. Today, the service mainly focuses on provisioning AWS infrastructure. As you build a complex project that also needs to publish a mobile application, it would be very bene be able to use the service to also publish the application itself into the different stores automatically. Of course there are other, 3rd party systems that already allow that, but if you want to use CI/CD natively on AWS this is currently not possible. I know I’ve mentioned that before, but as I have not seen the possibility to perform an iOS build on CodeCatalyst, I wanted to highlight it again.
Additional 3rd party integrations / workflows
One of the assets of the service is definitely the market place functionality that will quickly allow to integrate new 3rd party services. What I would like to see here is the support of additional repository stores besides Github (e.g. Bitbucket, Gitlab, and the corresponding “datacenter” versions), but also the possibility of a native integration of other tools that are required as part of the CI/CD pipeline – things like Codecov, SonarQube, Checkmarx, Snyk, Aquasec, etc. This might be possible through custom actions, but I am not yet clear on how fast these will be available.
Infrastructure as Code support for provisioning and setting up your CodeCatalyst environment
I’m not clear about the possibility to set up your CodeCatalyst environment through Cloudformation/CLI calls or through CDK/Terraform and I am unsure if that would be something that is required for this service. What would definitely be good to have is the possibility to set up integrations to your existing accounts and projects in an automated form – the CLI commands make that possible, but that’s not the same as directly defining it “as code”.
Native branch support for projects & workflows
CodeCatalyst allows to refer to different branches, but it would be cool to get a possibility to natively support “ephemeral” environments as part of your CI/CD process. There are two use cases that I have in mind:
Integration tests as part of the CI/CD process (workflow) that require the infrastructure to be available and provisioned
Environments to replicate/test defects or new functionality
For both of these use cases, the CI/CD tool should be able to deploy an environment from a branch – and then also automatically de-provision it based on rules. Today this is something that you would need to manually replicate within your workflows and infrastructure as code.
Overall summary and outlook
The service is overall is a great addition (or maybe a replacement?) of the existing Code* tools on AWS and it streamlines a lot of things that were previously not working perfectly on AWS. For a new service it’s a solid launch with a lot of benefits, especially for small companies or open source projects that do not have the possibility to use existing third party tools or really just want to focus on generating business value. I do not think that CodeCatalyst will be able to replace existing Jenkins installations or services for bigger enterprises right now, as there are a few things that are not yet possible.
What do you think of CodeCatalyst? Please share your thoughts and feedback with me (either on the comments, on LinkedIn or by mail.
As I’ve been sharing before, I am very fortunate this year and will be giving a DevChat at the biggest AWS conference of the world – at re:Invent 2022 in Las Vegas.
AWS offers different tools for all parts of your CI/CD lifecyle. In this post I am going to cover the set of Code* tools that are available on AWS today – and will share my thoughts about what these tools are missing.
As part of the preparation for the talk and as part of both my private project (code-name: MPAGA) and my main job @ FICO I have been researching and learning a lot about CI/CD (Continuous Integration and Continuous Deployment) – and for the private projects especially around CI/CD that natively runs on AWS. I’ve found out that not everything that these tools offer today is perfect and wanted to share some ideas on what could be improved. Where possible or applicable, I will also propose workarounds or alternatives.
We will look at a few of the tools in the order of the “product lifecycle”: 1. Code 2. Build/Test 3. Deploy 4. Release
Tools that are part of the “Code” phase
For the purpose of this post we are going to focus on tools that are natively offered by AWS as already mentioned and part of your CI/CD pipeline.
AWS CodeStar – Integration of projects
AWS CodeStar enables you to quickly develop, build, and deploy applications on AWS and provides a unified interface for your project. It provides you different templates that you can choose from to quickly start your project.
It allows you to manage your team, with permissions and integrates with your existing JIRA for issue management. It also integrates with your IDE (or with Cloud9). You can also integrate with an existing Github repository.
AWS CodeCommit – hosted Git
AWS CodeCommit is a managed service for Git (just like Bitbucket, Github, Gitlab, …. It provides a hosted “git” environment that is encrypted at rest and can be accessed using usual Git clients.
AWS CodeGuru
Amazon CodeGuru is a developer tool that provides intelligent recommendations to improve code quality and identify an application’s most expensive lines of code. Integrate CodeGuru into your existing software development workflow to automate code reviews during application development and continuously monitor application’s performance in production and provide recommendations and visual clues on how to improve code quality, application performance, and reduce overall cost.
Tools that are part of the “Build” or “Test” phase
AWS CodePipeline – Tool to manage your CI/CD pipeline
AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
AWS CodeBuild – Build tool based on containers
AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages.
AWS CodeArtifact – artifact storage
AWS CodeArtifact allows you to store artifacts using popular package managers and build tools like Maven, Gradle, npm, Yarn, Twine, pip, and NuGet.
Tools that are part of the “deploy” phase
AWS CodeDeploy
AWS CodeDeploy is a fully managed deployment service that automates software deployments to various compute services, such as Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), AWS Lambda, and your on-premises servers.
AWS FIS
AWS Fault Injection Simulator (FIS) is a fully managed service for running fault injection experiments to improve an application’s performance, observability, and resiliency.
Tools that are part of the “Release” phase
AWS AppConfig (part of Systems Manager)
AWS AppConfig makes it easy for customers to quickly and safely configure, validate, and deploy feature flags and application configuration.
Wishlist
I’ve been able to gain some experience with the tools while working on a few projects, including cdk-codepipeline-flutter and here is a list of things that I believe could be improved. My main focus here is on CodePipeline, as it serves as the glue between all of the other tools.
Native branch support for CodePipelines
Working with Jenkins and the MultiBranch plugin makes it easy to allow developers to quickly test and deploy code that they are working on using the CI/CD pipeline. Unfortunately, CodePipeline today does not allow automated branch discovery, so if you want to enable the automated execution of a pipeline for a branch, you will need to manually configure webhooks and then create a new pipeline (or delete an existing pipeline) when branches are created (or deleted). This is not easy to implement and it would be great if CodePipeline should natively allow creating a pipeline automatically for all branches of a linked Git repository.
Additional Templates and Best Practices
When setting up a CI/CD pipeline on AWS CodePipeline, this would be easier to use if additional best practices and templates would be available as part of the tool itself. AWS is starting to promote a new Open Source project called “Deployment Pipeline Reference Architecture“. this is a step in the right direction, but it needs to be expanded by other flavours of a deployment pipeline. Also the code examples need to be improved, made up to date and needs to include all languages supported by AWS CDK. This is critical to allow an efficient adoption of the different tools.
Native integration of 3rd party tools
AWS CodePipeline should natively support integrations to other 3rd party tools that should be part of your CI/CD pipeline – e.g. security scans like Aquasec and Checkmarx.
Remove dependency for a specific AWS account and support Cross-Account deployments natively
As indicated in this AWS Blog post, the best practice for setting up a CI/CD pipeline and for managing your deployments is to use multiple, different accounts to manage your deployments. CI/CD should not be bound to an account level and this includes the management of your accounts that are able to access and configure the CI/CD tools. Maybe a good option here would be the integration with the AWS Identity service. That might allow decoupling the CI/CD toolchain from the AWS account.
Up to date CodeBuild images
Docker Images provided by the CodeBuild team should be updated regularly and should support all “modern” toolkits. The open source project has some activity, but an issue for supporting newer Android versions is now open for some time…
Publishing options to the different mobile stores (AppStore, Play Store, Windows Store, etc….) should be possible
I’ve been looking at developing a mobile app using Flutter, but what I have not yet been able to achieve is pushing the created and build applications to the different app stores. Today, AWS does not support this natively. You CAN integrate this with 3rd party tools like CodeMagic, but natively there is no option on AWS to publish your application.
Wrap up
This concludes the wish list that I have today for the existing AWS CI/CD tools.
Did I miss anything that you believe should be added?
Use the comments to give feedback or reach out to me on LinkedIn or by E-Mail!
In this post, we are going to expand our existing project on Github to be able to build an “apk” file for Android and a zip file for iOS. Before I can show you how this is possible, let’s start with some challenges that I’ve faced 🙂
The aim of this CI/CD pipeline is (not yet) to be able to push the apps into the AppStore / PlayStore for testing. That’s something we can add later 😉
Challenges on the way to a full CI/CD pipeline for Flutter on CodeBuild
While preparing this post I unfortunately faced more problems building up the pipeline than expected. Several problems.
AWS CodePipeline does not support M1 / macOS build images
Currently, AWS CodePipeline unfortunately does not provide the possibility to use the famous M1 minis on AWS as CodeBuild images. Tis his a real problem, as is makes it impossible to use CodeBuild for building iOS apps. Running XCode on macOS is a requirement for building a Flutter app for iOS. The M1 minis on AWS are currently pricey as hell for this use case – if you start ONE build you are directly charged 24 hours, even if the build takes only a few minutes! You need to actual get a dedicated instance, … – not usable for our use case of quickly building something for a side-project. So we needed to find an alternative… read below! 😉
The current AWS CodePipeline standard runtimes are not able to build (modern) Android applications
The runtimes available and exposed by CodePipeline support Android runtime 29 – and the Docker images are provisioned using Java 8. Unfortunately, as of July 2021, the Android gradle tools (used by Flutter) require Java 11. I have created an issue in the corresponding Github (see here) but needed to find a workaround to move on – I think I’ve found one, but I hope that anyone reading this might have a better way or idea?
TypeScript dependencies on AWS Lambda can take your sleep
#awscommunity helps!
When implementing the trigger for the iOS App build (see more details below) I decided to “quickly” implement the HTTPS POST call using TypeScript – which turned out to be a bad decision 🙂 I had trouble getting the “axios” dependency that I am using installed correctly. I asked around, especially my fellow AWS Community Builders and got a lot of great tips and ideas (kudos to Martin and Matt). Martin had the right “stomach feeling – I was missing a “npm install”.
Matt enlightened me with the three different possibilities of making Typescript Lambda functions understand their dependencies:
1. Bundle dependency with your source code (can be achieved using esbuild)
2. Add a package.json and node_modules to the lambda function source – only a good idea if dependencies cannot be minified
3. Put the dependencies in a lambda layer
Matt Morgan
At the end, this challenge was especially difficult because I needed to add the required “npm install” in two places: In the “installCommands” for the CodePipeline itself and in the “installCommands” for the Flutter build step.
CodeBuild is slow, misses conditional steps and misses integrations – and does not easily allow multi-branch pipelines
While implementing the pipeline and solving the different challenges mentioned above, I lost some time because of CodeBuild being “slow” (>1min wait time during provisioining of the build containers). Thats understandable given the nature of the service, however it would be cool to have something like a “warm start” for a pipeline where the containers are re-used instead of re-provisioned.
There are no conditional steps – no chance to run a job only based on environment variables or anything similar. That made me implement a workaround. It would be cool to be able to use something like “branch-conditions” in the way Jenkins offers it.
CodeBuild offers only basic integration to SNS, but you cannot integrated a “lambda build step” to run the CodeMagic integration i nparallel to the flutter build job, but that is not possible, so I needed to run this “at the end” of the pipeline.
Another thing I’d love to have: multibranch pipelines. I needed to merge everything to main directly in order to test, because I couldnt figure out how my CDKPipeline would be able toe support multiple branches.
Reaching the goal: a full CI/CD pipeline running on AWS CodeBuild to build a Flutter app for Web, Android and iOS
Here is a diagram of the “final result” that I am presenting today:
Overview picture for CI/CD pipeline for Flutter App publishing to Web, Android APK and iOS zip
The “output” artifacts of our pipeline are: – Flutter Web application (located on S3 and reachable through HTTP call) – Flutter Android APK (that can be side-loaded on Android phones, located on S3 bucket) – Flutter iOS App (that can be side-loaded on iOS phones, located within CodeMagic)
As the diagram shows, we needed to fall back to an 3rd-party, non-AWS service to be able to package the iOS application. After doing a quick “vendor selection” and a shortlist that included Bitrise and CodeMagic I decided to integrate CodeMagic in this example – because I liked the API more and it offers more free build credits/minutes. Setting it up took less then 5 minutes – it connects natively to Github and the set up of the Flutter pipeline is very easy. The integration is set up using a Lambda function that calls the “start build” API.
How did I solve the challenges mentioned above?
The problem building the iOS image was resolved by integrating the external Service CodeMagic.
The Android Runtime Dependencies problems with Java 11 was resolved by switching to a custom docker container (Open source) – and then installing the requirements on top of it (npm/node, awscli, etc.).
What did you learn in this post?
In this post you have learned on how to expand the implementation of our CI/CD pipeline for an example Flutter application to not only building a “web” application, but also building an Android APK and an iOS zip file. You have also seen an extension and integration of the Codepipeline with SNS for notifications and those events being picked up by a lambda function to trigger an external HTTPs API. This is a major step – with this pipeline we are able to publish our application for three different “platforms” without a single code change – and it will all happen completely automatically!
I’d be glad to get your inputs into my Github repository as a pull request or just as comments on the project itself.
Next steps
Further expansions needed to this project: – CodePipeline already has a SNS topic that it reports to – but right now the build iOS / Android App packages are not exposed anywhere – the idea would be to publish to an SQS queue the name of the APK file and the CodeMagic Build Id – and have a lambda function that is triggered by the queue update a link on the example application to download the newest version of the app 😉 Today, we need to retrieve both from S3 / CodeMagic itself – use the CloudFormation Exports of the Lambda functions in the Flutter application instead of hardcoding the URLs for the Lambda Function URLs- enhance security for Lambda Function URLs – add CloudFront in front of S3 to allow HTTPs connections to the Flutter App – enhance CI/CD pipeline to package Windows App using Flutter – enhance CI/CD pipeline to push created apps to App Store / Play Store
Feel free to contribute and add your contributions to this project into my Github repository.
In this post I am going to use CDK Pipelines to build a demo Flutter application hosted on AWS S3 with a Backend powered by AWS Lambda (using Function URLs). The CDK code will be in Java, the Lambda functions in Typescript and the WebApp in Dart. Why? Because I love trying out things 🙂
The code used here is not production ready and does not fulfill required security best practices! 🙂
The CI/CD pipeline
The CI/CD pipeline for this project uses CDK Pipelines and that means it is build on top of AWS CodePipeline (which under the hood uses CodeBuild, CodeStar and other services to be functional).
It consists of different stages that build and deploy the corresponding part of the application: – one stage for each lambda function – one stage for the build and deployment of the Flutter application
The stages required by the CDK Pipeline to update itself are automatically added but not part of our code:
This is the definition of the CI/CD pipeline, it uses our Github repository as the source of the code and automatically starts after a push to the repository on the main branch. The “Flutter Build Stage” is the one that currently builds the Flutter web application, deploys it and makes it available to the end user. Going forward, to make best use of Flutter, we would need to expand this stage to also be able to build an iOS application, an android application or an application for any other platform supported by Flutter. As a “goal” I would personally also want to extend this stage to be able to publish the apps to the corresponding stores (App Store, Play Store, Windows Store, …) – Thanks to my friends at cosee for the help and guidance around this process!
The architecture diagram of the application
What we are using to show-case the usage of AWS CodePipeline, Flutter as an application and AWS Lambda Function URLs as backend is not really an “application” – but it can do dynamic things and it can easily be extended to include a database backend, etc.
Infrastructure as Code using AWS CDK in Java
In this section you are going to have a look on the CDK code required to provision the infrastructure on AWS. We are using AWS CDK written in Java – and because of that Maven as a build tool (Maven is the default tool for CDK projects – I’ve already used Gradle as build tool and that works in the same way.
The possibility of writing Infrastructure Code in Java is a great thing because it gives us the option to build on top of our existing skills – and I’ve written enough lines of Java in my career to feel comfortable using it to provision the infrastructure. This is one of the best things in CDK: You can write your Infrastructure code in Java, Typescript, Python, … – and that helps us to build teams that only have “one” language as a “core skill” – one team might choose to develop in Java, anotherone in Typescript, another team could use Go – this allows the teams to build up mastery in a specific language!
In our example however, we are not making use of this possibility – I’ve choosen to rather go the opossite way: Combine a few languages, just to show that it works 😉
Stacks in the Example Project
The CDK code consists of four “stacks” – each of the Stack representing one “component” in this application. In our example these four stacks are part of one CDK Application & one CodePipeline. In bigger projects, you might want to split these out into seperate applications – which introduces a lot tof things to consider (e.g. how do they fit and work together, etc.).
While writing this post, the four stacks combined are 129 lines of source code. With the help of the CDK Constructs that are being used this translates to over 1k lines of code in CloudFormation. We are only using L2 constructs here – there is way more constructs available that you can use in the Constructs Hub and also a lot of guidance regarding the usage of CDK over at CDKPatterns.
CDK Stack that provisions the infrastructure for the Flutter hosted website
Making a S3 bucket available to host our Flutter application becomes a very short piece of Java code as shown in the picture above.
Lambda functions behind Function URLs in Typescript and Python
This was definately one of the most awaited announcements in the Serverless space this year: The GA of the “Function URLs” for Lambda on AWS – and this is the reason for me using these as a backend in this showcase. With this announcment, it is possible to directly expose your application running on AWS Lambda behind an HTTPs endpoint! Without an ApiGateway, Proxy, … Provisioning the infrastructure for the lambda function with the function URL functionality activated is only a few lines in CDK:
CDK Stack for the Check Age Lambda function with the activation of the Function URL
These Lambda functions will now horizontally scale without us as a developer getting involved. More details on function URLs – there is a lot of good posts on it around like this one or this one. Around scalability, fellow community builder Vlad has written up a great guide around scalability for containers on his website.
Web Single Page Application built using Flutter and the benefits of using Flutter
Flutter as a multi-platform, developer driven tool gives a lot of flexibility. With a single code base, you are able to publish your application to various targets. In this example, we are using the target platform “web” – which compiles the Dart code to a single page application.
This application is automatically aware of the size of your screen and is interative – which is another cool thing that Flutter takes away from you as a developer. A lot of organizations use Flutter today and the cookbook gives you an easy and good start into developing Flutter applications.
Our example application has three input fields that take input and pass it back to our Function URLs – and then automatically update a Text Widget with the results of the Function URL call. This implementation will work on all platforms.
What did you learn in this post?
You have learned how easy it is to provision AWS infrastructure using AWS CDK. You have also seen that you can easily combine different programming languages in a single CDK application and got an experience on how a CI/CD pipeline can help to automatically deploy our application using AWS CodeBuild. In addition to that, you have looked at Flutter as a multi-platform development tool.
I’d be glad to get your inputs into my Github repository as a pull request or just as comments on the project itself.
Next steps
Further expansions needed to this project: – use the CloudFormation Exports of the Lambda functions in the Flutter application instead of hardcoding the URLs for the Lambda Function URLs- enhance security for Lambda Function URLs – add CloudFront in front of S3 to allow HTTPs connections to the Flutter App – enhance CI/CD pipeline to package Android App using Flutter – enhance CI/CD pipeline to package iOS App using Flutter – enhance CI/CD pipeline to package Windows App using Flutter Feel free to contribute and add your contributions to this project.
Listen to Werner Vogels talking about the benefits of CDK
