In this post you will get a short introduction and my personal assessment on how the new Packages component in CodeCatalyst can help you to set up your complete SDLC in Amazon CodeCatalyst.
Only npm supported – a early launch to show what we can expect going forward and to get feedback from users?
With the initial launch Amazon CodeCatalyst allows to manage npm package repositories inside CodeCatalyst. When seing this for the first time I was a bit concerned that this decision – to initially launch with only npm support – will not help a lot, as I expect that organizations would need to store other type of artifacts (jar files, containers, python paackages, …). But I think that at least for Cloud Native Projects that use typescript this will solve the problem of storing artifacts and accessing them natively within CodeCatalyst. Let’s look at how they can be used today.
Using packages repositories in CodeCatalyst
Today you can set up package repositories in CodeCatalyst and connect to upstream (public or private) repositories. You can also change the sort order and the upstream order of the package repositories. The documentation covers the different possibilities to set up repositories very well. You can access the package repositories from your local machine by setting up the connection locally
npm config set registry https://packages.region.codecatalyst.aws/npm/space-name/proj-name/repo-name/
After setting up multiple repositories you can set the order of retrieval and usage. You can also set it up as a “pass-through” repository and allow access to public repositories.
Using packages in CodeCatalyst
Packages in CodeCatalyst are integrated with workflows. You can read or store packages with native actions or by setting up the npm repository manually.
You can also read and publish packages from other systems by using Personal Access Tokens (PAT). The documentation outlines currently supported client commands.
The CodeCatalyst workflows will by default use the set up repositories that you have set up within CodeCatalyst.
What I think we need next in packages
With the launch of the packages component in CodeCatalyst AWS has added a definately missing functionality in CodeCatalyst. Users are not able to store artifacts and share them within a space which makes it possible to create, re-use and deploy immutable artifacts natively within the tool.
This is a much needed functionality, but as I already mentioned, I believe that limiting this to npm packages only limits the use cases for it. I would have expected npm, python and docker support at launch and was a bit disappointed that this was not included.
I do also believe that the “packages” functionality should be better integrated and allow further configuration options, especially when loading/reading packages from upstream repositories – e.g. being able to limit to only certain licences or ensuring that packages included are still “supported” (and regularly updated) or security-scanned.
These type of functionalities would have made the new option “meaningful” and would have empowered developers to build better software too.
I can imagine that we will see python, java and dockersupport pretty soon due to these reasons and it would be great to also support “internal” repositories as “upstream” repositories.
How do YOU think that this new functionality will help you to adapt CodeCatalyst? Please drop me a message on socials or an E-Mail!
In this post you will learn how the new integration of the IAM Identity center with Amazon CodeCatalyst allows to use your own IdP as single source of truth of users for CodeCatalyst spaces. We will look at how you can connect a space to an IdP and on how this can be used to set up certain permissions based on your IdP roles.
Single Sign On (SSO) using IAM Identity Center empowers CodeCatalyst users to use their own IdP to provision user accounts in CodeCatalyst
With the new integration for Amazon CodeCatalyst into the AWS Identity Center organizations can now connect to their own existing IdP (AzureAD, Ping, Okta, …) going through the AWS Identity Center.
Benefits of the new integration
With this new integration users and organizations go to the Space settings in CodeCatalyst and choose to integrate with an existing instance of the IAM Identity Center.
This also connects potentialy to any other IdP that uses the IAM Itentity Center as a proxy. Administrators will only need to manage roles and permissions in on central place. The roles from the IdP can then be mapped to certain team roles in Amazon CodeCatalyst.
What I don’ t like about the new integration
There are two things that I didn’t like when looking at this integration:
Why do we need to go through the IAM Identity Center and cannot allow direct SAML or OIDC integration for a specific space? With the “need” to have an IAM Identity center organizations also need to directly connect to an existing AWS Account to manage connectivity – and this once again brings back a “tooling” account that initially I was hoping to replace with CodeCatalyst.
The second thing that I missed was the possibility to automatically map existing roles to groups in Amazon CodeCatalyst – today this is a – one time – manually effort that needs to be conducted. If you need to do that only once for your organization that isn’t a problem, but if you decide to have multiple spaces for your organization and need to perform this multiple times, then this can be very time consuming.
Summary
I believe that this change will have a very big impact on the adoption for CodeCatalyst as administrators and security teams in organizations will make it easier to allow the usage now that there is a central place to manage identities. And with the possibility to actually try out CodeCatalyst in an enterprise world, I hope that we will also see more adoption of CodeCatalyst and with that also a more streamlined and better communicated roadmap of CodeCatalyst. If you would like to learn more on how to activate this, please refer to the documentation.
In this post I’m writing about how getting to re:Invent 2023 was and about how the last year has made a big impact on my personality and my efforts in the AWS community.
Thinking it couldn’t get better last year, 2023 changed me once more and leveled me up
While going to re:Invent 2022 I wrote down my thoughts and went home from re:Invent expecting things could not get better for me. But this year has prooven my expectations wrong, as the last 10 months have been life-changing in a lot of different aspects.
Blogging & Starting a YouTube channel – winning 1.5 hackathons and starting a project
When I reached my home town after last years re:Invent I knew that this community is what makes me happy and gives me back so much that I also would like to give something back. And so, I formed a few ideas in my head and started a few initiatives that I would like to share with you today:
Blogging & Writing
I have been blogging both on dev.to and on my personal blog – writing about topics and things that I do or did on a regular basis, trying to share experiences of day-to-day things and building experiences.
The most important thing for me he is to always be transarent and authentic – speaking out on things that I liked and challenges that I’ve faced.
For you reading this – thank you, for being part of my journey, for giving me feedback and for being you! Please let me know if you have any feedback on what I should change or do better.
Kicking off a YouTube channel
The last three years of my career have been focused on building secure and reliable CI/CD pipelines! But hey, there is so much more to learn. As part of my travel back to my home town after re:Invent, I had the thought to share my experiences and the experiences of other builders on a “podcast-like” YouTube channel which is today known as @cicdonaws.
Learning on how to produce a YouTube video to how to set up a good lightning and sound, preparing a script and creating better thumbnails – a lot of learnings, experiences and hours went into the videos that I have been producing.
This only kicked off because of all of the great builders that joined me on the show and shared their projects, learnings and experiences with me. Thank you all, every single one, for your time, dedication and experiences. Keep up sharing your knowledge, as this is how we can all grow!
Winning 1.5 hackathons and making a small project something big
Back in march Lily announced a, Community Builders-internal hackathon where it was possible to win some prices – and I thought I had a great idea on what to build that actually fit into the “rules” of the hackathon. And this is how the project started that has since then used up countless of hours of my after-work evenings and weekends. Millions of Slack messages have gone from Germany to US, from US and Germany to Africa and back – we kicked of a real project with our Speakers Directory that is finally ready for prime time! After becoming second in the first Hackathon, we also submitted the same project with a few tweaks for a 2nd hackathon powered by Hashnode and Amplify and this also brought us “honourable mentions” in the list of top 10.
We’re ready now to get more of you to add your talks and events! Please sign up and give us feedback, we are very proud of what we have produced so far and eager to get all of you on board!
Thanks to Danielle, Julian, Matt, Raphal, Baimam for actively working with me on this fun project – and thanks to everyone else that has supported us through 2023 to make this a bigger thing!
Organizing a community day – the EU (more technical) edition of re:Invent
Another piece last year has been organizing the AWS Community Day DACH that happened in Munich this year – it was fascinating to be part of the team that brought more than 500 AWS nerds together, including over 25 AWS Heroes and 30 Community Builers!
Thank you all for attending – and thanks for the sponsors for making this possible!
We’re up for doing this again next year – further details to be announced soon! Looking forward to see you there!
Hey, hiking before re:Invent is what you need!
And then, there is this “secret thing”: Ever since I went to re:Invent I used the sunday before re:Invent to go out “hiking” somewhere close to Las Vegas. The first two visits together with colleagues were three persons – a pretty cool small group.
Last year, they did not go with me to re:Invent, so I asked Community Builders to join me – at the end, we were nine persons from I think 8 different countries hiking together from 10 am till 5 pm – we had a lot of fun, so I decided: Let’s do this again!
Somehow, this became a bigger thing – for this years hike, we have over 50 Community Builders, Heros and UG Leaders signed up. The group became so big that the AWS Community team is helping us this year by sponsoring the transportation.
Thank you all, that you are supporting this thing – and thanks to everyone participating, I am sure this will be an amazing experience for everyone!
How and Why what I do for the community changed
Since June I can name myself an AWS Devtools Hero – which honors me a lot but also drove a bit on how and what I do for the AWS community. My work has become a little bit less “public” – a lot of my hours that I have “free” went into talking to AWS Service Teams, into mentoring other builders and helping them grow. This has lead to less new videos on YouTube and less blog posts – but instead, I’ve helped others to grow and to invest into the community – supporting the “re-start” of the AWS UG Frankfurt, the re-start of the User Groups Mainz, Karlsruhe and Heilbronn and overall making other builders “grow”. This makes me happy.Thank you all, that you are part of my journey.
Making new friends
This year has also shown to me how important relationships are and how much you can make friends by supporting each other. Friends, I’ve never met in person (Matt), friends I’ve finally met in September (Ran) and others like Markus, Thorsten, Philipp and Raphael that are part of most of my days and that I write 100s of Slack messages per week.
This is what community means! Making friends, learning, growing. Thank you, for being you!
What’s next for me?
Changing my role at work
I’ve recently moved to the global Platform Architecture team here at FICO and that gives me a lot of new possibility and learning opportunities.
The platform that FICO is building is an important one and helps organizations around the globe to take better decisions – I’m proud to be part of the team and looking forward to make this bigger than it is today.
Community Day DACH 2024
We’re already planning next year. Stay tuned for some announcements…soon!
Supporting and mentoring builders around the globe
I’m up for helping other builders grow – if you’re intersted to collaborate with me or to learn from me, reach out to me and we can talk!
If you have any CI/CD or CodeCatalyst specific topics that you would like to talk about on my channel – reach out to me!
If you need feedback, help or advice on anything (e.g. a blog post or anything else) – let me know and reach out!
Looking forward to an amazing re:Invent 2023 with a lot of friends in Las Vegas!
Have a great week – don’t be a stranger and say HI if you see me around!
The new service announced by Amazon in Las Vegas at re:Invent 2022 which is an integrated DevOps service to empower development teams to develop and deliver software faster finally reaches the “general availability” status. As I have previously outlined, this achievement is very important for Amazon and the CodeCatalyst team. Congratulations to the team for reaching this goal, which I can imagine is not an easy step for this product. The tool touches a lot of very sensitive parts of a software project and I can imagine the security standards being really high.
A hugh achievement – thank you everyone in the team for investing into CodeCatalyst and for listening as closely to the customer feedback as you are!
What changes did get implemented for GA?
As part of the GA release we see a lot of minor improvements in the User Interface and color changes. In the last weeks, we have seen a few “bigger” changes – like the possibility to use Dev Environments for Github based projects. We also got “graviton based” execution environments for CI/CD workflows which, according to AWS, should reduce our costs.
It is still hard to track down all of the changes in CodeCatalyst, as there is – to my knowledge – no public or semi-public roadmap. This is one of the things that I’d love to see, as for an integrated service that is at the core of the Developer Experience for teams, any minor change can either improve or destroy the “usage experience”. If you as a team invest into adoption a new tool like CodeCatalyst, they will need to know how changes in workflow, features or user interface can influence their day-to-day activities. Let’s see, maybe the team can share “something” like a “changelog” with us (or even an RFC process like Amplify or AppSync)?
Reached “GA” – so who can start using it now?
As of today CodeCatalyst is only availble in US regions and this means that it can be adopted mainly by US enterprise customers. CodeCatalyst already gives you the possibility to set up different Spaces for your account and within a space you can manage multiple projects. So in theory, CodeCatalyst is “ready to be used” by everyone.
Practically speaking, it is easier to adapt the service for new projects than for existing projects , as there is no real “import” functionality. Yes, you can integrate existing Github projects, but that only integrates the source code. Unfortunately that does not make all of the “cool” things available right from the start of integrating the source: existing workflows (CI/CD pipelines) are lost and need to be re-build, issues/tickets are not imported into CodeCatalyst (tho they can be made available through the JIRA integration).
I have been regularly using CodeCatalyst (both for imported and “new” projects) – and I really think that the tool already works very well.
The “killer feature” that I see for new projects are the “blue prints” which essentially get you started within minutes, e.g. to deploy a SPA application, or to have a “true” CI/CD pipeline for a full stack application following the DPRA.
Right now I would recommend using CodeCatalyst for any new project that you start to start building out your workflows and best practices.
So what do I still need to recommend CodeCatalyst for existing projects?
There are a few things that I have already been writing about:
“Import” of existing CI/CD workflows (e.g. Github actions, CDK Pipelines or CodePipelines)
Fully import projects
existing issues from Github or JIRA
Git-based projects including the history
Tighter security settings and permissions
Fine granular roles to allow or forbid access to specific parts of a project
Options to allow or forbid execution of workflows (or to deployments)
Additional workflow options
Manual approvals are very high on my wish list
Integration of other AWS services natively
A question for the readers: What do YOU think that you need to adopt CodeCatalyst?
A big question for the CodeCatalyst team – HOW MANY AWS TEAMS ARE USING CODECATALYST FOR PRODUCTION DEPLOYMENTS TODAY?
Where do I see the potential for CodeCatalyst?
CodeCatalyst is a big bet by AWS. There is a big potential that can really improve the life of development teams and these are the main things that I believe that can out-grow other existing solutions:
Integration of AWS Services / deployments metrics
the true integration with AWS APIs
Integration into “post-deployment” verifications (e.g. auto roll-back after failed CloudWatch metrics)
“At-hand” developer support to improve efficiency
with CodeWhisperer (who recently reached GA) AWS already aims to support developers during the development phase, but with CodeCatalyst AWS can take this to the next level:
AI support during Pull Request Reviews (or automated approvals for PRs – e.g. by including CodeGuru, etc., automated merges, etc.)
AI support during workflow executions (when to approve, when to deloy, when to promote, etc.)
With improvement proposals for workflows if the “AI model” recognizes patterns (in issue workflows or CI/CD workflows)
With automated improvements for existing projects based on blue prints
Best practices change – and so blue prints change – and if the CodeCatalyst team can automatically apply them for existing projects, customers will benefit from it
And last but not least:
I trully believe that every software project should start with a CI/CD pipeline – and with the Blue Prints including the CI/CD workflow that follows DPRA and other AWS best practices, we can trully make this possible: Empower developers to deliver their software projects in minutes right after starting their project.
Do you see the potential in CodeCatalyst? If you do not see any potential in the tool – why not?
At re:Invent 2022, as usualy, different new AWS Services or functionalities have been announced in Preview. Now, at the beginning of April 2023, a few of them have already reached the “General Availability” (GA) status – Application Composer (in early march), Latice (in late march). My favourite new service, Amazon CodeCatalyst, has not yet reached this goal – but I have a feeling that now is the right time to think about what and when we can expect this status.
You wonder what CodeCatalyst is? Watch this video on my YouTube channel or read my two initial posts about it.
Why is reaching the “GA” milestone so important?
Before starting with my assumptions on what we can expect for GA, lets clarify why reaching this milestone is so important. Being “in Preview” can mean a lot of different things. In a lot of organizations this usually translate to “limited availability”, a service not being available in all regions or not being reliable or scalable. For other organizations, it means that specific aspects of the product can be immature or not reliable. It can also mean that bigger API change are yet to be implemented or missing security guardrails.
In general, this can be seen as a “beta” offering which is not appropiate to use for productive workloads.
Because of these reasons and maybe others, a lot of organizations (especially US based) do not allow using or adapting services that are in “Preview”.
For all of my experiences, tests, videos and projects I was so far able to only be on the free tier. And I assume that this will also be the truth for most of my readers: You can get a long way using the Free Tier that Amazon CodeCatalyst offers today.
So thats another big reason for AWS to push this service out of “Preview”: It gives organizations that are forbidden to use the service in “Preview” the possibility to start using and adopting the service – and with that Amazon can start earning money with the service which unil now might be difficult.
And as we know, AWS tries to “work backwards” from client requirements and the early usage of CodeCatalyst will drive further investments into the service.
What to expect for GA of CodeCatalyst?
Simple: Nothing big – most probably only regional rollout.
I do personally not expect any major new features for the service as the team has been constantely releasing new features and functionalities to the service on a regular cadance. There was simply not more time to work on bigger features while preparing the “General Availability” (GA).
What the CodeCatalyst already has delivered until today…
Let’s look at what has been added to CodeCatalyst since its official release in december 2022:
Additional Reporting auto-discovery
Change Tracking – the possibility to see which changes have been deployed to a certain environment
Additional Workflow native actions and improvements, E.g.
a problem with the CDK action to be able to define the “workpath” of a CDK app
Additional native actions
Linked issues to Pull Requests – you are now able to link issues to a pull request
UX improvements
Log files wider accessible in UI – at the beginning you where not able to make the log view larger, now this is possible
This is not a complete list, but the things that I personally noticed and that I liked to see.
So…when is “the date”?
Hard to guess, but I would expect “soon”. Ideally right before a month starts, which will make the billing cycle easier 🙂
So I would guess “end of april” which would bring the service right in time for the Berlin Summit (3rd of May).
Next steps for CodeCatalyst
In my last posts I have already been communicating my thoughts and features that I would love to see. But what will AWS implement?
Given reaching the “GA” status opens the way to “enterprise clients” I would expect that one of the first features will be Single-Sign-On functionalities, maybe with an integration to Okta, Ping, Azure Active Directory or other already existing IdPs.
In addition to that I believe that the User Interface needs to get some tweaks to streamline the navigation and workflow – that’s something that I personally experience every day: not knowing when and where to click to get on the rigth place. Also I think that additional service integrations will be added – e.g. StepFunctions or SNS, maybe SQS – see also my post about sending notifications from workflows.
And then there is one last thing which has been getting only limited attention so far: APIs and CLI integrations that can be used – so I would expect a major update there.
I’m really looking forward to see CodeCatalyst reaching GA – I’ve had various conversations with the team in the last months and I know that they have a true vision to make CodeCatalyst successfull as a trully AWS integrated and fully functional DevOps tool.
Are there features you are missing? Please let me know and I will forward them to the team.
As Amazon CodeCatalyst is still in Preview, it has only limited integration possibilities with other AWS services or external tools. Sending notifications from a Workflow execution is something that I believe is critical for a CI/CD system – and as I focus on CI/CD at the moment I’ll focus on the notifications on Workflows in this article.
What kind of notifications do I need or expect?
As a user of a CI/CD and Workflow tool there are different levels of notifications that I would like to receive:
Start / End and Status of Workflow execution
State / Stage transitions (for longer running workflows)
Approvals (if required)
In addition to that, based on the context of the notification I would like to get context-specific information:
a) For the “Start” event I would like to know who or which trigger started the workflow, which branch and version it is running on, which project and workflow has been triggered. If possible getting the expected execution time / finish time would be good b) For the “End” event I would like to know how long the execution took and if it was successful or not. I would also like to know if artifacts have been created or if deployments have been done. If the “End” is because of a failure, I would love to know the failure reason (e.g. tests failed, deployment failed, …) c) For the state transitions I’d love to know the “time since started” and “expected completion time”. I would also like to, obviously, know the state that has been completed and the one that will now be started. d) For approvals I’d love to be able to get the information about the approval ask and all required information (commit Id, branch) to do the approval
What does CodeCatalyst Support today?
Right now CodeCatalyst allows to set up notifications to Slack. Please see details on how to set this up here. This notifications are also minimal right now:
In Slack this looks like this:
How can I enhance the notification possibilities?
Luckily one of the “core actions” is the possibility to trigger a Lambda function and this is what we are going to use here to be able to trigger advanced notifications using Amazon SNS. In our example we are going to use this to send an eMail to a specific address, but you can also use any other destinations supported by SNS like SMS or AWS ChatBot.
Setting up pre-requisites
Unfortunately we will need to set up an SNS topic and a Lambda function in a dedicated AWS account in order to use these advanced notifications. This means that we are “breaking” the concept of CodeCatalyst not requiring access to the AWS Console, but this is the only way that I found so far to be able to send additional notifications.
Ideally we would be setting up the SNS topic and the lambda function using CDK, but that increases the complexity of the workflow and of the setup and because of that I’m not including that in this blog post.
Setting up the SNS topic
Please create a SNS topic following the AWS documentation through the console. We assume the topic to be in “eu-central-1” and the name to be “codecatalyst-workflow-topic“.
You can follow this blog post to manually set up the lambda function through the AWS console, please ensure to give the Lambda functions permissions to use the SNS topic. The required code using Python will look like this:
Obviously the same can be achieved using Typscript, Go or any other supported function. Please adjust the topic_arn to match the topic that you just created. After creation this Lambda function will now have an ARN which should look similar to this: arn:aws:lambda:eu-central-1:<accountId>:function:send-sns-notification-python
We will need this ARN when setting up the notification in our Workflow.
Integration into the workflow
Integrating this Lambda function into a workflow is easy:
As you can see, we are integrating an “aws/lambda-invoke@v1” action which then points to the lambda function that we just created.
In the “RequestPayload” we are passing a few information to the Lambda function which will then be passed to the SNS topic as part of the message. This is how the message will look when received as eMail:
Missing information and next steps for enhanced notifications
As you can see we are able to send notifications from CodeCatalyst to multiple targets, including eMail with this option.
What we are missing is – and I am not sure if thats possible or not – is all of the “metadata” of the workflow execution like:
Workflow-Name
State-Name
Project Name and additional information
In the documentation I was not able to find out the available environment variables about these information…. If you do have any ideas on how to access this metadata, please let me know!
A few weeks ago, on december 1st 2022, Werner Vogels announced Amazon CodeCatalyst. I’ve previously shared my initial thoughts and findings in a blog post. In this post, I’m going to share a few more findings and insights into using Amazon CodeCatalyst and will also see if any of my wishes from the wishlist for CI/CD on AWS have been resolved with CodeCatalyst.
What I have been playing around with…
CodeCatalyst login page
One of my personal projects that I am working on together with a few friends is pegasus-galaxy.net and the CI/CD pipeline that I had built with CDK Pipelines (that I also presented at re:Invent 2022) was the first one to try to move over. In context, we’re talking about a Flutter application for Web running behind CloudFront, deployed using CDK.
I decided to try CodeCatalyst out and go “all in” – and that means moving the code from Bitbucket into CodeCatalyst as well as setting up the other users in CodeCatalyst and moving the workflows (=CI/CD pipelines) over to CodeCatalyst.
CodeCatalyst Menu
In this article I am going to go through each of the sections in CodeCatalyst and will share my experiences, thoughts and findings.
Where I have ideas on how to improve the day-to-day work with the tool, will try to share that.
Before going into details, lets start with the most important thing:
Amazon CodeCatalyst works very well and reliable and the current version of the service is a great foundation for moving all of your CI/CD and development practices to AWS.
The CodeCatalyst team has been very supportive on re:Post, so if you have a question, feel free to ask it there!
CodeCatalyst Overview – Spaces and Projects
Spaces are the “Top-Level” option to organize your CodeCatalyst account. You will need to associate an AWS Account for billing used AWS resources. Each AWS (billing) account can be associated only with one CodeCatalyst Space.
CodeCatalyst Spaces Billing page
While this seems like a limitation as you will need to create a different billing account for a 2nd space, I can right now not see an impact for my day to day work. For anything that I run on the same AWS account, I would assume that using a project within the same space should be good.
You can manage Projects, Members and AWS Account connections on the space page. In the “extensions”, CodeCatalyst currently allows a connection only to the JIRA Cloud. I would expect that additional 3rd party extensions will be supported in the GA version of CodeCatalyst.
Projects Overview and options
A project is a “unit of work” in your product or software that you are building. Within projects, you can manage issues, manage your code repositories, execute workflows (CI/CD) and review report results.
Projects are associated to a Space – and you can create as much projects in a Space as you want. You can add team members to a project, that are not able to access all projects in the space. Unfortunately I have not yet found an option to “hide” projects from Team members that are added on the Space itself.
Managing issues / tickets
CodeCatalyst currently provides two options to manage your issues or tasks: 1) Link to JIRA Cloud Project 2) Internal issue management
If you use the option to link to a JIRA cloud project, the “issues” link is replaced by a link to your JIRA Cloud project.
Internal issue management
The internal issue management system currently offers everything that is required for a simple Kanban workflow. You can create issues, add them to a backlog or a Kanban board, assign them to project members and track their current status. I personally think that the current functionalities are good enough for small teams and simple projects – I’m actually already working with it in a small project and will add additional feedback as soon as I gain more experience.
Code
Within the “Source” part of a project, you can manage source repositories or connections to source repositories in Github. I expect that other providers will be added going forward (e.g. Gitlab, CodeCommit, Bitbucket, …). You can also manage pull requests and approvals – I was only able to test this using internal source repositories, not using a linked repository.
The last option – the Dev Environments – is the most exciting functionality – it gives you the possibility to host development environments (similar to Gitpod) on AWS using Cloud9 but also, and this is really cool, using Visual Studio or JetBrains IDEs. When using that option, the IDE on your local PC is only the “presentation layer”, the source code is stored and run on an AWS instance and the IDE uses remote connectivity to talk to the Dev Environment in the background.
CI/CD
CodeCatalyst currently uses the same approach as Github Actions to manage your workflows or CI/CD pipelines – you are able to manage your Workflows using YAML files. The syntax is simple and understandable. There is a minimal set of Actions available as part of the preview. You are also able to use existing Github Actions as part of your workflow.
Workflow overview in CodeCatalyst
The workflow functionality is very powerful. In my tests I have not yet been able to test all parts of the capabilities. Workflows can be defined for certain directories, for certain triggers or branches. Test reports will be exposed in the “reports” functionality.
CodeCatalyst offers a graphical overview for workflows and alows to edit them in the UI, too. This functionaly works pretty well and helps to quickly get you started building your first workflow in CodeCatalyst.
I’ll need to test the workflows more to be able to give additional insights into how good or bad they are currently running. My simple pipeline that builds a Flutter Application, deploys my Infrastructure as Code using CDK and then publishes the new version of the Flutter app runs without problems.
One of my main concerns so far is the execution time, however the team has been working on a possibility to use Lambda as an execution environment. This option however does not yet support the execution of Github actions and also has some other limitations.
The other features that are part of the “CI/CD” – Environments, Compute and Secrets – I did not have time to play around with this. If you have any experiences with it, please add your thoughts in a comment to this article!
Reports
The reports today only suport test reports. I have not used the functionality enough to assess this, but I am sure that the CodeCatalyst team is going to add additional reporting options going forward.
Things I like most about CodeCatalyst (Preview) after 6 weeks of usage
Just a short list of things that I already like: – Integration of Github Actions as workflow actions – Managing workflows using UI & code
Things I miss in CodeCatalyst (Preview) after 6 weeks of usage
– macOS builds (e.g. for Flutter iOS apps) are still not possible – granular permissions for workflow and Pull Request triggers – and….
Let’s talk about Open Source Projects
Right now there is no option to share a project or a repository that is hosted within CodeCatalyst as an Open Source project. This is really a limitation if you want to use CodeCatalyst for Open Source project – or if I would like to share a CodeCatalyst repository with example workflows. I hope this functionality will be added soon.
Wrap up and next steps for me with CodeCatalyst
I need to admit – writing this post took longer than expected 🙂 I wanted to publish it before christmas and now it seems to be a bit “late” already as I am sure that a lot of you have made your own experiences with CodeCatalyst today – please SHARE your findings with me – links of Blogs that you have written or other content you have created, I am eager to consume it!
My next steps with CodeCatalyst
I am working on migrating my project pegasus-galaxy.net completely to CodeCatalyst and collaborate with my team on it there. With that, I will be able to proof CodeCatalyst in a “real world” application that it is “multi-platform” application – using Flutter for Web, Android and iOS – and a Serverless AWS based backend. If you’re interested to join this project, please do not hesitate to reach out – skills that we need right now: AppSync, DynamoDB and development/software engineering (Flutter, Typescript, Java, or Node?)
As we are now in the post:Invent phase of 2022 and over 10 days have passed since re:Invent 2022 in Las Vegas was concluded, it’s time for a lot of re:Cap Blog posts and events. I’ve read so many of those “major announcments” articles that I’ve decided to write a different type of re:Cap for myself this year: Sharing a few stories from my 10+ days in Las Vegas, as they are as equally important as the technical announcements made by AWS.
Indeed, it was a great conference with a lot of learnings and a lot of very interesting sessions. I focused on Chalk Talks, Builder’s sessions and events (like Gameday) as these are not recorded.
Making new friends before re:Invent kicks off
My flight this year got moved from Saturday to Friday, so I had one more day to get over Jet-Lag. On Friday, I spend a good time shopping and besides that met with Oliver vor dinner. On saturday morning (early morning!), I looked at the AWS Community Builders Slack and found out that Traian set up a “spontanous breakfast” for the Jet-lagged folks – and I ended up sitting over two hours with different parties, having fun, chatting and getting to know people. It was exciting to meet Rafael, who had been our Solutions Architect for a while, for the first time in person – without planning it 🙂 It was also great to meet Heitor in person – the person that owns the Lambda Power Tools at AWS. His talk is now on Youtube and I would encourage you to listen to it if you are interested in Open Source.
The rest of the saturday I spend with Markus, who shared so much Las Vegas knowledge with me that I think my brain is still burning – and I would not dare to claim remembering more than half of what we discussed – but it was a great saturday which ended with meeting Philipp for dinner at “The Cheasecake factory”.
I kicked off the sunday with a lot of excitement about my very first talk at re:Invent – final technical check in the “Speaker Ready Room” for my slide deck! That needed to be early morning, because afterwards I had planned to go out hiking with fellow AWS Community Builders.
Hiking across time zones and cultures
Definately one of my highlights this year: The ever first AWS Community Builders pre:Invent Hiking Trip!
Thanks to everyone that joined – Oliver, Richard, Jenn, Ganesh, Traian, Pubudu, Niklas. It was great to see how we supported each other, had great conversations and all managed to get across different challenges we had to fight!
Thank you Oliver (and kreuzwerker) for the amazing video. Traian, you’re my hero. Congratulations on finishing off the hike with us with. Thats an achievement noone can take away.
It was fascinating to meet you all for the first time and notice that we get a long well, without ever meeting before. That’s the power of the AWS community!
We got back at 6pm, after a over 5 hours hike, just in time to get our AWS re:Invent badges and to meet other AWS Community Builders from around the globe for a great dinner. Lilly & Jason – thanks for joining us, that really made me happy!
Welcome to re:Invent
Kicking off re:Invent with a GameDay with a great team an Jeff Barr
I decided to kick off my re:Invent on Monday with a GameDay – which is a fascinating opportunity for gamified learning. On sunday, during our hike, I had aligned with Niklas to form a team together – and the other two team members, JaeJun and Martin, we met in the morning. We had great fun, ended up 4th even tho Jeff Barr distracted us for some time as we won him on our table with a quizz. It was great meeting him in person – and I can tell you: He is a human as we are, even if his Newsblog is legendary 🙂
Meeting people from the AWS Community
I had so much great hours in Las Vegas – thanks for the time spend together, everyone that I’ve met – Stefanie, Oliver, Manuel, Mike, Stefan, Philipp, Markus, Thorsten – and others – from the german community.
Finally met Danielle and Matt in person. Another of my highlights.
The Community Builders Mixer and the User Group Leader Mixer where both great events to get to know each other better and network with great people from everywhere in the world. I met so many people that I had been interacting with in written (Slack, Twitter, LinkedIn) – it was a blast for myself.
Famous Jenga-game with AWS Heros – so much fun!
Famous Jenga-game with AWS Heros – so much fun!
Speaking at re:Invent 2022 – my DevChat
As I’ve already shared before this year I had the opportunity to speak at re:Invent – COM307 – Using CDK pipelines (in Java) to build a multi-platform Flutter application
Thanks for everyone that made this possible: Ernesto, Shantavia, Lilly, Jason, Maria! It was my biggest honor to share my experiences and my open source initiative. Looking forward to keep sharing knowledge!
Announcing Amazon CodeCatalyst
With the announcement of Amazon CodeCatalyst the conference brought for me a new service that I am eager to use and try out as I am very much interested in CI/CD on AWS. This was for me definately the most exciting anouncement of re:Invent 2022 and I had a lot of interest to talk to the service team, product managers and others after the service had been announced. I’m looking forward to share more about that as I get to play around with it more.
Flying out
On saturday my trip to re:Invent was over and it ended as my re:Invent trip began: meeting AWS interested persons at the airport (Thanks Maria for the introduction!) and with great conversations with Oliver on the way back to Frankfurt.
Thanks to everyone that I met and talked to at re:Invent 2022 – you really made this conference be a different one for me than it was before.
I’m looking forward to hopefully meet all of you again in 2023!
In this post you are going to get to know the new service that AWS announced at re:Invent 2022 in Las Vegas. With this new service AWS brings the Code* tools (CodeStar, CodeBuild, CodePipelines, etc.) into one user interface and opens up the usage to new audiences.
Amazon CodeCatalyst is a service that helps you to manage your project code in a central place and allows integrating it with existing CI/CD tools. It also simplifies the cross account functionalities of CI/CD best practices. With the market place functionality it opens up the tool for further expansion through third party contributors.
Functionality highlights
Organizations and projects
CodeCatalyst allows the set up of an organization and projects within that organization. This allows a structured set up of your projects and allows you to segregate access. What I have not been able to verify is if existing organizations of AWS Organizations are automatically imported and available in the service. This would be beneficial for larger organizations that want to get started with CI/CD on AWS.
Integration with your “builder ID”
Amazon CodeCatalyst is integrated with AWS your builder id – and that means that you can have a unique, personal account. This account is not tied or attached to an existing AWS account. The integration with the AWS accounts it done through cross account permissions that are created automatically for you. This concept has various benefits:
You (as a developer) can have access to the required CI/CD structure and code management without the requirement to have an IAM user or access to the AWS console to the specific AWS account that you are aiming to deploy to
You (as a developer) can have access to different “organizations” and “projects” within the service – which will make it easier for 3rd party teams working on AWS projects (e.g. consultancy engagements)
You can allow the CodeCatalyst workflows to only be able to deploy into certain accounts and the list of accounts that you can connect to from a specific project or organization can be modified on demand
Unified user interface and workflows
The new user interface of CodeCatalyst allows to look at the complete lifecycle of your product – form development to deployment – within the same window. You can easily set up a project, connect or set up a repository and then create a CI/CD pipeline (or workflow) for it. You will also be able to visualize the workflow of your CI/CD pipeline during creation or editing.
Integration of Github Actions
With the integration of existing Github actions and the possibility to re-use already available automation steps within your workflow AWS opens the door to re-use and migrate a lot of already existing functionality. This is going to be very beneficial and will help to drive the adoption of the service. The availability of “native” actions as part of CodeCatalyst is still limited and I assume with the Marketplace new actions will be made available quickly and the list of available actions will grow rapidly.
Project Templates – CI/CD pipelines
I believe that this is the most important functionality of the service. Allowing to quickly set up new applications, workflows or projects on the service empowers developers to get started efficiently. I’m eager to see additional templates going forward. As you can include both infrastructure and application code, but also the required workflow in a template (=that’s your deployment pipeline), I hope that with this functionality we will be able to empower developers to re-use better CI/CD pipelines that enforce a DevSecOps mindset and include all of the “best practices”. I would love to see all of the reference implementations of the “DPRA” (Deployment Pipeline Reference Implementation) implemented as templates in this service.
A good, automated CI/CD pipeline helps your developers to focus on creating business value – the more you can “re-use” or drive through standards and templates, there more efficient your teams will be.
Additional wishes
As I have not explored all areas of CodeCatalyst yet, some of my “wishes” might already be implemented and I did not yet discover them – here’s a few things that I’d love to see but haven’t found yet. Please reach out to me if this is functionality that is available somewhere and I’ve missed to find it 😊
Re-applying project templates
I would love to be able to re-apply a project template to an existing project, especially for the workflows. This allows two things: 1) if you’ve changed the workflow and it doesn’t work anymore, you can quickly go back to the working version. 2) if the template changes, you can re-apply the changes to a project without the need to access every project manually.
In combination, some times a project changes or evolves and you might want to move it from a “backend” project-template to a “full-stack” project. This can today only be achieve manually but not in an automated fashion.
Better “mobile app” support
As you’ve maybe read, I’m working on a “Flutter” application – and because of that I would really love to see a possibility in CodeCatalyst that better supports the “cross platform” app functionality. Today, the service mainly focuses on provisioning AWS infrastructure. As you build a complex project that also needs to publish a mobile application, it would be very bene be able to use the service to also publish the application itself into the different stores automatically. Of course there are other, 3rd party systems that already allow that, but if you want to use CI/CD natively on AWS this is currently not possible. I know I’ve mentioned that before, but as I have not seen the possibility to perform an iOS build on CodeCatalyst, I wanted to highlight it again.
Additional 3rd party integrations / workflows
One of the assets of the service is definitely the market place functionality that will quickly allow to integrate new 3rd party services. What I would like to see here is the support of additional repository stores besides Github (e.g. Bitbucket, Gitlab, and the corresponding “datacenter” versions), but also the possibility of a native integration of other tools that are required as part of the CI/CD pipeline – things like Codecov, SonarQube, Checkmarx, Snyk, Aquasec, etc. This might be possible through custom actions, but I am not yet clear on how fast these will be available.
Infrastructure as Code support for provisioning and setting up your CodeCatalyst environment
I’m not clear about the possibility to set up your CodeCatalyst environment through Cloudformation/CLI calls or through CDK/Terraform and I am unsure if that would be something that is required for this service. What would definitely be good to have is the possibility to set up integrations to your existing accounts and projects in an automated form – the CLI commands make that possible, but that’s not the same as directly defining it “as code”.
Native branch support for projects & workflows
CodeCatalyst allows to refer to different branches, but it would be cool to get a possibility to natively support “ephemeral” environments as part of your CI/CD process. There are two use cases that I have in mind:
Integration tests as part of the CI/CD process (workflow) that require the infrastructure to be available and provisioned
Environments to replicate/test defects or new functionality
For both of these use cases, the CI/CD tool should be able to deploy an environment from a branch – and then also automatically de-provision it based on rules. Today this is something that you would need to manually replicate within your workflows and infrastructure as code.
Overall summary and outlook
The service is overall is a great addition (or maybe a replacement?) of the existing Code* tools on AWS and it streamlines a lot of things that were previously not working perfectly on AWS. For a new service it’s a solid launch with a lot of benefits, especially for small companies or open source projects that do not have the possibility to use existing third party tools or really just want to focus on generating business value. I do not think that CodeCatalyst will be able to replace existing Jenkins installations or services for bigger enterprises right now, as there are a few things that are not yet possible.
What do you think of CodeCatalyst? Please share your thoughts and feedback with me (either on the comments, on LinkedIn or by mail.
At re:Invent 2022 in Las Vegas AWS has announced a new feature for AWS Lambda that allows you to reduce your lambda startup times for Java based functions – SnapStart. With this post you are going to understand how the feature works, how it can be enabled, how you will benefit from it and how it will reduce your costs for AWS lambda. The functionality needs to be activated (“opt-in”) and has a few pre-requites for your functions that I will also share with you. It is currently only available if you use Java (Corretto 11) as a runtime. Other runtimes (e.g. Python, Typescript) will be available at a later date.
What is AWS Lambda SnapStart – and reduced latency for cold starts?
SnapStart is a new feature that helps to improve the “cold starts” of your lambda function.
AWS Lambda is a serverless possibility to execute code or a part of your application on AWS. The service can be used with different languages – Typescript/Node.JS, Python, Golang, … and Java. Lambda can be seen as “function as a service” with very convenient integration possibilities. Under the hood of Lambda, AWS has a very quick and mature provisioning system that brings up a container that executes your code. AWS Lambda as a “serverless” service allows you to not pay anything for your deployed infrastructure if it is not being used – this is also known as “scale-to-zero”. AWS Lambda is billed by “usage time” of the capacity that your lambda functions use – a lambda function can run up to 15min, but most of the lambda functions execute within a few seconds. Because of the “scale-to-zero” functionality, there is not always a “running” version of your code on AWS. This is where AWS distinguishes between “cold starts” and “warm starts” for your lambda function.
When SnapStart is enabled, function code is initialized once when a function version is published. Lambda then takes a snapshot of the memory and disk state of the initialized execution environment, persists the encrypted snapshot, and caches it for low-latency access. When the function is first invoked or subsequently scaled, Lambda resumes new execution environments from the persisted snapshot instead of initializing from scratch, avoiding several seconds of variable latency.
So – why is SnapStart an “opt-in” feature?
AWS has decided to not make SnapStart a default functionality of AWS Lambda, because there are certain pre-requisites for being able to use this new functionality.
The most important one is that the code executed as part of the lambda function cannot rely on the “randomization” features of Java, and this has implications for the code that you write. Please read the SnapChart documentation for further details – the FAQ is very detailed on this. This does not only include the code that YOU write as part of your lambda function but also the code of every library that you use as part of your code.
Other things to consider are that you cannot rely on network connections that you created/set up during the initialization of your code to be still valid and available when your function is resumed from a snapshort. This means that you will need to verify any network connections (e.g. connections for a backend database) before you actually use it.
Any ephemeral data that you rely on can be invalid when your function is resumed from a snapshot – and this means that you will need to ensure that any ephemeral data is still valid if you want to access it. This could be temporary credentials, machine learning models or just temporary data that you use within your function.
In it’s documentation, AWS also provides a bunch of examples and best practices to make sure that your code is “SnapSafe” and can work with this new functionality without causing problems.
How do you activate AWS Lambda SnapStart?
You can activate AWS Lambda either in the AWS console or through the AWS cli. Surprisingly this feature also comes with direct CloudFormation, CDK and SDK support. Please remember that after activating it, you will need to create a new version of lambda function that you want to test.
Further details can be found in the AWS documentation.
Let’s go back to the “teaser” of this post – How can YOU safe costs with this functionality?
For Lambda functions you are based on the number of requests for your functions, the allocated memory and the time that it takes to execute your java code. The duration is calculated from the time your code begins executing until it returns or otherwise terminated, rounded up to the nearest 1ms.
The total duration is a combination of the initialization code (in the constructor of your function) and the code in the “handler”. With SnapStart enabled, the “initialization code” / phase is moved to the provisioning of the function (when you set it up) – and when the function is executed the “initialization phase” is replaced by re-starting the function from the already created snapshot.
This means that for functions, that have an intense “initialization” phase the potential savings are bigger than for functions that run all of their code in the “handler”. More details around how to optimize your Java functions for cold starts can be found in this blog.
Let’s do a few calculations on how this helps us to safe money:
The table assumes that the initialization phase is reduced to 25 milliseconds (which is not technically correct according to the AWS documentation, but its as close as I can guess today).
What this table shows is that the SnapStart feature will help us to reduce our costs for AWS Lambda significantly, if we have Lambda functions that have a longer initialization time than “handler” execution time.
Overall I calculated between 10% and 60% savings with this feature, depending on your use case. Please be aware: This is calculation/guessing only, I have not yet measured that during a test! For further analysis and details you will need to analyze your current performance metrics of your lambda functions and perform additional tests and calculations. Before you can use the feature, as already mentioned, you will need to verify that your lambda code is “SnapSafe”.
If you try out this new feature and gain meaningful experiences out of it, please let me know and reach out to me in the comments section, through LinkedIn or E-Mail.
