We just finished migrating the ActiveUI development process to the Cloud.
Our new tools and workflow allow us to focus on improving our product instead of spending time administrating auxiliary software and their associated servers.
More importantly, they help us guarantee the high quality of any addition to the codebase while at the same time making development easier, faster, and more enjoyable.
Here’s the story of how we went from a fully self-hosted tooling to a workflow relying on best-in-class service providers.
In 2015, we hosted everything in our Paris data center:
- Our Git repository was hosted in Gitolite.
- Our nightly builds and releases were triggered from Jenkins.
- Our artifacts were uploaded to an HTTP File Server ignorant of their Maven/npm package nature.
We don’t maintain any of this infrastructure ourselves anymore. Instead, here is our lineup for 2019:
- GitHub is our central Git server. All changes are made through pull requests.
- Continuous deployment is handled by CircleCI.
- Artifacts (npm packages, Maven packages, etc.) are deployed to their corresponding repository type in Artifactory.
Tools
We started by reviewing the available services, picking one for each category, and migrating to it.
Gitolite to GitHub
This is the first move we made. We wanted to do code review on pull requests instead of going to sit next to another coworker to review a suggested code change on a single shared screen. We picked GitHub for its quality, familiarity, and evergrowing set of features. Its popularity was also a plus because we wanted a well-known platform to share our open source work:
Jenkins to CircleCI
Jenkins is very extensible and has a big plugin ecosystem. It’s a double-edged sword: plugins can be installed and configured to the team’s specific needs but using lots of them increases the probability of breakage for each Jenkins update. This is how our “Manage Jenkins” page looked right before we stopped using it:
We picked CircleCI over Travis CI for its cheaper plan, ability to easily debug jobs by connecting to the VMs running them with SSH, and free hosting of build artifacts. They’re pretty close to each other in terms of features though, and switching from one to another shouldn’t be too hard if we feel the need to in the future.
HTTP File Server to Artifactory
All the artifacts created by the company’s projects were uploaded, through SFTP, as static files in a directory structure served by Apache HTTP Server and protected with basic authentication by Atlassian Crowd. However, some of our artifacts are .tgz archives of npm packages or .jar of Maven packages.
Consuming such packages from a generic HTTP server isn’t straightforward. The npm / mvn CLIs don’t know how to authenticate against it and how to download packages from it.
To make it simple for internal projects to depend on each other, we wanted to use a service providing private npm and Maven registries. We didn’t want to use npm Enterprise or Nexus Repo Pro because of their priced per user policy. Indeed, artifact repositories are not collaborative platforms such as GitHub or JIRA, it makes more sense to pay for them based on the storage or bandwidth used. So we picked Artifactory Cloud Pro X instead, with its high SLA and embedded JFrog Xray for security vulnerability and license analysis. Here’s a screenshot of what an npm package published on Artifactory looks like in their web app:
Workflows
With these new tools adopted, we were able to tweak our workflow to make it really efficient and satisfying.
Design sessions
Before starting implementing any new feature or significant change to ActiveUI, we schedule a design session with our lead UX designer and 2 or 3 engineers. The results of these sessions are written down in a Markdown document often linking to screenshots, mock-ups, or RealTime Boards. All these assets are pushed to our activeui-design GitHub repository as a pull request. This is where we iterate on the design and gather feedback from other ActiveViam employees. Once the corresponding development lands on ActiveUI, we merge the pull request. It’s similar to what the React and Rust teams do with their RFC repositories: https://github.com/reactjs/rfcs and https://github.com/rust-lang/rfcs.
Pull requests
Every change to the ActiveUI codebase goes through a GitHub pull request.
Git branches attached to a pull request are continuously built and tested on CircleCI. We use workflows to split CI builds into several jobs running concurrently in separate containers. These jobs run:
- Our Jest-based unit test suite.
- Our Cypress-based end-to-end test suite.
- Code analysis and formatting tools: Prettier, ESLint, and Flow.
A complete build usually lasts around 10 minutes after which a bot posts a comment on the pull request containing links to artifacts that can be used by the reviewer to quickly see the impact of the suggested changes:
The Showcase artifact linked in the comment above is a web application based on ActiveUI SDK. It connects, by default, to a beefy Microsoft Azure VM running multiple versions of our backend: ActivePivot Server. It helps us quickly test the changes introduced by the pull request against all the versions of ActivePivot we need to support.
Code reviews
Every pull request on ActiveUI needs to be reviewed an approved by another team member before being merged. We rely on GitHub’s CODEOWNERS support for that.
master always green
Back when we didn’t have code reviews and pull requests, the nightly build would sometimes fail on Jenkins. You would start your day by seeing “[R&D] Build failed in Jenkins: ActiveUI-nightly” in your inbox and having to hastily find the cause and push a fix on the master branch. Not the best morning ever!
That’s why, after we made the switch to GitHub, its pull requests and code reviews, our next goal was to “keep master always green”. We spent some time increasing our code coverage by writing more unit, integration, and end-to-end tests. Now we’re at a point where we’re confident that a pull request won’t break anything too obvious once reviewed and merged. That’s why we actually never run our test suites on master anymore.
Note: One important GitHub option to enable to “keep master always green” is “Require branches to be up to date before merging”. We use the Autorebase GitHub App to help us with that.
Continuous deployment
Once we knew master would always be in a correct state, we decided to stop doing nightly builds and switched to continuous builds instead. Every pull request merged on master triggers a CircleCI workflow that builds the whole project and deploys its artifacts to Artifactory. Like this, other internal projects never have to wait a day to start using the latest version of ActiveUI!
Automated releases
Official releases of ActiveUI are similar to continuous builds but they:
- Target different Artifactory repositories.
- Upload our documentation to an Amazon S3 bucket.
- Flag the corresponding version as released in JIRA.
- Push some Git commits to different branches and create Git tags.
We trigger them from CircleCI too. Every push to master has the potential of starting a release. We use CircleCI’s manual approval feature to hold release workflows until someone in the team actually decides to cut a release:
The last step of our release script is to announce the new version on Slack so that people in the company know about it:
Release candidates
We release a new stable version every month. The week before releasing a stable version, we usually release one or two release candidates. They are mostly used by internal projects to make sure that the upgrade to the latest version is smooth. During that week, we backport bug-fix pull requests merged on master to another Git branch called prerelease. We use this prerelease branch as the starting point for the stable release. This means that new features and significant refactorings can still be merged on master without threatening the stability of the final release.
Conclusion
We’re convinced that these Cloud-based services, and the automated workflow we have built around them, tightened our feedback loop and thus made developments more interactive. They allow us to rapidly deliver high-quality new features and improvements to ActiveUI. They also minimize context-switching and small yet tedious manual operations. This helps us stay focused on the kind of more interesting work that brings the most value to our product.
We encourage you to spend some time streamlining your own workflow too. It will make your developments faster, your product better, and your team happier!
