Loading policy/tpa-rfc-10-jenkins-retirement.md +87 −27 Original line number Diff line number Diff line Loading @@ -4,18 +4,22 @@ title: TPA-RFC-10: Jenkins retirement Summary: Jenkins will be retired in 2021, replaced mostly by GitLab CI, with special hooks to keep the static site mirror system operational. and Debian package builds operational. # Background [Jenkins][Jenkins CI] was a fine piece of software when it came out: builds! We [Jenkins][] was a fine piece of software when it came out: builds! We can easily do builds! On multiple machines too! And a nice web interface with [weird blue balls](https://www.jenkins.io/blog/2012/03/13/why-does-jenkins-have-blue-balls/)! It was great. But then [Travis CI](https://travis-ci.org/) came along, and then GitLab CI, and then GitHub actions, and it interface with [weird blue balls][]! It was great. But then [Travis CI][] came along, and then GitLab CI, and then GitHub actions, and it turns out it's much, much easier and intuitive to delegate the build configuration to the project as opposed to keeping it in the CI system. [Jenkins CI]: https://en.wikipedia.org/wiki/Jenkins_(software) [Travis CI]: https://travis-ci.org/ [weird blue balls]: https://www.jenkins.io/blog/2012/03/13/why-does-jenkins-have-blue-balls/ The design of Jenkins, in other words, feels dated now. It imposes an unnecessary burden on the service admins, which are responsible for configuring and monitoring builds for their users. Introducing a job Loading @@ -32,13 +36,17 @@ In the short term, Jenkins can keep doing what it does, but in the long term, we would greatly benefit from retiring yet another service, since it basically duplicates what GitLab CI already does. This section tries to establish alternatives to the venerable Jenkins. Note that the [2020 user survey][] also had a few voices suggesting that Jenkins be retired in favor of GitLab CI. Some users also expressed "sadness" with the Jenkins service. Those results were the main driver behind this proposal. [Jenkins CI]: https://en.wikipedia.org/wiki/Jenkins_(software) [2020 user survey]: roadmap/2021#survey-results ## Goals The goal of this migration is to retire the Jenkins service and server The goal of this migration is to retire the Jenkins service and servers (`henryi` but also the multiple `build-$ARCH-$NN` servers) with minimal disruption to its users. ### Must have Loading @@ -50,23 +58,74 @@ with minimal disruption to its users. ### Nice to have * full and transparent integration with GitLab * retire all the existing `build-$ARCH-$NN` machines in favor of the GitLab CI runners architecture ### Non-Goals * replacing the current static mirror system is out of scope * retiring the gitolite / gitweb infrastructure is out of scope, even though it is planned as part of the 2021 roadmap. therefore solutions here should not rely too much on gitolite-specific features or hooks * replacing the current static mirror system is out of scope, and is not planned in the 2021 roadmap at all, so solutions proposed must still be compatible with the static site mirror system # Proposal Replacing Jenkins should be done progressively. Some teams have already started using GitLab CI to run some jobs, so it is the obvious and most likely candidate. Some jobs will be harder to migrate than others, so a piecemeal approach will be necessary. Replacing Jenkins will be done progressively, over the course of 2021, by the different Jenkins users themselves. TPA will coordinate the effort and progressively remove jobs from the Jenkins configuration until none remain, at which point the server -- along with the build boxes -- will be retired. No archive of the service will be kept. ## GitLab Ci as main option, and alternatives GitLab will be suggested as an alternative for Jenkins users, but users will be free to implement their own build system in other ways if they do not feel GitLab CI is a good fit for their purpose. In particular, GitLab has a powerful web hook system that can be used to trigger builds on other infrastructure. Alternatively, external build systems could periodically pull Git repositories for changes. ## Stakeholders and responsibilities We know of the following teams currently using Jenkins and affected by this: * **web team**: virtually all websites are built in Jenkins, and heavily depend on the static site mirror for proper performance * **network team**: the core tor project is also a heavy user of Jenkins, mostly to run tests and checks, but also producing some artefacts (Debian packages and documentation) * **TPA**: uses Jenkins to build the status website * **metrics team**: onionperf's documentation is built in Jenkins When this proposal is adopted, a ticket will be created to track all the jobs configured in Jenkins and each team will be responsible to migrate their jobs before the deadline. It is *not* up to TPA to rebuild those pipelines, as this would be too time-consuming and would require too much domain-specific knowledge. Besides, it's important that teams become familiar with the GitLab CI system so this is a good opportunity to do so. When a job has been migrated, it should be marked as such in this wiki page (or a ticket?). A more detailed analysis of the jobs currently configured in Jenkins is available in [the Configured Jobs section of the Jenkins service documentation][]. [the Configured Jobs section of the Jenkins service documentation]: service/jenkins#configured-jobs ## Specific job recommendations With the above in mind, here are some recommendation on specific group of jobs currently configured on the Jenkins server and how they could be migrated to the GitLab CI infrastructure. Some jobs will be harder to migrate than others, so a piecemeal approach will be used. Here's a breakdown by job type, from easiest to hardest: Loading @@ -85,9 +144,11 @@ to GitLab CI. Hopefully this should be a fairly low-hanging fruit... ### Windows CI tests GitLab CI will eventually gain Windows (and Mac!) based runners (see [issue 40095](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095)) which *should* be able to replace the Windows CI [issue 40095][]) which *should* be able to replace the Windows CI jobs from Jenkins. [issue 40095]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095 ### Critical website builds Critical websites should be built by GitLab CI just like non-critical Loading Loading @@ -118,19 +179,14 @@ There are two possible solutions: TODO: pick a solution for Debian package builds. @weasel may have ideas. ## Scope ## ... # Examples Examples: * ... Counter examples: * ... * the network team is migrating their CI jobs to GitLab CI * the research.torproject.org site would end up as a GitLab pages site * the www.torproject.org site would stay in the static mirror system, but would be built in GitLab CI # Deadline Loading @@ -155,6 +211,10 @@ This proposal is currently in the `draft` state. # References See the [GitLab](howto/gitlab), [GitLab CI](service/ci), and [Jenkins service documentation](service/jenkins) for more background on how See the [GitLab][], [GitLab CI][], and [Jenkins service documentation][] for more background on how Jenkins and GitLab CI work. [Jenkins service documentation]: service/jenkins [GitLab CI]: service/ci [GitLab]: howto/gitlab Loading
policy/tpa-rfc-10-jenkins-retirement.md +87 −27 Original line number Diff line number Diff line Loading @@ -4,18 +4,22 @@ title: TPA-RFC-10: Jenkins retirement Summary: Jenkins will be retired in 2021, replaced mostly by GitLab CI, with special hooks to keep the static site mirror system operational. and Debian package builds operational. # Background [Jenkins][Jenkins CI] was a fine piece of software when it came out: builds! We [Jenkins][] was a fine piece of software when it came out: builds! We can easily do builds! On multiple machines too! And a nice web interface with [weird blue balls](https://www.jenkins.io/blog/2012/03/13/why-does-jenkins-have-blue-balls/)! It was great. But then [Travis CI](https://travis-ci.org/) came along, and then GitLab CI, and then GitHub actions, and it interface with [weird blue balls][]! It was great. But then [Travis CI][] came along, and then GitLab CI, and then GitHub actions, and it turns out it's much, much easier and intuitive to delegate the build configuration to the project as opposed to keeping it in the CI system. [Jenkins CI]: https://en.wikipedia.org/wiki/Jenkins_(software) [Travis CI]: https://travis-ci.org/ [weird blue balls]: https://www.jenkins.io/blog/2012/03/13/why-does-jenkins-have-blue-balls/ The design of Jenkins, in other words, feels dated now. It imposes an unnecessary burden on the service admins, which are responsible for configuring and monitoring builds for their users. Introducing a job Loading @@ -32,13 +36,17 @@ In the short term, Jenkins can keep doing what it does, but in the long term, we would greatly benefit from retiring yet another service, since it basically duplicates what GitLab CI already does. This section tries to establish alternatives to the venerable Jenkins. Note that the [2020 user survey][] also had a few voices suggesting that Jenkins be retired in favor of GitLab CI. Some users also expressed "sadness" with the Jenkins service. Those results were the main driver behind this proposal. [Jenkins CI]: https://en.wikipedia.org/wiki/Jenkins_(software) [2020 user survey]: roadmap/2021#survey-results ## Goals The goal of this migration is to retire the Jenkins service and server The goal of this migration is to retire the Jenkins service and servers (`henryi` but also the multiple `build-$ARCH-$NN` servers) with minimal disruption to its users. ### Must have Loading @@ -50,23 +58,74 @@ with minimal disruption to its users. ### Nice to have * full and transparent integration with GitLab * retire all the existing `build-$ARCH-$NN` machines in favor of the GitLab CI runners architecture ### Non-Goals * replacing the current static mirror system is out of scope * retiring the gitolite / gitweb infrastructure is out of scope, even though it is planned as part of the 2021 roadmap. therefore solutions here should not rely too much on gitolite-specific features or hooks * replacing the current static mirror system is out of scope, and is not planned in the 2021 roadmap at all, so solutions proposed must still be compatible with the static site mirror system # Proposal Replacing Jenkins should be done progressively. Some teams have already started using GitLab CI to run some jobs, so it is the obvious and most likely candidate. Some jobs will be harder to migrate than others, so a piecemeal approach will be necessary. Replacing Jenkins will be done progressively, over the course of 2021, by the different Jenkins users themselves. TPA will coordinate the effort and progressively remove jobs from the Jenkins configuration until none remain, at which point the server -- along with the build boxes -- will be retired. No archive of the service will be kept. ## GitLab Ci as main option, and alternatives GitLab will be suggested as an alternative for Jenkins users, but users will be free to implement their own build system in other ways if they do not feel GitLab CI is a good fit for their purpose. In particular, GitLab has a powerful web hook system that can be used to trigger builds on other infrastructure. Alternatively, external build systems could periodically pull Git repositories for changes. ## Stakeholders and responsibilities We know of the following teams currently using Jenkins and affected by this: * **web team**: virtually all websites are built in Jenkins, and heavily depend on the static site mirror for proper performance * **network team**: the core tor project is also a heavy user of Jenkins, mostly to run tests and checks, but also producing some artefacts (Debian packages and documentation) * **TPA**: uses Jenkins to build the status website * **metrics team**: onionperf's documentation is built in Jenkins When this proposal is adopted, a ticket will be created to track all the jobs configured in Jenkins and each team will be responsible to migrate their jobs before the deadline. It is *not* up to TPA to rebuild those pipelines, as this would be too time-consuming and would require too much domain-specific knowledge. Besides, it's important that teams become familiar with the GitLab CI system so this is a good opportunity to do so. When a job has been migrated, it should be marked as such in this wiki page (or a ticket?). A more detailed analysis of the jobs currently configured in Jenkins is available in [the Configured Jobs section of the Jenkins service documentation][]. [the Configured Jobs section of the Jenkins service documentation]: service/jenkins#configured-jobs ## Specific job recommendations With the above in mind, here are some recommendation on specific group of jobs currently configured on the Jenkins server and how they could be migrated to the GitLab CI infrastructure. Some jobs will be harder to migrate than others, so a piecemeal approach will be used. Here's a breakdown by job type, from easiest to hardest: Loading @@ -85,9 +144,11 @@ to GitLab CI. Hopefully this should be a fairly low-hanging fruit... ### Windows CI tests GitLab CI will eventually gain Windows (and Mac!) based runners (see [issue 40095](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095)) which *should* be able to replace the Windows CI [issue 40095][]) which *should* be able to replace the Windows CI jobs from Jenkins. [issue 40095]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095 ### Critical website builds Critical websites should be built by GitLab CI just like non-critical Loading Loading @@ -118,19 +179,14 @@ There are two possible solutions: TODO: pick a solution for Debian package builds. @weasel may have ideas. ## Scope ## ... # Examples Examples: * ... Counter examples: * ... * the network team is migrating their CI jobs to GitLab CI * the research.torproject.org site would end up as a GitLab pages site * the www.torproject.org site would stay in the static mirror system, but would be built in GitLab CI # Deadline Loading @@ -155,6 +211,10 @@ This proposal is currently in the `draft` state. # References See the [GitLab](howto/gitlab), [GitLab CI](service/ci), and [Jenkins service documentation](service/jenkins) for more background on how See the [GitLab][], [GitLab CI][], and [Jenkins service documentation][] for more background on how Jenkins and GitLab CI work. [Jenkins service documentation]: service/jenkins [GitLab CI]: service/ci [GitLab]: howto/gitlab
