From 6cb349e989837a78cdaa1e557194473a20361bba Mon Sep 17 00:00:00 2001 From: David Goulet Date: Mon, 1 Nov 2021 15:29:59 -0400 Subject: [PATCH 1/2] readme: CI Pipeline status icon at the top Signed-off-by: David Goulet --- README | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README b/README index 397f6b927e..a1421e5186 100644 --- a/README +++ b/README @@ -1,3 +1,5 @@ +[![pipeline status](https://gitlab.torproject.org/tpo/core/tor/badges/main/pipeline.svg)](https://gitlab.torproject.org/tpo/core/tor/-/commits/main) + Tor protects your privacy on the internet by hiding the connection between your Internet address and the services you use. We believe Tor is reasonably secure, but please ensure you read the instructions and From c9e2ee076cafd694e0de664d3fc8f1ab1216ceb1 Mon Sep 17 00:00:00 2001 From: David Goulet Date: Tue, 2 Nov 2021 11:56:50 -0400 Subject: [PATCH 2/2] doc: Brand new ReleasingTor.md Closes #40508 Signed-off-by: David Goulet --- doc/HACKING/ReleasingTor.md | 261 +++++++++----------------------- doc/HACKING/ReleasingTor.md.old | 255 +++++++++++++++++++++++++++++++ 2 files changed, 323 insertions(+), 193 deletions(-) create mode 100644 doc/HACKING/ReleasingTor.md.old diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index 490c100fcb..12f5d1b16b 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -1,239 +1,114 @@ -# CHECKLIST - -Here's a summary checklist, with the things that Nick messes up most often. - -Did you: - - * [ ] Copy the ChangeLog to the ReleaseNotes? - * [ ] Check that the new versions got approved? - * [ ] Check the release date in the ChangeLog? - * [ ] Update the GeoIP file? - -# Putting out a new release +# How to Release Tor Here are the steps that the maintainer should take when putting out a -new Tor release: +new Tor release. It is split in 3 stages and coupled with our Tor CI Release +pipeline. -## 0. Preliminaries +Before we begin, first rule is to make sure: -1. Get at least three of weasel/arma/Sebastian/Sina to put the new - version number in their approved versions list. Give them a few - days to do this if you can. + - Our CI pass for each version to release + - Coverity has no new alerts -2. If this is going to be an important security release, give the packagers - advance warning, via `tor-packagers@lists.torproject.org`. +## 0. Security Release +To start with, if you are doing a security release, this must be done few days +prior to the release: -3. Given the release date for Tor, ask the TB team about the likely release - date of a TB that contains it. See note below in "commit, upload, - announce". + 1. If this is going to be an important security release, give the packagers + advance warning, via `tor-packagers@lists.torproject.org`. -## I. Make sure it works -1. Make sure that CI passes: have a look at the branches on gitlab. +## 1. Preliminaries - _Optionally_, have a look at Travis - (https://travis-ci.org/torproject/tor/branches), Appveyor - (https://ci.appveyor.com/project/torproject/tor/history), and - Jenkins (https://jenkins.torproject.org/view/tor/). - Make sure you're looking at the right branches. +The following must be done 2 days at the very least prior to the release: - If there are any unexplained failures, try to fix them or figure them - out. + 1. Add the version(s) in the dirauth-conf git repository as the + RecommendedVersion and RequiredVersion so they can be approved by the + authorities and be in the consensus before the release. -2. Verify that there are no big outstanding issues. You might find such - issues -- + 2. Send a pre-release announcement to `tor-project@lists.torproject.org` in + order to inform every teams in Tor of the upcoming release. This is so + we can avoid creating release surprises and sync with other teams. - * On Gitlab + 3. Ask the network-team to review the `changes/` files in all versions we + are about to release. This step is encouraged but not mandatory. - * On coverity scan - * On OSS-Fuzz +## 2. Tarballs -## II. Write a changelog +To build the tarballs to release, we need to launch the CI release pipeline: + https://gitlab.torproject.org/tpo/core/tor-ci-release -1a. (Alpha release variant) +The `versions.yml` needs to be modified with the Tor versions you want to +release. Once done, git commit and push to trigger the release pipeline. - Gather the `changes/*` files into a changelog entry, rewriting many - of them and reordering to focus on what users and funders would find - interesting and understandable. +The first two stages (Preliminary and Patches) will be run automatically. The +Build stage needs to be triggered manually once all generated patches have +been merged upstream. - To do this, run `./scripts/maint/sortChanges.py changes/* > changelog.in` - to combine headings and sort the entries. Copy the changelog.in file into - the ChangeLog. Run `format_changelog.py --inplace` (see below) to clean up - the line breaks. + 1. Download the generated patches from the `Patches` stage. - Remove the `changes/*` files that you just merged into the ChangeLog. + 2. For the ChangeLog and ReleaseNotes, you need to write a blurb at the top + explaining a bit the release. - After that, it's time to hand-edit and fix the issues that - lintChanges can't find: + 3. Review, modify if needed, and merged them upstream. - 1. Within each section, sort by "version it's a bugfix on", else by - numerical ticket order. + 4. Manually trigger the `maintained` job in the `Build` stage so the CI can + build the tarballs without errors. - 2. Clean them up: +Once this is done, each selected developers need to build the tarballs in a +reproducible way using: - Make stuff very terse + https://gitlab.torproject.org/tpo/core/tor-ci-reproducible - Describe the user-visible problem right away +Simply run the `./build.sh` which will commit interactively the signature for +you. You then only need to git push. - Mention relevant config options by name. If they're rare or unusual, - remind people what they're for +Once all signatures have been committed: - Avoid starting lines with open-paren + 1. Manually trigger the `signature` job in the `Post-process` stage of the + CI release pipeline. - Present and imperative tense: not past. + 2. If it passes, the tarball(s) and signature(s) will be available as + artifacts and should be used for the release. - "Relays", not "servers" or "nodes" or "Tor relays". + 3. Put them on `dist.torproject.org`: - "Onion services", not "hidden services". + Upload the tarball and its sig to the dist website, i.e. + `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run + "static-update-component dist.torproject.org" on dist-master. - "Stop FOOing", not "Fix a bug where we would FOO". + In the `project/web/tpo.git` repository, update `databags/versions.ini` + to note the new version. Push these changes to `master`. - Try not to let any given section be longer than about a page. Break up - long sections into subsections by some sort of common subtopic. This - guideline is especially important when organizing Release Notes for - new stable releases. + (NOTE: Due to #17805, there can only be one stable version listed at once. + Nonetheless, do not call your version "alpha" if it is stable, or people + will get confused.) - If a given changes stanza showed up in a different release (e.g. - maint-0.2.1), be sure to make the stanzas identical (so people can - distinguish if these are the same change). + (NOTE: It will take a while for the website update scripts to update the + website.) - 3. Clean everything one last time. - 4. Run `./scripts/maint/format_changelog.py --inplace` to make it prettier +## 3. Post Process -1b. (old-stable release variant) +Once the tarballs have been uploaded and are ready to be announced, we need to +do the following: - For stable releases that backport things from later, we try to compose - their releases, we try to make sure that we keep the changelog entries - identical to their original versions, with a "backport from 0.x.y.z" - note added to each section. So in this case, once you have the items - from the changes files copied together, don't use them to build a new - changelog: instead, look up the corrected versions that were merged - into ChangeLog in the main branch, and use those. + 1. Merge upstream the artifacts from the `patches` job in the + `Post-process` stage of the CI release pipeline. - Add "backport from X.Y.Z" in the section header for these entries. + 2. Write and post the release announcement for the `forum.torproject.net` + in the `News -> Tor Release Announcement` category. -2. Compose a short release blurb to highlight the user-facing - changes. Insert said release blurb into the ChangeLog stanza. If it's - a stable release, add it to the ReleaseNotes file too. If we're adding - to a release-* branch, manually commit the changelogs to the later - git branches too. + Mention in which Tor Browser version (with dates) the release will be + in. This usually only applies to the latest stable. -3. If there are changes that require or suggest operator intervention - before or during the update, mail operators (either dirauth or relays - list) with a headline that indicates that an action is required or - appreciated. +### New Stable -4. If you're doing the first stable release in a series, you need to - create a ReleaseNotes for the series as a whole. To get started - there, copy all of the Changelog entries from the series into a new - file, and run `./scripts/maint/sortChanges.py` on it. That will - group them by category. Then kill every bugfix entry for fixing - bugs that were introduced within that release series; those aren't - relevant changes since the last series. At that point, it's time - to start sorting and condensing entries. (Generally, we don't edit the - text of existing entries, though.) + 1. Create the `maint-x.y.z` and `release-x.y.z` branches and update the + `./scripts/git/git-list-tor-branches.sh` with the new version. -## III. Making the source release. - -1. In `maint-0.?.x`, bump the version number in `configure.ac` and run - `./scripts/main/update_versions.py` to update version numbers in other - places, and commit. Then merge `maint-0.?.x` into `release-0.?.x`. - - When you merge the maint branch forward to the next maint branch, or into - main, merge it with `-s ours` to avoid conflict with the version - bump. - -2. In `release-0.?.x`, run `make distcheck`, put the tarball up in somewhere - (how about your homedir on people.torproject.org?) , and tell `#tor-dev` - about it. - - If you want, wait until at least one person has built it - successfully. (We used to say "wait for others to test it", but our - CI has successfully caught these kinds of errors for the last several - years.) - -3. Make sure that the new version is recommended in the latest consensus. - (Otherwise, users will get confused when it complains to them - about its status.) - - If it is not, you'll need to poke Roger, Weasel, Sebastian, and Sina - again: see the note at the start of the document. - -## IV. Commit, upload, announce - -1. Sign the tarball, then sign and push the git tag: - -```console -$ gpg -ba -$ git tag -s tor-0.4.x.y- -$ git push origin tag tor-0.4.x.y- -``` - - (You must do this before you update the website: the website scripts - rely on finding the version by tag.) - - (If your default PGP key is not the one you want to sign with, then say - "-u " instead of "-s".) - -2. scp the tarball and its sig to the dist website, i.e. - `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run - "static-update-component dist.torproject.org" on dist-master. - - In the `project/web/tpo.git` repository, update `databags/versions.ini` - to note the new version. Push these changes to `master`. - - (NOTE: Due to #17805, there can only be one stable version listed at - once. Nonetheless, do not call your version "alpha" if it is stable, - or people will get confused.) - - (NOTE: It will take a while for the website update scripts to update - the website.) - -3. Email the tor-packagers@lists.torproject.org mailing list to tell them - about the new release. - - Also, email tor-packagers@lists.torproject.org. - - Mention where to download the tarball (`https://dist.torproject.org/`). - - Include a link to the changelog. - -4. Wait for the download page to be updated. (If you don't do this before you - announce, people will be confused.) - -5. Mail the release blurb and ChangeLog to tor-talk (development release) or - tor-announce (stable). - - Post the changelog on the blog as well. You can generate a - blog-formatted version of the changelog with - `./scripts/maint/format_changelog.py -B` - - When you post, include an estimate of when the next TorBrowser - releases will come out that include this Tor release. This will - usually track https://wiki.mozilla.org/RapidRelease/Calendar , but it - can vary. - - For templates to use when announcing, see: - https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/AnnouncementTemplates - -## V. Aftermath and cleanup - -1. If it's a stable release, bump the version number in the - `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours` - merge to avoid taking that change into main. - -2. If there is a new `maint-x.y.z` branch, create a Travis CI cron job that - builds the release every week. (It's ok to skip the weekly build if the - branch was updated in the last 24 hours.) - -3. Forward-port the ChangeLog (and ReleaseNotes if appropriate) to the - main branch. - -4. Keep an eye on the blog post, to moderate comments and answer questions. ## Appendix: An alternative means to notify packagers diff --git a/doc/HACKING/ReleasingTor.md.old b/doc/HACKING/ReleasingTor.md.old new file mode 100644 index 0000000000..490c100fcb --- /dev/null +++ b/doc/HACKING/ReleasingTor.md.old @@ -0,0 +1,255 @@ +# CHECKLIST + +Here's a summary checklist, with the things that Nick messes up most often. + +Did you: + + * [ ] Copy the ChangeLog to the ReleaseNotes? + * [ ] Check that the new versions got approved? + * [ ] Check the release date in the ChangeLog? + * [ ] Update the GeoIP file? + +# Putting out a new release + +Here are the steps that the maintainer should take when putting out a +new Tor release: + +## 0. Preliminaries + +1. Get at least three of weasel/arma/Sebastian/Sina to put the new + version number in their approved versions list. Give them a few + days to do this if you can. + +2. If this is going to be an important security release, give the packagers + advance warning, via `tor-packagers@lists.torproject.org`. + + +3. Given the release date for Tor, ask the TB team about the likely release + date of a TB that contains it. See note below in "commit, upload, + announce". + +## I. Make sure it works + +1. Make sure that CI passes: have a look at the branches on gitlab. + + _Optionally_, have a look at Travis + (https://travis-ci.org/torproject/tor/branches), Appveyor + (https://ci.appveyor.com/project/torproject/tor/history), and + Jenkins (https://jenkins.torproject.org/view/tor/). + Make sure you're looking at the right branches. + + If there are any unexplained failures, try to fix them or figure them + out. + +2. Verify that there are no big outstanding issues. You might find such + issues -- + + * On Gitlab + + * On coverity scan + + * On OSS-Fuzz + +## II. Write a changelog + + +1a. (Alpha release variant) + + Gather the `changes/*` files into a changelog entry, rewriting many + of them and reordering to focus on what users and funders would find + interesting and understandable. + + To do this, run `./scripts/maint/sortChanges.py changes/* > changelog.in` + to combine headings and sort the entries. Copy the changelog.in file into + the ChangeLog. Run `format_changelog.py --inplace` (see below) to clean up + the line breaks. + + Remove the `changes/*` files that you just merged into the ChangeLog. + + After that, it's time to hand-edit and fix the issues that + lintChanges can't find: + + 1. Within each section, sort by "version it's a bugfix on", else by + numerical ticket order. + + 2. Clean them up: + + Make stuff very terse + + Describe the user-visible problem right away + + Mention relevant config options by name. If they're rare or unusual, + remind people what they're for + + Avoid starting lines with open-paren + + Present and imperative tense: not past. + + "Relays", not "servers" or "nodes" or "Tor relays". + + "Onion services", not "hidden services". + + "Stop FOOing", not "Fix a bug where we would FOO". + + Try not to let any given section be longer than about a page. Break up + long sections into subsections by some sort of common subtopic. This + guideline is especially important when organizing Release Notes for + new stable releases. + + If a given changes stanza showed up in a different release (e.g. + maint-0.2.1), be sure to make the stanzas identical (so people can + distinguish if these are the same change). + + 3. Clean everything one last time. + + 4. Run `./scripts/maint/format_changelog.py --inplace` to make it prettier + +1b. (old-stable release variant) + + For stable releases that backport things from later, we try to compose + their releases, we try to make sure that we keep the changelog entries + identical to their original versions, with a "backport from 0.x.y.z" + note added to each section. So in this case, once you have the items + from the changes files copied together, don't use them to build a new + changelog: instead, look up the corrected versions that were merged + into ChangeLog in the main branch, and use those. + + Add "backport from X.Y.Z" in the section header for these entries. + +2. Compose a short release blurb to highlight the user-facing + changes. Insert said release blurb into the ChangeLog stanza. If it's + a stable release, add it to the ReleaseNotes file too. If we're adding + to a release-* branch, manually commit the changelogs to the later + git branches too. + +3. If there are changes that require or suggest operator intervention + before or during the update, mail operators (either dirauth or relays + list) with a headline that indicates that an action is required or + appreciated. + +4. If you're doing the first stable release in a series, you need to + create a ReleaseNotes for the series as a whole. To get started + there, copy all of the Changelog entries from the series into a new + file, and run `./scripts/maint/sortChanges.py` on it. That will + group them by category. Then kill every bugfix entry for fixing + bugs that were introduced within that release series; those aren't + relevant changes since the last series. At that point, it's time + to start sorting and condensing entries. (Generally, we don't edit the + text of existing entries, though.) + +## III. Making the source release. + +1. In `maint-0.?.x`, bump the version number in `configure.ac` and run + `./scripts/main/update_versions.py` to update version numbers in other + places, and commit. Then merge `maint-0.?.x` into `release-0.?.x`. + + When you merge the maint branch forward to the next maint branch, or into + main, merge it with `-s ours` to avoid conflict with the version + bump. + +2. In `release-0.?.x`, run `make distcheck`, put the tarball up in somewhere + (how about your homedir on people.torproject.org?) , and tell `#tor-dev` + about it. + + If you want, wait until at least one person has built it + successfully. (We used to say "wait for others to test it", but our + CI has successfully caught these kinds of errors for the last several + years.) + +3. Make sure that the new version is recommended in the latest consensus. + (Otherwise, users will get confused when it complains to them + about its status.) + + If it is not, you'll need to poke Roger, Weasel, Sebastian, and Sina + again: see the note at the start of the document. + +## IV. Commit, upload, announce + +1. Sign the tarball, then sign and push the git tag: + +```console +$ gpg -ba +$ git tag -s tor-0.4.x.y- +$ git push origin tag tor-0.4.x.y- +``` + + (You must do this before you update the website: the website scripts + rely on finding the version by tag.) + + (If your default PGP key is not the one you want to sign with, then say + "-u " instead of "-s".) + +2. scp the tarball and its sig to the dist website, i.e. + `/srv/dist-master.torproject.org/htdocs/` on dist-master. Run + "static-update-component dist.torproject.org" on dist-master. + + In the `project/web/tpo.git` repository, update `databags/versions.ini` + to note the new version. Push these changes to `master`. + + (NOTE: Due to #17805, there can only be one stable version listed at + once. Nonetheless, do not call your version "alpha" if it is stable, + or people will get confused.) + + (NOTE: It will take a while for the website update scripts to update + the website.) + +3. Email the tor-packagers@lists.torproject.org mailing list to tell them + about the new release. + + Also, email tor-packagers@lists.torproject.org. + + Mention where to download the tarball (`https://dist.torproject.org/`). + + Include a link to the changelog. + +4. Wait for the download page to be updated. (If you don't do this before you + announce, people will be confused.) + +5. Mail the release blurb and ChangeLog to tor-talk (development release) or + tor-announce (stable). + + Post the changelog on the blog as well. You can generate a + blog-formatted version of the changelog with + `./scripts/maint/format_changelog.py -B` + + When you post, include an estimate of when the next TorBrowser + releases will come out that include this Tor release. This will + usually track https://wiki.mozilla.org/RapidRelease/Calendar , but it + can vary. + + For templates to use when announcing, see: + https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/AnnouncementTemplates + +## V. Aftermath and cleanup + +1. If it's a stable release, bump the version number in the + `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours` + merge to avoid taking that change into main. + +2. If there is a new `maint-x.y.z` branch, create a Travis CI cron job that + builds the release every week. (It's ok to skip the weekly build if the + branch was updated in the last 24 hours.) + +3. Forward-port the ChangeLog (and ReleaseNotes if appropriate) to the + main branch. + +4. Keep an eye on the blog post, to moderate comments and answer questions. + +## Appendix: An alternative means to notify packagers + +If for some reason you need to contact a bunch of packagers without +using the publicly archived tor-packagers list, you can try these +people: + + - {weasel,sysrqb,mikeperry} at torproject dot org + - {blueness} at gentoo dot org + - {paul} at invizbox dot io + - {vincent} at invizbox dot com + - {lfleischer} at archlinux dot org + - {Nathan} at freitas dot net + - {mike} at tig dot as + - {tails-rm} at boum dot org + - {simon} at sdeziel.info + - {yuri} at freebsd.org + - {mh+tor} at scrit.ch + - {security} at brave.com