1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
|
======================
Ceph Release Process
======================
Prerequisites
=============
Signing Machine
---------------
The signing machine is a virtual machine in the `Sepia lab
<https://wiki.sepia.ceph.com/doku.php?id=start>`_. SSH access to the signing
machine is limited to the usual Infrastructure Admins along with a few other
component leads (e.g., nfs-ganesha, ceph-iscsi).
The ``ubuntu`` user on the machine has some `build scripts <https://github.com/ceph/ceph-build/tree/main/scripts>`_ that help with pulling, pushing, and signing packages.
The GPG signing key permanently lives on a `Nitrokey Pro <https://shop.nitrokey.com/shop/product/nkpr2-nitrokey-pro-2-3>`_ and is passed through to the VM via RHV. This helps to ensure that the key cannot be exported or leave the datacenter in any way.
New Major Releases
------------------
For each new major (alphabetical) release, you must create one ``ceph-release`` RPM for each RPM repo (e.g., one for el8 and one for el9). `chacra <https://github.com/ceph/chacra>`_ is a python service we use to store DEB and RPM repos. The chacra repos are configured to include this ceph-release RPM, but it must be built separately. You must make sure that chacra is properly configured to include this RPM for each particular release.
1. Update chacra so it is aware of the new Ceph release. See `this PR <https://github.com/ceph/chacra/pull/219>`_ for an example.
2. Redeploy chacra (e.g., ``ansible-playbook chacra.ceph.com.yml``)
3. Run https://jenkins.ceph.com/view/all/job/ceph-release-rpm/
Summarized build process
========================
1. QE finishes testing and finds a stopping point. That commit is pushed to the ``$release-release`` branch in ceph.git (e.g., ``quincy-release``). This allows work to continue in the working ``$release`` branch without having to freeze it during the release process.
2. The Ceph Council approves and notifies the "Build Lead".
3. The "Build Lead" starts the `Jenkins multijob <https://jenkins.ceph.com/view/all/job/ceph>`_, which triggers all builds.
4. Packages are pushed to chacra.ceph.com.
5. Packages are pulled from chacra.ceph.com to the Signer VM.
6. Packages are signed.
7. Packages are pushed to download.ceph.com.
8. Release containers are built and pushed to quay.io.
Hotfix Release Process Deviation
--------------------------------
A hotfix release has a couple differences.
1. Check out the most recent tag. For example, if we're releasing a hotfix on top of 17.2.3, ``git checkout -f -B quincy-release origin/v17.2.3``
2. ``git cherry-pick -x`` the necessary hotfix commits
3. ``git push -f origin quincy-release``
4. Notify the "Build Lead" to start the build.
5. The "Build Lead" should set ``RELEASE_TYPE=HOTFIX`` instead of ``STABLE``.
Security Release Process Deviation
----------------------------------
A security/CVE release is similar to a hotfix release with two differences:
1. The fix should be pushed to the `ceph-private <https://github.com/ceph/ceph-private>`_ repo instead of ceph.git (requires GitHub Admin Role).
2. The tags (e.g., v17.2.4) must be manually pushed to ceph.git by the "Build Lead."
1. Check out the most recent tag. For example, if we're releasing a security fix on top of 17.2.3, ``git checkout -f -B quincy-release origin/v17.2.3``
2. ``git cherry-pick -x`` the necessary security fix commits
3. ``git remote add security git@github.com:ceph/ceph-private.git``
4. ``git push -f security quincy-release``
5. Notify the "Build Lead" to start the build.
6. The "Build Lead" should set ``RELEASE_TYPE=SECURITY`` instead of ``STABLE``.
7. Finally, the `ceph-tag <https://github.com/ceph/ceph-build/blob/main/ansible/roles/ceph-release/tasks/push.yml>`_ steps need to be manually run by the "Build Lead" as close to the Announcement time as possible::
# Example using quincy pretending 17.2.4 is the security release version
# Add the ceph-releases repo (also requires GitHub Admin Role). The `ceph-setup <https://jenkins.ceph.com/job/ceph-setup>`_ job will have already created and pushed the tag to ceph-releases.git.
git remote add releases git@github.com:ceph/ceph-releases.git
git fetch --all
# Check out the version commit
git checkout -f -B quincy-release releases/quincy-release
git push -f origin quincy-release
git push origin v17.2.4
# Now create a Pull Request of quincy-release targeting quincy to merge the version commit and security fixes back into the quincy branch
1. Preparing the release branch
===============================
Once QE has determined a stopping point in the working (e.g., ``quincy``) branch, that commit should be pushed to the corresponding ``quincy-release`` branch.
Notify the "Build Lead" that the release branch is ready.
2. Starting the build
=====================
We'll use a stable/regular 15.2.17 release of Octopus as an example throughout this document.
1. Browse to https://jenkins.ceph.com/view/all/job/ceph/build?delay=0sec
2. Log in with GitHub OAuth
3. Set the parameters as necessary::
BRANCH=octopus
TAG=checked
VERSION=15.2.17
RELEASE_TYPE=STABLE
ARCHS=x86_64 arm64
4. Use https://docs.ceph.com/en/latest/start/os-recommendations/?highlight=debian#platforms to determine the ``DISTROS`` parameter. For example,
+-------------------+--------------------------------------------------+
| Release | Distro Codemap |
+===================+==================================================+
| pacific (16.X.X) | ``focal bionic buster bullseye`` |
+-------------------+--------------------------------------------------+
| quincy (17.X.X) | ``jammy focal centos9 bullseye`` |
+-------------------+--------------------------------------------------+
| reef (18.X.X) | ``jammy focal centos9 windows bookworm`` |
+-------------------+--------------------------------------------------+
5. Click ``Build``.
3. Release Notes
================
Packages take hours to build. Use those hours to create the Release Notes and Announcements:
1. ceph.git Release Notes (e.g., `v15.2.17's ceph.git (docs.ceph.com) PR <https://github.com/ceph/ceph/pull/47198>`_)
2. ceph.io Release Notes (e.g., `v15.2.17's ceph.io.git (www.ceph.io) PR <https://github.com/ceph/ceph.io/pull/427>`_)
3. E-mail announcement
See `the Ceph Tracker wiki page that explains how to write the release notes <https://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO_write_the_release_notes>`_.
.. _Signing and Publishing the Build:
4. Signing and Publishing the Build
===================================
#. Obtain the sha1 of the version commit from the `build job <https://jenkins.ceph.com/view/all/job/ceph>`_ or the ``sha1`` file created by the `ceph-setup <https://jenkins.ceph.com/job/ceph-setup/>`_ job.
#. Download the packages from chacra.ceph.com to the signing virtual machine. These packages get downloaded to ``/opt/repos`` where the `Sepia Lab Long Running (Ceph) Cluster <https://wiki.sepia.ceph.com/doku.php?id=services:longrunningcluster>`_ is mounted. Note: this step will also run a command to transfer the source tarballs from chacra.ceph.com to download.ceph.com directly, by ssh'ing to download.ceph.com and running /home/signer/bin/get-tarballs.sh.
.. prompt:: bash $
ssh ubuntu@signer.front.sepia.ceph.com
sync-pull ceph [pacific|quincy|etc] <sha1>
Example::
$ sync-pull ceph octopus 8a82819d84cf884bd39c17e3236e0632ac146dc4
sync for: ceph octopus
********************************************
Found the most packages (332) in ubuntu/bionic.
No JSON object could be decoded
No JSON object could be decoded
ubuntu@chacra.ceph.com:/opt/repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubuntu/bionic/flavors/default/* /opt/repos/ceph/octopus-15.2.17/debian/jessie/
--------------------------------------------
receiving incremental file list
db/
db/checksums.db
180.22K 100% 2.23MB/s 0:00:00 (xfr#1, to-chk=463/467)
db/contents.cache.db
507.90K 100% 1.95MB/s 0:00:00 (xfr#2, to-chk=462/467)
db/packages.db
etc...
#. Sign the DEBs:
.. prompt:: bash
merfi gpg /opt/repos/ceph/octopus-15.2.17/debian
Example::
$ merfi gpg /opt/repos/ceph/octopus-15.2.17/debian
--> Starting path collection, looking for files to sign
--> 18 matching paths found
--> will sign with the following commands:
--> gpg --batch --yes --armor --detach-sig --output Release.gpg Release
--> gpg --batch --yes --clearsign --output InRelease Release
--> signing: /opt/repos/ceph/octopus-15.2.17/debian/jessie/dists/bionic/Release
--> Running command: gpg --batch --yes --armor --detach-sig --output Release.gpg Release
--> Running command: gpg --batch --yes --clearsign --output InRelease Release
--> signing: /opt/repos/ceph/octopus-15.2.17/debian/jessie/dists/focal/Release
--> Running command: gpg --batch --yes --armor --detach-sig --output Release.gpg Release
--> Running command: gpg --batch --yes --clearsign --output InRelease Release
etc...
#. Sign the RPMs:
.. prompt:: bash
sign-rpms ceph octopus
Example::
$ sign-rpms ceph octopus
Checking packages in: /opt/repos/ceph/octopus-15.2.17/centos/7
signing: /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-release-1-1.el7.src.rpm
/opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-release-1-1.el7.src.rpm:
signing: /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-15.2.17-0.el7.src.rpm
/opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-15.2.17-0.el7.src.rpm:
signing: /opt/repos/ceph/octopus-15.2.17/centos/7/noarch/ceph-mgr-modules-core-15.2.17-0.el7.noarch.rpm
etc...
#. Publish the packages to download.ceph.com:
.. prompt:: bash $
sync-push ceph octopus
This leaves the packages, and the tarball, in a password-protected
prerelease area at https://download.ceph.com/prerelease/ceph. Verify them
from there. When done and ready for release, log into download.ceph.com and
mv the directories and the tarballs from the prerelease home
(/data/download.ceph.com/www/prerelease/ceph) to the release directory
(/data/download.ceph.com/www).
5. Build Containers
===================
Unlike CI builds, which have access to packages in the correct form for
the container, release builds do not, because the build does not
sign the packages. Thus, release builds do not build the containers.
This must be done after :ref:`Signing and Publishing the Build`.
A Jenkins job named ceph-release-containers exists so that we can
test the images before release. The job exists both for convenience and
because it requires access to both x86_64 and arm64 builders. Start the
job manually on the Jenkins server. This job:
* builds the architecture-specific container imagess and pushes them to
quay.ceph.io/ceph/prerelease-amd64 and
quay.ceph.io/ceph/prerelease-arm64
* fuses the architecture-specific images together into a 'manifest-list'
or 'fat' container image and pushes it to quay.ceph.io/ceph/prerelease
Finally, when all appropriate testing and verification is done on the
container images, run ``make-manifest-list.py --promote`` from the ceph
source tree (at ``container/make-manifest-list.py``) to promote them to
their final release location on ``quay.io/ceph/ceph`` (you must ensure
you're logged into ``quay.io/ceph`` with appropriate permissions):
.. prompt:: bash
cd <ceph-checkout>/src/container
./make-manifest-list.py --promote
The --promote step should only be performed as the final step in releasing
containers, when the container images have been tested to be good.
6. Announce the Release
=======================
Version Commit PR
-----------------
The `ceph-tag Jenkins job <https://jenkins.ceph.com/job/ceph-tag>`_ creates a Pull Request in ceph.git that targets the release branch.
If this was a regular release (not a hotfix release or a security release), the only commit in that Pull Request should be the version commit. For example, see `v15.2.17's version commit PR <https://github.com/ceph/ceph/pull/47520>`_.
Request a review and then merge the Pull Request.
Announcing
----------
Publish the Release Notes on ceph.io before announcing the release by email, because the e-mail announcement references the ceph.io blog post.
|
