File: pkg-zsh-workflow.md

package info (click to toggle)
zsh 5.3.1-4
  • links: PTS, VCS
  • area: main
  • in suites: stretch
  • size: 16,324 kB
  • ctags: 8,554
  • sloc: ansic: 104,868; sh: 6,584; perl: 813; makefile: 774; awk: 388; sed: 16
file content (512 lines) | stat: -rw-r--r-- 17,351 bytes parent folder | download | duplicates (3)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
Branches
========

* **upstream**: The upstream sources from `git://zsh.git.sf.net/gitroot/zsh/zsh`.
* **debian**: The debian changes for the debian `zsh` package. Only
  difference to the upstream branch is the `debian` directory.
* master: This is the _old_ repository's main branch. Only kept for
  _historical_ reasons.
* Other branches: These branches are rather optional and not required
  for basic maintenance for of the `zsh` package. Most likely feature
  branches.


Workflow
========

This diagram outlines the workflow with git branches and
tags. Basically, the `debian` branch walks alongside the `upstream`
branch and upstream changes get merged into `debian` at defined points.
For the zsh package, this _must_ be upstream commit tags -- otherwise
Jenkins
[fails to build the source package](http://jenkins.grml.org/view/Debian/job/zsh-source/219/console).

            * debian/4.3.11-4
            |
            |   * debian/4.3.11-5
            |   |
            |   |       * debian/4.3.12-1
            |   |       |
            |   |       |   * debian/4.3.12-2
            |   |       |   |
            |   |       |   |           * debian/4.3.13-1
            |   |       |   |           |
            |   |       |   |           | * debian/4.3.13-2
            |   |       |   |           | |
            |   |       |   |           | |   * debian/4.3.13-3
            |   |       |   |           | |   |
            |   |       |   |           | |   |
          a-b-c-d---e-f-g-h-i-------j-k-l-m-n-o-p   debian
         /         /               /
      A-B-C-D-E-F-G-H-I-J-K-L-M-N-O-P-Q             upstream
        |         |               |
        |         |               |
        |         |               * zsh-4.3.13
        |         * zsh-4.3.12
        |
        * zsh-4.3.11

Working on the package
----------------------

### Patching upstream code

Every change to the source (outside the `debian` directory) must be
done by adding quilt patches.

#### Patch naming conventions

* File names of patches which are not meant for upstream should be
  prefixed with `debian-`.
* File names of patches which are cherry-picked from upstream should be prefixed
  with `cherry-pick-$shortcommitid` where `$shortcommitid` are the
  first 8 characters of the upstream git commit id.

In case a vast number of patches are required, quilt patches should be
prefixed by ascending numbers according to the order in which "quilt
series" would list them.

#### Patch format

Every patch should have
[DEP3 conforming patch headers](http://dep.debian.net/deps/dep3/)
above the actual unified diff content, outlining the status of the
patch.

Except for the obvious case of a cherry-picked patch from upstream
(which should contain an `Origin: commit $commitid` header), it is
important that the fields `Forwarded` and `Applied-Upstream` are kept
uptodate to know whether a change is already applied upstream.  This
helps to drop the right patches before merging a new upstream release
into the `debian` branch.


#### Example of adding a fix via a quilt patch

Let's say, there is an issue _#12345_ which can be fixed by patching
`Functions/Misc/colors`. Here is how you could add that patch
(assuming clean git working directory):

First, push all existing patches, so the new one is added on top.

    % quilt push -a

Alternatively push the patches one by one (by calling `quilt push`
multiple times) until you are at the patch queue position where you
want to insert the patch.

##### Adding a patch from a file or by editing

Add the new patch (say, the topmost patch is 0002-foo.diff).

    % quilt new fix-colors.diff

Tell quilt which files are going to be changed.

    % quilt add Functions/Misc/colors

Import the fix either manually using your favourite editor…

    % $EDITOR Functions/Misc/colors

… or by patching:

    % patch -p$n < ../patch_from_somewhere_else.diff

##### Cherry-picking patches from upstream

When there is an existing patch (e.g. from upstream's git repository),
the above can be largely automated if the patch applies to the current
state of the debian branch.

    % patchname="cherry-pick-$shortcommitid-description-with-dashes"
    % git show $commitid | filterdiff -x a/ChangeLog > debian/patches/$patchname
    % sed -e '1 s/^commit/Origin: commit/' -i debian/patches/$patchname
    % echo $patchname >> debian/patches/series
    % $EDITOR debian/patches/$patchname
    % git add debian/patches/series debian/patches/$patchname
    % git commit -v

Patches from upstream will likely include changes to the ChangeLog
file. Those changes will probably not apply cleanly, so just open the
created patch file and delete all hunks that do changes in
ChangeLog. The `filterdiff` command above should catch that.

    % $EDITOR debian/patches/$patchname

Check if the patch applies

    % quilt push

All the above is mostly automated in `debian/bin/commit2patch`.

##### Finish import of the patch

Refresh the patch to get rid of any fuzz or offset:

    % quilt refresh

Pop all patches again to clean up the upstream source.

    % quilt pop -a

Edit the patch headers according to
[DEP3](http://dep.debian.net/deps/dep3/).

    % $EDITOR debian/patches/$patchname

Commit the new patch and the changed `series` file to git.

    % git add debian/patches/fix-colors.diff
    % git add debian/patches/series
    % git commit -m 'Fixing foo in colors function (Closes: #12345)'

That's all.

### Releases

When a change justifies the release of a new package version, the
debian/changelog file should be updated and the resulting commit
should be tagged `debian/${zsh_version}-${n+1}`.


### Updating debian/changelog

This file should *not* be updated manually. The changes should be
inserted by running the `gbp dch` command from the package
`git-buildpackage` (see gbp-dch(1)) before a new release is about to
be made.

Changelog entries should be prefixed by a `[$hashsum] ` string, where
`$hashsum` is a string that represents the first eight characters of
commit the changelog entry was generated from.

Also, if multiple authors are involved in a changelog entry-set, each
author should only appear once in the series with all her/his changes
listed below her/him in chronological order.

Given that `debian/gbp.conf` is up-to-date, using the `gbp dch`
command will result in the desired changelog format:

    % gbp dch

If you absolutely *must* make changelog entries by other means, you
should make sure that you prefix any resulting commits with
"[dch-ignore] ", so those commits can be weeded out easily.

There is a helper script `debian/bin/do-dch` which takes care of all
formatting options as well as the "[dch-ignore] " weeding. The
script should be used unless there is a good reason not to.


Transitioning to a new upstream version
---------------------------------------

When upstream releases a new version, we should follow these steps:

### Merging new upstream tag (`zsh-$version`) into our upstream branch

    % git checkout upstream
    % git pull origin
    % git fetch zsh
    % git merge --ff-only zsh-$version

If that doesn't do a fast-forward merge, a fast-forward merge can be
enforced as follows:

    % git checkout upstream
    % git reset --hard zsh-$version

### Remove all quilt patches which are applied upstream

All patches applied should be removed from `debian/patches` directory,
unless they fix an issue that was *not* addressed upstream and is
therefore missing from upstream's code base.

### Merging the branch upstream into the branch debian

After the `debian/patches` directory was cleaned up in the previous
step, merging `upstream` into `debian` should generally lead to a
working package again.

If old patches were still around, that could lead to conflicts
when those would be applied during the build process.

The message for the merge commit should be set to "New upstream
release" to allow `gbp dch` to pick it up correctly later. **TODO**:
Doesn't really work.

### Insert initial changelog for the new upstream release

`gbp dch` seems to be in trouble with non-linear histories. Therefore
we introduced a small helper script that will help `gbp dch` to a
linear history again.

Basically, you after merging the upstream release tag into the debian
branch, you'll be left with an history that looks something like this:

    *    Updating autotools patches
    M    Merge commit 'zsh-4.3.13' into debian
    |'*  unposted: released 4.3.13
    | *  …
    | *  … lots of other upstream commits …
    | *  …
    * |  Removing upstream patches due to new release
    * |  Last debian/4.3.12-* commit
    * |  …
    * |  … lot's of other debian/4.3.12-* commits
    * |  …
    M´   Merge commit 'zsh-4.3.12' into debian
    |'*  unposted: released 4.3.12
    …    older history

And what you really want added to debian/changelog is the "Updating
autotools patches" and the "Removing upstream patches due to new
release" commits. You need to figure out the sha1 sums of the commits
and then call this:

    % debian/bin/urcl -p=zsh -v=4.3.13-1 b495ba1e f575f568

… where `4.3.13-1` is the version of the upcoming debian package and
`b495ba1e` and `f575f568` are the SHA1 hashsums of the wanted commits.

At the end the script will drop you into an editor pointed at the
changelog file so you can sanity-check the generated output.

At this point it would also make sense to add a line like this:

    * New upstream release

or, if someone explicitly requested a package of this upstream
release, with mentioning of the according bug report number:

    * New upstream release (Closes: #1234567)

If the upstream release fixes additional bugs reported in Debian or
security relevant bugs, the corresponding upstream commits should be
listed indented by two spaces, together with a short description and
the according bug report and/or CVE numbers, e.g. like this:

    * New upstream release
      + [abcdefgh] Fixes foo (Closes: #1234567)
      + [deadcafe] Adds Completion for bar (Closes: #987654)
      + [babeabed] Fixes CVE-2014-9876

When creating a commit with these changelog changes, you may want to
prefix the commit message with `[dch-ignore] ` or add `-m "Gbp-Dch:
Ignore"` to the commit command so it doesn't come up in later `gbp dch`
runs.

### Create the fake orig tar ball (until we can work with upstream's tarball)

This requires the upstream release to be properly tagged.

    % make -f debian/rules get-orig-source

### Fix outstanding bugs

If *any* outstanding bugs are known, they should be fixed before
releasing a new package. Obviously, if any of the known bugs are very
hard to fix and the issue is not serious in nature, releasing the
package with the issue may be more important.

Again, all changes to non `debian/*` files should be done via quilt
patches.


### Verify that the package builds

There are many ways to do this. Important is:

* Use a clean and uptodate Debian Sid chroot (e.g. by using `sbuild`,
  or `pbuilder` and friends) to make a comprehensive test or for
  uploading.

  Axel prefers: `pdebuild --debbuildopts -j6`

* For a quick sanity check, a simple `dpkg-buildpackage -B` (just
  builds the architecture-dependent binary packages or a `debuild -uc
  -us` (builds source and binary packages, runs `lintian` afterwards)
  may suffice.

* Use `gbp buildpackage` to automatically make sure, you are on the
  correct branch and that the working directory is clean.


### Update changelog again for the release

The `do-dch` helper script should be used to do this. It wraps `gbp dch`
with appropriate options and weeds out any commits that are prefixed
with "[dch-ignore] ". All options to the script are turned over to
`gbp dch` and at least `--since=…` should be used.

At this particular point the sha1 of the previous initial changelog
update commit would be a good idea. Also "-R" to tell `gbp dch` to
prepare the changelog for an actual commit. So:

    % debian/bin/do-dch --since=1234deadbeef -R

You'll be dropped into an editor again to double check the generated
changelog.


### Tag debian/${new_zsh_version}-1

After fixes for all serious and trivially fixable issues have been
added and it has been verified that the package builds and `do-dch`
has updated `debian/changelog` and the resulting commit should be
tagged as `debian/${new_zsh_version}-1`.


Generating packages
-------------------

### git-buildpackage aka gbp

Alternatively, `git-buildpackage` (short `gbp`) also provides ways of
building packages from our packaging codebase. And since we are using
the `gbp dch` command (formerly the `git-dch` tool from this utility
suite anyway, the tool should be available already.

`git-buildpackage` allows building the package from within the
package repository:

    % gbp buildpackage

Make sure that the local repository is cleaned up after doing this
before working on the package again, to avoid accidentially committing
anything. See *Cleaning up the local repository* below for details.

`git-buildpackage` is available as Debian package or from
https://honk.sigxcpu.org/piki/projects/git-buildpackage/


Git repository setup
--------------------

Getting the basic pkg-zsh git repository is quite easy. If you want
a read only clone, use this:

    % gbp clone git://anonscm.debian.org/collab-maint/zsh.git pkg-zsh

If you are reading this, though, you probably want write access. To
get a thusly cloned repository, first get an alioth login and upload
an ssh-public key. As soon as the key made it to all involved
machines, use this:

    % gbp clone $user@git.debian.org:/git/collab-maint/zsh.git pkg-zsh

Where `$user` is your Alioth login. (Note, that this may be something
with a `-guest` suffix, in case you're not a Debian Developer.)

### Branches

Like described earlier, pkg-zsh development involves two branches;
`debian` and `upstream`. The former is checked out by default for
freshly cloned repositories.

If you cloned the repository with `gbp clone` as shown above, gbp
already took care of also creating a local `upstream` branch.

If you didn't, you can get a local version of the `upstream` branch by
calling

    % git checkout -b upstream origin/upstream

This is useful to update the remote upstream branch with ongoing
development from the zsh project.

### Remotes

There is one remote repository with direct interest for pkg-zsh, and
that is the zsh project's git repository. Currently, this is only a
mirror of the project's cvs repository. But it is updated every ten
minutes by one of zsh's developers. (Also note, that there has been a
brief discussion about whether git may become the official VCS for git
after a bigger future release.)

In order to have zsh's ongoing development available from within
your pkg-zsh repository, do this:

    % git remote add zsh.git git://zsh.git.sf.net/gitroot/zsh/zsh -t master
    % git fetch zsh.git

### Merging and pushing upstream changes

To get updates back into `origin/upstream`, do this:

Get the latest updates.

    % git fetch zsh.git

Switch to the local `upstream` branch for integration.

    % git checkout upstream

Merge upstream's changes (*).

    % git merge zsh.git/master

Push the code into pkg-zsh's central repository.

    % git push origin

Make sure the central repository also has all tags.

    % git push --tags origin

(*) This step should *always* result in a fast-forward merge. If it
does not, something went terribly wrong. Investigate and fix the
situation *before* pushing to origin.

### Dealing with quilt's .pc directory

When quilt works, it keeps track of what it does in a directory by the
name `.pc`. This directory will show up in the output of `git status`
etc. It should *never* *ever* by committed to the git repository at
any point.

We cannot add the directory to `.gitignore` because we would change
the zsh source directly instead of via `debian/patches`.

To deal with the directory, there is another git-facility, that we can
use for fun and profit.

    % echo .pc/ >> .git/info/exclude

Now git will ignore quilt's directory for this repository.
Unfortunately, this has to be done for each checkout. Luckily, this
only has an impact for people who want to work on *pkg-zsh*. Everyone
else can savely ignore the directory.


General Git Hints
-----------------

### Keeping the local repository clean

Before making changes of any kind, it should be made sure that the
local repository you are working on is in a clean state. To clean up
the local repository do this:

     % git clean -xdf
     % git reset --hard

That will make sure that any non-tracked files are removed and that
any changes in tracked files are reverted. The latter will also make
sure that no parts of a quilt patch-queue are still applied.


TODO
====

* How and when to tag releases → `gbp buildpackage --git-tag` or even
  `gbp buildpackage --git-tag-only` only after the upload has been
  uploaded/accepted.
* How and when to push tags. Debian Perl Group's `dpt push` (from the
  package `pkg-perl-tools`) comes in handy.
* `git commit -m 'Something unimportant' -m 'Gbp-Dch: Ignore'`
* `export QUILT_PATCHES=debian/patches` should be mentioned under
  *Repository setup*, not under *Verify that the package builds*.
* `* New upstream release` changelog entries should have the git
  commit id of the upstream tag.