1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
|
Metadata-Version: 2.1
Name: ostree-push
Version: 1.2.0
Summary: Push and receive OSTree commits
Home-page: https://github.com/dbnicholson/ostree-push
Author: Dan Nicholson
Author-email: dbn@endlessos.org
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)
Classifier: Operating System :: POSIX
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: System :: Archiving :: Mirroring
Classifier: Topic :: System :: Archiving :: Packaging
Classifier: Topic :: System :: Software Distribution
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: COPYING
Requires-Dist: PyGObject
Requires-Dist: PyYAML
# ostree-push
## Background
`ostree-push` uses `ssh` to push commits from a local OSTree repo to a
remote OSTree repo. This is to fill a gap where currently you can only
pull commits in core ostree. To publish commits to a remote repository,
you either have to `pull` from the local repo to the remote repo or use
an out of band mechanism like `rsync`.
Both approaches have significant limitations. To pull over the network,
only http is supported. So, in addition to having to login on the remote
machine and run `ostree pull`, the local repository needs to be served
over http. This means your build machine needs to be an http server with
appropriate configuration in addition to simply making commits. This
pushes the builds to be done on the public repository server, which
prevents reasonable separation of duties and makes multiarch
repositories impossible.
Using `rsync` for publishing has some major benefits since only updated
objects are published. However, it has no concept of the OSTree object
store or refs structures. There are a few problems deriving from this
issue. First, objects are published in sort order, but this means that
objects can be published before their children. In the most extreme
case, a commit object could be published before it's complete. The
remote repo would assume this commit object was valid even though some
children might be missing. Second, the refs might get updated before the
commit objects are in place. If a client pulls while `rsync` is
publishing, it may attempt to pull an incomplete or entirely missing
commit. Finally, `rsync` will push the objects directly into the store
rather than using a staging directory like `pull` or `commit` do. If
`rsync` is interrupted, it could leave partial objects in the store.
`ostree-push` tries to offer functionality like `git` where commits can
be pushed over `ssh` to avoid these issues.
## Operation
When `ostree-push` is started, it first starts a local HTTP server
providing the contents of the local ostree repo. It then connects to the
remote host with `ssh` and tunnels the HTTP server port through the SSH
connection. Finally, it runs `ostree-receive` on the remote host with
the URL of the tunneled HTTP server. `ostree-receive` then creates a
temporary remote using this URL and pulls the desired refs from it.
In essence, `ostree-push` and `ostree-receive` coordinate to pull from
the local repo to a remote repo while avoiding the limitations described
above. Namely, no HTTP server needs to be running and no port needs to
be exposed on the local host. Both resources are created temporarily and
only exposed to the remote host through the secure SSH connection.
## Installation
Use `pip` to install the `otpush` package and the `ostree-push` and
`ostree-receive` scripts. From a git checkout, run:
```
pip install .
```
If `ostree-receive` is not in a default `PATH` location, it may not be
located when run in the environment spawned by the SSH server. As a
workaround, make a symbolic link in a standard location:
```
sudo ln -s /path/to/ostree-receive /usr/bin/ostree-receive
```
In order to restrict SSH usage to only running `ostree-receive`, the
`ostree-receive-shell` script can be used as a login shell. This way
someone with SSH access to the remote machine cannot run arbitrary
commands as the user owning the repositories. To use it, set the login
shell of the repo owner to `ostree-receive-shell`:
```
sudo chsh -s /path/to/ostree-receive-shell <user>
```
`ostree-receive-shell` will also append the directory it's installed in
to `PATH` to allow `ostree-receive` to be found in non-standard
locations. In that scenario, the symbolic link to `ostree-receive`
described above is not needed.
Both `ostree-push` and `ostree-receive` require the OSTree GObject
Introspection bindings. Typically these would be installed from the host
distro. On Debian systems the package is `gir1.2-ostree-1.0` while on
RedHat systems they are in the `ostree-libs` package.
`ostree-push` relies on the connection sharing and port forwarding
features of OpenSSH and is unlikely to work with another SSH client.
Similarly, `ostree-receive` has only be tested with the OpenSSH server,
but it might work correctly with other SSH servers.
## Configuration
`ostree-receive` can be configured from YAML formatted files. It will
load `~/.config/ostree/ostree-receive.conf` and
`/etc/ostree/ostree-receive.conf` or a file specified in the
`OSTREE_RECEIVE_CONF` environment variable. See the example
[`ostree-receive.conf`](ostree-receive.conf) file for available options.
## Testing
A test suite is provided using [pytest][pytest]. Most of the time simply
running `pytest` from a git checkout will run it correctly. [tox][tox]
can also be used to automate running the test suite in a prepared Python
environment.
In addition to the `ostree-push` dependencies, many of the tests depend
on using OpenSSH `sshd` locally. On both Debian and RedHat systems this
is available in the `openssh-server` package.
[pytest]: https://docs.pytest.org/en/stable/
[tox]: https://tox.readthedocs.io/en/stable/
|
