1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
|
## gof3
As a CLI or as a library, GoF3 provides a single operation: mirroring. The origin and destination are designated by the URL of a forge and a path to the resource. For instance, `mirror --from-type forgejo --from https://code.forgejo.org/forgejo/lxc-helpers --to-type F3 --to /some/directory` will mirror a project in a local directory using the F3 format.
## Building
* Install go >= v1.21
* make f3-cli
* ./f3-cli mirror -h
## Example
### To F3
Login to https://code.forgejo.org and obtain an application token with
read permissions at https://code.forgejo.org/user/settings/applications.
```sh
f3-cli mirror \
--from-type forgejo --from-forgejo-url https://code.forgejo.org \
--from-forgejo-token $codetoken \
--from-path /forge/organizations/actions/projects/cascading-pr \
--to-type filesystem --to-filesystem-directory /tmp/cascading-pr
```
### From F3
Run a local Forgejo instance with `serials=1 tests/setup-forgejo.sh` and obtain
an application token with:
```sh
docker exec --user 1000 forgejo1 forgejo admin user generate-access-token -u root --raw --scopes 'all,sudo'
```
Mirror issues
```sh
f3-cli mirror \
--from-type filesystem --from-filesystem-directory /tmp/cascading-pr \
--from-path /forge/organizations/actions/projects/cascading-pr/issues \
--to-type forgejo --to-forgejo-url http://0.0.0.0:3001 \
--to-forgejo-token $localtoken
```
Visit them at http://0.0.0.0:3001/actions/cascading-pr/issues
## Testing
### Requirements
The tests require a live GitLab instance as well as a live Forgejo instance and will use up to 16GB of RAM.
* Install docker
* `./test/run.sh`
## License
This project is [MIT licensed](LICENSE).
## Architecture
[F3](https://f3.forgefriends.org/) is a hierarchy designed to be stored in a file system. It is represented in memory with the [tree/generic](tree/generic) abstract data structure that can be saved and loaded from disk by the [forges/filesystem](forges/filesystem) driver. Each forge (e.g. [forges/forgejo](forges/forgejo)) is supported by a driver that is responsible for the interactions of each resource (e.g `issues`, `asset`, etc.).
### Tree
[tree/f3](tree/f3) implements a [F3](https://f3.forgefriends.org/) hierarchy based on the [tree/generic](tree/generic) data structure. The [tree](tree/generic/tree.go) has a [logger](logger) for messages, [options](options) defining which forge it relates to and how and a pointer to the root [node](tree/generic/node.go) of the hierarchy (i.e. the `forge` F3 resource).
The node ([tree/generic/node.go](tree/generic/node.go)) has:
* a unique id (e.g. the numerical id of an `issue`)
* a parent
* chidren (e.g. `issues` children are `issues`, `issue` children are `comments` and `reactions`)
* a kind that maps to a F3 resource (e.g. `issue`, etc.)
* a driver for its concrete implementation for a given forge
It relies on a forge driver for the concrete implemenation of a F3 resource (issue, reaction, repository, etc.). For instance the `issues` driver for Forgejo is responsible for listing the existing issues and the `issue` driver is responsible for creating, updating or deleting a Forgejo issue.
### F3 archive
The [F3 JSON schemas](https://code.forgejo.org/f3/f3-schemas/-/tree/main) are copied in [f3/schemas](f3/schemas). Their internal representation and validation is found in a source file named after the resource (e.g. an `issue` represented by [f3/schemas/issue.json](f3/schemas/issue.json) is implemented by [f3/issue.go](f3/issue.go)).
When a F3 resource includes data external to the JSON file (i.e. a Git repository or an asset file), the internal representation has a function to copy the data to the destination given in argument. For instance:
* [f3/repository.go](f3/repository.go) `FetchFunc(destination)` will `git fetch --mirror` the repository to the `destination` directory.
* [f3/attachment.go](f3/attachment.go) `DownloadFunc()` returns a `io.ReadCloser` that will be used by the caller to copy the asset to its destination.
### Options
The Forge options at [options/interface.go](options/interface.go) define the parameters given when a forge is created:
Each forge driver is responsible for registering the options (e.g. [Forgejo options](forges/forgejo/options/options.go)) and for registering a factory that will create these options (e.g. [Forgejo options registration](forgejo/main.go)). In addition to the options that are shared by all forges such as the logger, it may define additional options.
### Driver interface
For each [F3](https://f3.forgefriends.org/) resource, the driver is responsible for:
* copying the [f3](f3) argument to `FromFormat` to the forge
* `ToFormat` reads from the forge and convert the data into an [f3/resources.go](f3/resources.go)
A driver must have a unique name (e.g. `forgejo`) and [register](forges/forgejo/main.go):
* an [options factory](options/factory.go)
* a [forge factory](tree/f3/forge_factory.go)
#### Tree driver
The [tree driver](tree/generic/driver_tree.go) functions (e.g. [forges/forgejo/tree.go](forges/forgejo/tree.go)) specialize [NullTreeDriver](tree/generic/driver_tree.go).
* **Factory(ctx context.Context, kind generic.Kind) generic.NodeDriverInterface** creates a new node driver for a given [`Kind`](tree/f3/kind.go).
* **GetPageSize() int** returns the default page size.
#### Node driver
The [node driver](tree/generic/driver_node.go) functions for [each `Kind`](tree/f3/kind.go) (e.g. `issues`, `issue`, etc.) specialize [NullNodeDriver](tree/generic/driver_node.go). The examples are given for the Forgejo [`issue`](forges/forgejo/issue.go) and [`issues`](forges/forgejo/issues.go) drivers, matching the REST API endpoint to the driver function.
* **ListPage(context.Context, page int) ChildrenSlice** returns children of the node paginated [GET /repos/{owner}/{repo}/issues](https://code.forgejo.org/api/swagger/#/issue/issueListIssues)
* **Get(context.Context)** get the content of the resource (e.g. [GET /repos/{owner}/{repo}/issues/{index}](https://code.forgejo.org/api/swagger/#/issue/issueGetIssue))
* **Put(context.Context) NodeID** create a new resource and return the identifier (e.g. [POST /repos/{owner}/{repo}/issues](https://code.forgejo.org/api/swagger/#/issue/issueCreateIssue))
* **Patch(context.Context)** modify an existing resource (e.g. [PATCH /repos/{owner}/{repo}/issues/{index}](https://code.forgejo.org/api/swagger/#/issue/issueEditIssue))
* **Delete(context.Context)** delete an existing resource (e.g. [DELETE /repos/{owner}/{repo}/issues/{index}](https://code.forgejo.org/api/swagger/#/issue/issueDelete))
* **NewFormat() f3.Interface** create a new `issue` F3 object
* **FromFormat(f3.Interface)** set the internal representation from the given F3 resource
* **ToFormat() f3.Interface** convert the internal representation into the corresponding F3 resource. For instance the internal representation of an `issue` for the Forgejo driver is the `Issue` struct of the Forgejo SDK.
#### Options
The [options](options) created by the factory are expected to provide the [options interfaces](options/interface.go):
* Required
* LoggerInterface
* URLInterface
* Optional
* CLIInterface if additional CLI arguments specific to the forge are supported
For instance [forges/forgejo/options/options.go](forges/forgejo/options/options.go) is created by [forges/forgejo/options.go](forges/forgejo/options.go).
### Driver implementation
A driver for a forge must be self contained in a directory (e.g. [forges/forgejo](forges/forgejo)). Functions shared by multiple forges are grouped in the [forges/helpers](forges/helpers) directory and split into one directory per `Kind` (e.g. [forges/helpers/pullrequest](forges/helpers/pullrequest)).
* [options.go](forges/forgejo/options.go) defines the name of the forge in the Name variable (e.g. Name = "forgejo")
* [options/options.go](forges/forgejo/options/options.go) defines the options specific to the forge and the corresponding CLI flags
* [main.go](forges/forgejo/main.go) calls f3_tree.RegisterForgeFactory to create the forge given its name
* [tree.go](forges/forgejo/tree.go) has the `Factory()` function that maps a node kind (`issue`, `reaction`, etc.) into an object that is capable of interacting with it (CRUD).
* one file per `Kind` (e.g. [forges/forgejo/issues.go](forges/forgejo/issues.go)).
### Idempotency
Mirroring is idempotent: it will produce the same result if repeated multiple times. The drivers functions are not required to be idempotent.
* The `Put` function will only be called if the resource does not already exist.
* The `Patch` and `Delete` functions will only be called if the resource exists.
### Identifiers mapping
When a forge (e.g. Forgejo) is mirrored on the filesystem, the identifiers are preserved verbatim (e.g. the `issue` identifier). When the filesystem is mirrored to a forge, the identifiers cannot always be preserved. For instance if an `issue` with the identifier 1234 is downloaded from Forgejo and created on another Forgejo instance, it will be allocated an identifier by the Forgejo instance. It cannot request to be given a specific identifier.
### References
A F3 resource may reference another F3 resource by a path. For instance the user that authored an issue is represented by `/forge/users/1234` where `1234` is the unique identifier of the user. The reference is relative to the forge. The mirroring of a forge to another is responsible for converting the references using the identifier mapping stored in the origin forge. For instance if `/forge/users/1234` stored in the filesystem is created in Forgejo as `/forge/users/58`, the `issue` stored in the filesystem with its authored as `/forge/users/1234` will be created in Forgejo to be authored by `/forge/users/58` instead.
### Logger
The [tree/generic](tree/generic) has a pointer to a logger implementing [logger.Interface](logger/interface.go) which is made available to the nodes and the drivers.
### Context
All functions except for setters and getters have a `context.Context` argument which is checked (using [util/terminate.go](util/terminate.go)) to not be `Done` before performing a long lasting operation (e.g. a REST API call or a call to the Git CLI). It is not used otherwise.
### Error model
When an error that cannot be recovered from happens, `panic` is called, otherwise an `Error` is logged.
### CLI
The CLI is in [cmd](cmd) and relies on [options](options) to figure out which options are to be implemented for each supported forge.
## Hacking
### Local tests
The forge instance is deleted before each run and left running for forensic analysis when the run completes.
```sh
./tests/run.sh test_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh test_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh test_gitea # http://0.0.0.0:3001 user root, password admin1234
```
Restart a new forge with:
```sh
./tests/run.sh run_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh run_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh run_gitea # http://0.0.0.0:3001 user root, password admin1234
```
The compliance test resources are deleted, except if the environment variable `GOF3_TEST_COMPLIANCE_CLEANUP=false`.
```sh
GOF3_TEST_COMPLIANCE_CLEANUP=false GOF3_FORGEJO_HOST_PORT=0.0.0.0:3001 go test -run=TestF3Forge/forgejo -v code.forgejo.org/f3/gof3/...
```
### Code coverage
```sh
export SCRATCHDIR=/tmp/gof3
./tests/run.sh # collect coverage for every test
./tests/run.sh run_forgejo # update coverage for forgejo
./tests/run.sh test_merge_coverage # merge coverage from every test
go tool cover -func /tmp/gof3/merged.out # show coverage per function
uncover /tmp/gof3/merged.out GeneratorSetReviewComment # show which lines of the GeneratorSetReviewComment function are not covered
```
### F3 schemas
The JSON schemas come from [the f3-schemas repository](https://code.forgejo.org/f3/f3-schemas) and
should be updated as follows:
```
cd f3 ; rm -fr schemas ; git --work-tree schemas clone https://code.forgejo.org/f3/f3-schemas ; rm -fr f3-schemas schemas/.gitignore schemas/.forgejo
```
## Funding
See the page dedicated to funding in the [F3 documentation](https://f3.forgefriends.org/funding.html)
|
