1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
|
= JSON2FILE-GO (8)
:doctype: manpage
== NAME
json2file-go - simple web server that provides entry points to store JSON files
== SYNOPSIS
json2file-go
== DESCRIPTION
The json2file-go(1) is a mini-application that runs as a Web server and is used
to store the information sent by remote WebHooks in JSON format using the HTTP
POST method.
Initially it has been used to process GitLab and Gitea WebHooks, but it can be
used with other systems.
To verify that a caller has permission to store a file the program compares a
JSON field or an HTTP HEADER to the _secret_ it has configured for the given
PATH. If the _secret_ matches the program verifies that the data received is a
valid JSON file and only stores it if it is.
The application is configured using environment variables and is designed to
run inside a `docker` container or supervised with `systemd` if we are on a
regular Linux server.
The application reads two mandatory environment variables:
- `J2F_BASEDIR`: Working directory of the application, must exist before
launching it.
+
The default value for this variable when using the `systemd` helper (see later)
is `/var/spool/json2file-go`. If we use the application inside a container the
idea is to set the value to `/data` (is a volume in the `Dockerfile`) and use a
volume or mount a HOST directory there.
- `J2F_DIRLIST`: List of directories in which we allow one to save files and
an optional token that must be passed in a header or within the JSON to
authorize the save operation for the file.
+
The variable is a string of the form: `DIR1:TOKEN1;...;DIR#:TOKEN#`
+
There is no default for this variable, it must be set.
Other optional variables include:
- `J2F_SIGNATURE_HEADERS`: Comma-separated list of names for the HTTP header
containing the signature hash (by default the list contains the name
`X-Hub-Signature`)
- `J2F_TOKEN_HEADERS`: Comma-separated list of names for the HTTP header
containing the token (by default the list contains the names `X-Auth-Token`
and `X-Gitlab-Token`)
- `J2F_TOKEN_FIELDS`: Comma-separated list of names for the received JSON
field that contains the token (by default we use secret)
- `J2F_URL_PREFIX`: Service base path on URLs, the directory names are computed
removing this prefix
- `J2F_HOST`: IP address to bind to (default 0.0.0.0)
- `J2F_PORT`: TCP Port (80 if there are no certificates, 443 if there are any)
- `J2F_BIND`: Listen Address and Port (default `$J2F_HOST:$J2F_PORT`, if used
replaces the other two variables)
- `J2F_CERTFILE` and `J2F_KEYFILE`: public certificate and key file for TLS (if
both are set it is assumed that we are using TLS and the server uses that
certificate and private key)
- `J2F_SOCKET`: UNIX Socket to listen on (has no default value, if passed the
server listens for HTTP requests on a UNIX socket ignoring the address, port
and TLS settings); this is the preferred mode when the service is accessed
using NGINX as a reverse proxy
By default the system saves the JSON data received in the sub folder of
`J2F_BASEDIR` that matches the requested directory name using files with
names of the form `TIMESTAMP.json`, where the format of `TIMESTAMP` is
`YYYYmmdd-HHMMSS.MS.json`.
While this program does nothing else, the files are usually processed by
scripts launched by cron(8) or incrond(8).
To be able to work with systemd.socket(5) when the application launches it
checks if the variable `LISTEN_PID` is set and in that case the `J2F_HOST`,
`J2F_PORT`, `J2F_BIND` and `J2F_SOCKET` are ignored and the program listens on
the first file descriptor available after _stdout_ (the one with value `3`)
ignoring other descriptors if any (the daemon only _listens_ on a file
descriptor). Note that in this case the `J2F_CERTFILE` and `J2F_KEYFILE` are
NOT ignored (systemd can pass us a file descriptor where we should use HTTPS).
If the program is launched without `systemd` and the variables `J2F_HOST`, `J2F_PORT`,
`J2F_BIND` or `J2F_SOCKET` have no value the program tries to listen on
0.0.0.0:80 (or 0.0.0.0:443 if we provide a certificate and a key file).
== CONFIGURATION
The binary does not read any configuration files by itself, but the packaged
version includes a helper script (`/usr/lib/json2file-go/json2file-go`) to
adjust environment variables when launching the program with `systemd` or other
`init` systems (see the `/usr/share/doc/json2file-go/examples` directory for
samples).
The launcher script reads files from the directory `/etc/json2file-go` and
adjusts the variables as follows:
- `dirlist` (file with lines of the form 'dir:token' that are passed on the
variable `J2F_DIRLIST` separated by ';'). This file is required when using
`systemd`.
- `basedir` (value of the `J2F_BASEDIR` variable), has a default value when using
`systemd`
- `debug` (value of the `J2F_DEBUG` variable)
- `signature_headers` (lines with of HTTP header names that can contain
signature hashes, the values are stored separated by commas on the
environment variable `J2F_SIGNATURE_HEADERS`)
- `token_headers` (lines with of HTTP header names that can contain
authorization tokens, the values are stored separated by commas on the
environment variable `J2F_TOKEN_HEADERS`)
- `token_fields` (lines with names of JSON fields that can contain
authorization tokens, the values are stored separated by commas on the
environment variable `J2F_TOKEN_FIELDS`)
- `url_prefix` (value of the `J2F_URL_PREFIX`)
- `bind` (value of the `J2F_BIND` variable)
- `host` (value of the `J2F_HOST` variable)
- `port` (value of the `J2F_PORT` variable)
- `certfile` (value of the `J2F_CERFILE` variable)
- `keyfile` (value of the `J2F_KEYFILE` variable)
- `socket` (value of the `J2F_SOCKET` variable)
== BUGS
None so far... ;)
== AUTHOR
`json2file` was originally written by Sergio Talens-Oliag in *Python* using the
*Flask* micro framework, later it was rewritten using *FastAPI* (another
*Python* _framework_) and to make it lighter for container usage this third
version in *Go* was born.
== SEE ALSO
docker(1), systemd(1), incrond(8)
== COPYING
Copyright (c) 2019-2020 Sergio Talens-Oliag. Free use of this software is
granted under the terms of the Apache License Version 2.0.
|
