1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=US-ASCII">
<title>Advanced Server Configuration</title>
<link rel="previous" href="config-servquick.html">
<link rel="ToC" href="index.html">
<link rel="up" href="index.html">
<link rel="next" href="secure.html">
</head>
<body>
<p><a href="config-servquick.html">Previous</a> | <a href="index.html">Contents</a> | <a href="secure.html">Next</a></p>
<ul>
<li><a href="#config-serv">Chapter 4: Advanced Server Configuration</a>
<ul>
<li><a href="#Vocabulary">4.1 Vocabulary</a></li>
<li><a href="#cfgfiles">4.2 Configuration file types</a></li>
<li><a href="#repmap">4.3 Repositories and URL mapping</a>
<ul>
<li><a href="#basic-map">4.3.1 Writing Remap-... configuration</a></li>
<li><a href="#remap-trickz">4.3.2 Special tricks and additional notes</a></li>
</ul></li>
</ul></li>
</ul>
<h1><a name="config-serv"></a>Chapter 4: Advanced Server Configuration</h1>
<h2><a name="Vocabulary"></a>4.1 Vocabulary</h2>
<p>
This chapter introduces some terminology which is needed to understand the functionality of apt-cacher-ng; it's recommended to understand it before continuing with the advanced configuration.
</p>
<ul><li>
"Backend": a text file consisting of a list of mirror URLs, one per line (a more complex RFC822-like format is also supported). Used for URL remapping; see <a href="#repmap">section 4.3</a>.
</li>
<li>
"Volatile files": nothing to do with debian-volatile, volatile here only means that they are volatile, i.e. their contents are expected to be regularly changed on the server. For example, metadata pertaining to package files stored in a remote archive is classified as 'volatile'. They are usually 'index files' known as Packages, Sources, Release, Pdiff and the like.
</li>
<li>
"Package files": files that contain software packages and other "solid" data: DEBs, source files for their creation (.tar.gz, .diff, .dsc), various metadata which is not subject to change after first appearance on the server.
</li>
<li>
"Configuration line": one single line in the configuration file. Some examples in this chapter may contain wrapped lines but should be stored as a single line in the configuration.
</li>
</ul>
<h2><a name="cfgfiles"></a>4.2 Configuration file types</h2>
<p>
By default, the /etc/apt-cacher-ng directory (or the one specified with program options) contains all config files, HTML page templates, the stylesheet and other text-based support files used by apt-cacher-ng. The contents may vary depending on the installation of apt-cacher-ng, refer to the package documentation for Linux Distribution packages.
</p>
<p>
There are a few certain file types distinguished by apt-cacher-ng:
</p>
<ol><li>
Main configuration files:
<p>
*.conf files are assumed to contain configuration directives in the form of "key: value" pairs. The package comes with a commented example configuration file. apt-cacher-ng reads all files matching *.conf in alphabetical order and merges the contents. For options documentation, see commented example file shipped with apt-cacher-ng (conf/ directory in original source).
</p>
<p>
For security reasons, the files can be made readable only to the daemon and administorator accounts, e.g. when they contain passwords or other sensitive data.
</p>
</li>
<li>
URL lists and remote repository list files. The file names are arbitrary, no special suffix is required. They are read and included during processing of configuration files and can contain data in one of the following formats:
<ul><li>
simple text files with one URL per line (the URL should point to the base directory of the repository, e.g. "http://ftp.de.debian.org/debian/"). A URL must start with http:// and should end with a slash
</li>
<li>
an RFC822-like format, with lines like 'Site: <hostname>' and 'Archive-http: /base/directory/of/repository/'. Optional fields are also used in this remapping descriptions to add more possible variants (Alias, Aliases, X-Archive-http:) of the URLs to the lookup list
</li>
</ul>
</li>
<li>
Various support files used for the configuration web interface, named like *.css and *.html.
</li>
<li>
*.default files are used in some rare cases as replacement for list files having the same name without .default suffix.
</li>
<li>
*.hooks files specify custom actions which can be executed upon connection/disconnection (see <a href="#remap-trickz">section 4.3.2</a> for details).
</li>
</ol>
<p>
Except from .conf files, most files listed above can be moved to another "support" directory and the daemon will look for them therein if they are not present in the primary configuration directory. This feature is intended to help keeping pure configuration data and mostly static data in different locations. The directory path is specified at build time and can be overriden with the <em>SupportDir</em> directive (and if used, this should be set as early as possible).
</p>
<h2><a name="repmap"></a>4.3 Repositories and URL mapping</h2>
<p>
With the most simple configuration, apt-cacher-ng will act almost like an ordinary HTTP proxy with improved caching behaviour. When files are requested, they are downloaded from a remote location specified in client's request and are stored in a unique way.
</p>
<p>
However, for some use cases it can be beneficial to specify additional rules to achieve further improvements, e.g. in order to detect and prevent avoidable downloads, to reduce space requirements for the cache directory or simply hide real download locations from the APT clients.
</p>
<p>
These modifications are generally achieved by two strategies, <code>Merging</code> and <code>Redirection</code>, which are configured in a context of a specified cache <code>Repository</code>. The configuration for them is created using one or multiple Remap-... configuration directives (see below).
</p>
<p>
<em>Merging:</em>
</p>
<p>
"Merging" of incoming requests can be done if some subdirectories of different remote servers are considered equal where the last part of the remote file path leads to the same file content. When specified, the internal cache content is shared and the live download stream is shared. The configuration work consists of setting an "equality list" containing a set of URLs representing the base directories (like <code>http://ftp.debian.org/debian</code> and <code>http://ftp.uni-kl.de/pub/linux/debian</code>).
</p>
<p>
<em>Redirection:</em>
</p>
<p>
With redirection, client requests cause a download from a remote location which is different from what clients requested and believe to receive from. Redirection is an optional feature; if used, it's configured by one or multiple URL(s) pointing to target servers. The URL(s) must include a directory spec which matches the directory level of the URLs in the <code>Merging</code> URL(s), for example all ending with /ubuntu/ for usual Ubuntu mirror URLs. If redirection is not used (i.e. the target URL list is empty) the original URL from client's request is used to get the data.
</p>
<p>
<em>Repository:</em>
</p>
<p>
A (cache) repository is the internal identifier which declares the scope in which <code>Merging/Redirection</code> specs are applied. It also represents the name of an internal cache subdirectory.
</p>
<h3><a name="basic-map"></a>4.3.1 Writing Remap-... configuration</h3>
<p>
When use cases for merging/redirection are identified and a repository name is chosen, these components are written into configuration directives starting with Remap- which follow the simple syntax:
</p>
<p>
Remap-<code>RepositoryName</code>: <code>MergingURLs</code> ; <code>TargetURLs</code> ; <code>OptionalFlags</code>
</p>
<p>
The repository name is a symbolic name which should be chosen carefully and should not be changed afterwards, otherwise the data might become inaccessible for clients until the files are extracted and reimported semi-manually. Internally, this string shares the namespace with host names and/or top directory names of other URLs. Name collisions can cause nasty side effects and should be avoided. Recommended names are made up from alphanumeric or URL-friendly characters. Also, a repository name should not be associated to a real hostname. Examples for good names: <code>archlinux-repo</code>, <code>debianlocal</code>. Examples for bad names: <code>fedora.example.com</code>, <code>_very&weird</code>.
</p>
<p>
The <code>TargetURLs</code> part is optional (see <code>Redirection</code> description above). If multiple targets are specified, the order of servers here defines their order of preference (see also the <code>NetworkTimeout</code> option and additional notes below).
</p>
<p>
Both URL lists simply contain URLs separated by spaces. The strings must be properly URL-encoded. Since all URLs are assumed to belong to http:// protocol and point to a remote directory, the <code>http://</code> protocol prefix and trailing slashes are optional. There is no hard limit to the number of URLs. However, for readability reasons it's recommended to put them into separate list files (see <a href="#cfgfiles">section 4.2</a>) and specify the particular list files with tags like <code>file:urlsDebian.list</code> instead of writing them into a single line. Raw URLs and <code>file:...</code> lists can be mixed.
</p>
<p>
Fully configured Remap lines can look like:
</p>
<p>
<em>Example I:</em>
</p>
<p>
<code>Remap-debrep: ftp.de.debian.org/debian http://ftp.at.debian.org/debian</code>
</p>
<p>
for the use case: small home network, clients have de... or at... servers in their sources.list files and use acng as HTTP proxy. Now the files are still downloaded from at... or de... mirrors depending on the user request, but already cached data is served to both, at... and de... users.
</p>
<p>
<em>Example II:</em>
</p>
<p>
<code>Remap-ubuntu: file:ubumir.lst ; 192.168.17.23/pu ca.archive.ubuntu.com/ubuntu</code>
</p>
<p>
for the use case: small home network, clients have various Ubuntu mirrors (which are listed in ubumir.lst) in their sources.list files and use acng as HTTP proxy. All requests are redirected to a mirror in the /pu directory of some local machine. When that machine is down, Canadian public server is used instead.
</p>
<h3><a name="remap-trickz"></a>4.3.2 Special tricks and additional notes</h3>
<p>
There are some implementation details (partially explained above) and some configuration options related to repository settings which should be mentioned explicitly.
</p>
<p>
The internal cache directory tree follows the URL requests from the clients unless modified by Remapping rules. For proxy-style configuration on the user side, it is always the hostname of the requested URL. But if clients access the apt-cacher-ng server like a regular mirror (not using APT's proxy config) then it's just passed as regular directory name. And at this point, it's possible to use Remapping constructs to access random remote locations while the client assumes to download from a subdirectory of apt-cacher-ng (as http server). This is configured by simply using /some/directory/string/ instead of URLs in the <code>Merging</code> list to let your clients download from http://acngserver/some/directory/string/... paths.
</p>
<p>
If multiple Remap- lines for the same <code>Repository</code> are specified, the contents of both URL lists are merged.
</p>
<p>
On some restricted networks, it may be needed to enforce the use of predefined mirrors. If the <code>ForceManaged</code> option is set, only requests to URL matched in some Remap-... config is allowed.
</p>
<p>
Sometimes, it may be needed to execute a system command before connection to certain machines is established. This is possible by associating commands with a repository declaration, i.e. by storing a file named like <code>repositoryname</code>.hooks in the main configuration directory. It can contain PreUp, Down and DownTimeout settings. PreUp/Down are executed by the system shell and it's up to the administrator to make sure that no malicious code is contained there and that the execution of these commands does not cause significant delays for other apt-cacher-ng users. See package documentation for an exemplary hooks file.
</p>
<p>
If the Redirection part contains multiple URLs, the server prefers to use them in the order of appearance. On success, the first target is used all the time, and so this should be the preferred mirror (note: "success" means getting a started download or a non-critical failure in this context. A "404 File not found" status is not considered critical since client's apt can expect and use it to check the existence of remote files and then change its own behaviour accordingly).
</p>
<p>
And finally, there is an optional third field in the <code>Remap</code> directives which can contain extra flags to modify downloading behavior in the scope of that particular cache repository.
</p>
<ul><li>
<code>keyfile=...</code> The meaning of this setting is: if any real download error (status code 400 and higher) happens on a file which path ends with the specified string then the target server is blacklisted (considered faulty) immediately and this download (and subsequent ones requested by this client connection) are retried from other servers (see <code>TargetURLs</code> description above). Can be used multiple times to define a list. See below for documented example.
</li>
<li>
<code>deltasrc=URL</code> Configures the base URL used to download .debdelta files. The path hierarchy below this URL should correspond to the source URLs and file paths in the cache. Only one URL can be specified at the moment. It is used for explicit mirroring operations, see <a href="howtos.html#mirroring">section 8.13</a> for details.
</li>
<li>
<code>proxy=proxyspec</code> Configures an alternative proxy URL which overrides the global proxy setting in the context of this repository. Can be set empty to disable proxy usage.
</li>
</ul>
<p>
Config example:
</p>
<pre><code>Remap-debrep: file:deb_mirror*.gz ; file:backends_debian ;
keyfile=Release keyfile=.deb
</code></pre>
<p>
If the first mirror from backends_debian goes wild and returns 404 responses for everything then the next candidate will be used. However, while this feature can improve redundancy for certain installations it needs to be used with care! Some file types are allowed to be missing and apt interprets their absence to change its behavior as needed. keyfile= should only match files which have an essential role and which disappearance is undoubtful indication of a broken server.
</p>
<hr><address>Comments to <a href='mailto:blade@debian.org'>blade@debian.org</a>
<br>
[Eduard Bloch, Sat, 08 Oct 2011 23:18:17 +0200]</address></body>
</html>
|
