File: hacking.html

package info (click to toggle)
libtorrent-rasterbar 1.1.13-1
  • links: PTS, VCS
  • area: main
  • in suites: bullseye, sid
  • size: 17,248 kB
  • sloc: cpp: 134,519; sh: 4,666; python: 2,591; ansic: 1,754; makefile: 914
file content (241 lines) | stat: -rw-r--r-- 11,709 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" />
<title>libtorrent hacking</title>
<meta name="author" content="Arvid Norberg, arvid&#64;libtorrent.org" />
<meta name=viewport content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="style.css" type="text/css" />
<link rel="stylesheet" type="text/css" href="rst.css" />
<style type="text/css">
/* Hides from IE-mac \*/
* html pre { height: 1%; }
/* End hide from IE-mac */
</style>
</head>
<body>
<div class="document" id="libtorrent-hacking">
    <div id="container">
    <a href="index.html"><table id="header">
    <tr><td id="orange"></td>
    <td id="logo">libtorrent</td></tr>
    </table></a>
    <div id="main">
<h1 class="title">libtorrent hacking</h1>
<table class="docinfo" frame="void" rules="none">
<col class="docinfo-name" />
<col class="docinfo-content" />
<tbody valign="top">
<tr><th class="docinfo-name">Author:</th>
<td>Arvid Norberg, <a class="last reference external" href="mailto:arvid&#64;libtorrent.org">arvid&#64;libtorrent.org</a></td></tr>
<tr><th class="docinfo-name">Version:</th>
<td>1.1.13</td></tr>
</tbody>
</table>
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#terminology" id="id1">terminology</a></li>
<li><a class="reference internal" href="#structure" id="id2">structure</a><ul>
<li><a class="reference internal" href="#session-impl" id="id3">session_impl</a></li>
<li><a class="reference internal" href="#session" id="id4">session</a></li>
<li><a class="reference internal" href="#torrent-handle" id="id5">torrent_handle</a></li>
<li><a class="reference internal" href="#torrent" id="id6">torrent</a></li>
<li><a class="reference internal" href="#peer-connection" id="id7">peer_connection</a></li>
<li><a class="reference internal" href="#peer-list" id="id8">peer_list</a></li>
<li><a class="reference internal" href="#piece-picker" id="id9">piece_picker</a></li>
<li><a class="reference internal" href="#torrent-info" id="id10">torrent_info</a></li>
</ul>
</li>
<li><a class="reference internal" href="#threads" id="id11">threads</a></li>
<li><a class="reference internal" href="#disk-cache" id="id12">disk cache</a></li>
</ul>
</div>
<p>This describe some of the internals of libtorrent. If you're looking for
something to contribute, please take a look at the <a class="reference external" href="todo.html">todo list</a>.</p>
<div class="section" id="terminology">
<h1>terminology</h1>
<p>This section describes some of the terminology used throughout the
libtorrent source. Having a good understanding of some of these keywords
helps understanding what's going on.</p>
<p>A <em>piece</em> is a part of the data of a torrent that has a SHA-1 hash in
the .torrent file. Pieces are almost always a power of two in size, but not
necessarily. Each piece is plit up in <em>blocks</em>, which is a 16 kiB. A block
never spans two pieces. If a piece is smaller than 16 kiB or not divisible
by 16 kiB, there are blocks smaller than that.</p>
<p>16 kiB is a de-facto standard of the largest transfer unit in the bittorrent
protocol. Clients typically reject any request for larger pieces than this.</p>
<p>The <em>piece picker</em> is the part of a bittorrent client that is responsible for
the logic to determine which requests to send to peers. It doesn't actually
pick full pieces, but blocks (from pieces).</p>
<p>The file layout of a torrent is represented by <em>file storage</em> objects. This
class contains a list of all files in the torrent (in a well defined order),
the size of the pieces and implicitly the total size of the whole torrent and
number of pieces. The file storage determines the mapping from <em>pieces</em>
to <em>files</em>. This representation may be quite complex in order to keep it extremely
compact. This is useful to load very large torrents without exploding in memory
usage.</p>
<p>A <em>torrent</em> object represents all the state of swarm download. This includes
a piece picker, a list of peer connections, file storage (torrent file). One
important distiction is between a connected peer (<em>peer_connection</em>) and a peer
we just know about, and may have been connected to, and may connect to in the
future (<em>torrent_peer</em>). The list of (not connected) peers may grow very large
if not limited (through tracker responses, DHT and peer exchange). This list
is typically limited to a few thousand peers.</p>
<p>The <em>peer_list</em> maintains a potentially large list of known peers for a swarm
(not necessarily connected).</p>
</div>
<div class="section" id="structure">
<h1>structure</h1>
<p>This is the high level structure of libtorrent. Bold types are part of the public
interface:</p>
<img alt="hacking.png" src="hacking.png" />
<div class="section" id="session-impl">
<h2>session_impl</h2>
<p>This is the session state object, containing all session global information, such as:</p>
<blockquote>
<ul class="simple">
<li>the list of all torrents <tt class="docutils literal">m_torrent</tt>.</li>
<li>the list of all peer connections <tt class="docutils literal">m_connections</tt>.</li>
<li>the global rate limits <tt class="docutils literal">m_settings</tt>.</li>
<li>the DHT state <tt class="docutils literal">m_dht</tt>.</li>
<li>the port mapping state, <tt class="docutils literal">m_upnp</tt> and <tt class="docutils literal">m_natpmp</tt>.</li>
</ul>
</blockquote>
</div>
<div class="section" id="session">
<h2>session</h2>
<p>This is the public interface to the session. It implements pimpl (pointer to implementation)
in order to hide the internal representation of the <tt class="docutils literal">session_impl</tt> object from the user and
make binary compatibility simpler to maintain.</p>
</div>
<div class="section" id="torrent-handle">
<h2>torrent_handle</h2>
<p>This is the public interface to a <tt class="docutils literal">torrent</tt>. It holds a weak reference to the internal
<tt class="docutils literal">torrent</tt> object and manipulates it by sending messages to the network thread.</p>
</div>
<div class="section" id="torrent">
<h2>torrent</h2>
</div>
<div class="section" id="peer-connection">
<h2>peer_connection</h2>
</div>
<div class="section" id="peer-list">
<h2>peer_list</h2>
</div>
<div class="section" id="piece-picker">
<h2>piece_picker</h2>
</div>
<div class="section" id="torrent-info">
<h2>torrent_info</h2>
</div>
</div>
<div class="section" id="threads">
<h1>threads</h1>
<p>libtorrent starts 3 to 5 threads.</p>
<blockquote>
<ul class="simple">
<li>The first thread is the main thread that will sit
idle in a <tt class="docutils literal">select()</tt> call most of the time. This thread runs the main loop
that will send and receive data on all connections. In reality it's typically
not actually in <tt class="docutils literal">select()</tt>, but in <tt class="docutils literal">kqueue()</tt>, <tt class="docutils literal">epoll_wait()</tt> or <tt class="docutils literal">poll</tt>,
depending on operating system.</li>
<li>The second thread is the disk I/O thread. All disk read and write operations
are passed to this thread and messages are passed back to the main thread when
the operation completes.</li>
<li>The third thread is the SHA-1 hash thread. By default there's only one hash thread,
but on multi-core machines downloading at very high rates, libtorrent can be configured
to start any number of hashing threads, to take full use of multi core systems.
(see <tt class="docutils literal"><span class="pre">settings_pack::aio_threads</span></tt>).</li>
<li>The fourth and fifth threads are spawned by asio on systems that don't support
asynchronous host name resolution, in order to simulate non-blocking <tt class="docutils literal">getaddrinfo()</tt>.</li>
</ul>
</blockquote>
</div>
<div class="section" id="disk-cache">
<h1>disk cache</h1>
<p>The disk cache implements <em>ARC</em>, Adaptive Replacement Cache. This consists of a number of LRUs:</p>
<ol class="arabic simple">
<li>lru L1 (recently used)</li>
<li>lru L1 ghost (recently evicted)</li>
<li>lru L2 (frequently used)</li>
<li>lru L2 ghost (recently evicted)</li>
<li>volatile read blocks</li>
<li>write cache (blocks waiting to be flushed to disk)</li>
</ol>
<img alt="disk_cache.png" src="disk_cache.png" />
<p>These LRUs are stored in <tt class="docutils literal">block_cache</tt> in an array <tt class="docutils literal">m_lru</tt>.</p>
<p>The cache algorithm works like this:</p>
<pre class="literal-block">
if (L1-&gt;is_hit(piece)) {
        L1-&gt;erase(piece);
        L2-&gt;push_back(piece);
} else if (L2-&gt;is_hit(piece)) {
        L2-&gt;erase(piece);
        L2-&gt;push_back(page);
} else if (L1-&gt;size() == cache_size) {
        L1-&gt;pop_front();
        L1-&gt;push_back(piece);
} else {
        if (L1-&gt;size() + L2-&gt;size() == 2*chache_size) {
                L2-&gt;pop_front();
        }
        L1-&gt;push_back(piece);
}
</pre>
<p>It's a bit more complicated since within L1 and L2 in this pseudo code
have to separate the ghost entries and the in-cache entries.</p>
<p>Note that the most recently used and more frequently used pieces are at
the <em>back</em> of the lists. Iterating over a list gives you low priority pieces
first.</p>
<p>In libtorrent pieces are cached, not individual blocks, a single peer would
typically trigger many cache hits when downloading a piece. Since ARC is
sensitive to extra cache hits (a piece is moved to L2 the second time it's
hit) libtorrent only move the cache entry on cache hits when it's hit by
another peer than the last peer that hit it.</p>
<p>Another difference compared to the ARC paper is that libtorrent caches pieces,
which aren't necessarily fully allocated. This means the real cache size is
specified in number of blocks, not pieces, so there's not clear number of pieces
to keep in the ghost lists. There's an <tt class="docutils literal">m_num_arc_pieces</tt> member in <tt class="docutils literal">block_cache</tt>
that defines the <em>arc cache size</em>, in pieces, rather than blocks.</p>
</div>

    </div>
    </div>
    <div id="gradient"></div>
    <div id="footer">
    <table>
    <tr>
      <td><a href="index.html">home</a></td>
      <td><a href="https://blog.libtorrent.org">blog</a></td>
      <td><a href="utp.html">uTP</a></td>
    </tr>
    <tr>
      <td><a href="https://sourceforge.net/projects/libtorrent/files/libtorrent/">download</a></td>
      <td><a href="reference.html">documentation</a></td>
      <td><a href="dht_store.html">DHT put extension</a></td>
    </tr>
    <tr>
      <td><a href="https://sourceforge.net/projects/libtorrent/files/py-libtorrent/">python bindings</a></td>
      <td><a href="features.html">features</a></td>
      <td><a href="dht_sec.html">DHT security extension</a></td>
    </tr>
    <tr>
      <td><a href="https://sourceforge.net/p/libtorrent/mailman/libtorrent-discuss/">mailing list archive</a></td>
      <td><a href="contributing.html">contributing</a></td>
      <td><a href="streaming.html">streaming</a></td>
    </tr>
    <tr>
      <td><a href="https://github.com/arvidn/libtorrent/issues">report a bug</a></td>
      <td><a href="building.html">building</a></td>
      <td><a href="bittorrent.pdf">bittorrent slides</a></td>
    </tr>
    </table>
    </div>
    <div id="filler"></div>

</div>
</body>
</html>