1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
|
Command Line Usage Overview
===========================
You can run PATHspider from the command line. In order for the Observer to
work, you will need permissions to capture raw packets from the network
interface. You may also need elevated privileges when generating traffic using
raw sockets or to modify the local TCP/IP stack. This will require you to use
``sudo`` or equivalent in order to run PATHspider if you are not logged in as
the root user.
.. code-block:: text
# pspdr --help
usage: pspdr [-h] [--verbose] COMMAND ...
PATHspider will spider the paths.
optional arguments:
-h, --help show this help message and exit
--verbose Enable verbose logging
Commands:
filter Pre-process a target list
measure Perform a PATHspider measurement
observe Passively observe network traffic
test Run the built in test suite
Spider safely!
Performing Active Measurement
-----------------------------
PATHspider provides the "measure" command to perform active traffic generation
and observation of that traffic for path transparency measurement. Based on
the observations made, paths are assigned conditions such as
`ecn.connectivity.works` indicating that the use of ECN does not cause
connectivity impairment between the vantage point and the particular target.
It is possible to enable the output of flow records along with the derived
observations using the ``--output-flows`` flag. This will generate considerably
more output and so is disabled by default.
You may specify input and output files using flags, however by default these
are set to be stdin and stdout and so you can, and are recommended to, use
shell redirection instead. To see output as it is written to the file, you can
pipe the output to ``tee`` to print it on the screen while also saving it to
a file.
You will be required to set your interface name and PATHspider will not start
if it detects that the chosen interface is not active.
.. code-block:: text
# pspdr measure --help
usage: pspdr measure [-h] [-i INTERFACE] [-w WORKERS] [--input INPUTFILE]
[--output OUTPUTFILE] [--output-flows]
PLUGIN ...
optional arguments:
-h, --help show this help message and exit
-i INTERFACE, --interface INTERFACE
The interface to use for the observer. (Default: eth0)
-w WORKERS, --workers WORKERS
Number of workers to use. (Default: 100)
--input INPUTFILE A file containing a list of PATHspider jobs. Defaults
to standard input.
--output OUTPUTFILE The file to output results data to. Defaults to
standard output.
--output-flows Include flow results in output.
Plugins:
The following plugins are available for use:
tfo TCP Fast Open
ecn Explicit Congestion Notification
h2 HTTP/2
dscp Differentiated Services Codepoints
dnsresolv Simple Input List DNS Resolver
udpopts UDP Options Trailer
udpzero UDP Zero Checksum
Spider safely!
Quickstart Example
~~~~~~~~~~~~~~~~~~
You can run a small study using the ECN plugin and the included
``webtest.ndjson`` file to measure path transparency to ECN for a small selection
of web servers and save the results in ``results.ndjson`` (ensure to change the
interface name to match an active interface on your machine):
.. code-block:: shell
pspdr measure -i eth0 ecn </usr/share/doc/pathspider/examples/webtest.ndjson >results.ndjson
.. note::
If you've not installed PATHspider from apt, you will find the webinput.ndjson
example input file in the examples folder of the source distribution.
Performing Passive Observation
------------------------------
PATHspider provides the "observe" command to perform passive traffic
observation for path transparency measurement. In this version of PATHspider
we do not attempt to determine path conditions during passive observation, and
instead only output flow records. This may change in future versions of
PATHspider.
You can list the available chains with ``--list-chains`` and then select any
number of chains that you would like to use. It is recommended that you include
the `basic` chain as this will add the IP addresses and port numbers to the
flow records.
You may specify the output file using a flag, however by default this is set to
be ``/dev/stdout`` and so you can, and are recommended to, use shell
redirection instead. To see output as it is written to the file, you can
pipe the output to ``tee`` to print it on the screen while also saving it to
a file. You will be required to set your interface name and PATHspider will not
start if it detects that the chosen interface is not active.
It is also possible to perform offline analysis of a PCAP file using the
"observe" command. Instead of an interface name, pass the name of the pcap file
to ``-i`` instead. The PCAP file must have a ``.pcap`` extension to be
recognised.
.. code-block:: text
usage: pspdr observe [-h] [--list-chains] [-i INTERFACE] [--output OUTPUTFILE]
[chains [chains ...]]
positional arguments:
chains Observer chains to use
optional arguments:
-h, --help show this help message and exit
--list-chains Prints a list of available chains
-i INTERFACE, --interface INTERFACE
The interface to use for the observer. (Default: eth0)
--output OUTPUTFILE The file to output results data to. Defaults to
standard output.
Quickstart Example
~~~~~~~~~~~~~~~~~~
You can observe network traffic passively to perform observations without
actively generating traffic. In this case no input file is needed.
.. code-block:: shell
pspdr observe -i eth0 basic tcp ecn >results.ndjson
Data Formats
------------
PATHspider uses `newline delimited JSON <http://ndjson.org/>`_ (ndjson) for
both the input and output format. The ndjson format gives flexibility in the
contents of the data as different tests may require data to remain associated
with jobs so that it can be present in the final output (the Alexa ranking of a
webserver, for example), or used as part of the test (running tests against
authoritative DNS servers and needing to know a domain for which the server
should be authoritative).
.. _defaultoutput:
Input Format
~~~~~~~~~~~~
At a minimum, each job should contain an IP address in a ``dip`` field.
Depending on the plugin in use, more details may be required. Refer to the
documentation for the specific plugin for more information.
Output Format
~~~~~~~~~~~~~
For each job, the output JSON dictionary will contain the original job
information, a computed path using available information and a set of
conditions seen for the path as generated by the plugins.
With ``--output-flows`` enabled, PATHspider's output will include an additional
field in the JSON dictionary for each job containing an array of flow records,
one for each configuration. Usually one record will be for the baseline (A)
connection, and one for the experimental (B) connection. These JSON records
contain the original job information, any information added by the connection
functions and any information added by the Observer.
Additionally, internal information may be retained:
+------------+----------------------------------------------------------------+
| Key | Description |
+============+================================================================+
| config | 0 for baseline, 1..n for experimental |
+------------+----------------------------------------------------------------+
| spdr_state | 0 = OK, 1 = TIMEOUT, 2 = FAILED, 3 = SKIPPED |
+------------+----------------------------------------------------------------+
For detail on the values in individual plugins, see the section for that plugin
later in this documentation.
|
