1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
# Blackbox Protobuf Mitmproxy Addon
## Description
This directory contains an addon for mitmproxy (<https://mitmproxy.org/>) which
uses the Blackbox Protobuf library to decode and edit proxied protobuf messages.
## Installation
1. Clone the Blackboxprotobuf repository:
```
git clone https://github.com/nccgroup/blackboxprotobuf.git
```
2. Update the submodules which contain dependencies:
```
cd blackboxprotobuf/
git submodule update --init
```
3. Run mitmproxy with the addon:
```
mitmproxy -s mitmproxy/bbpb.py
```
### Alternative Installation
1. Install the `bbpb` python package in the same context as
`mitmproxy`.
```
pip install bbpb
```
If `mitmproxy` is installed within a virtual environment, then the `bbpb`
package needs to be install within that virtual environment.
2. Download the `bbpb.py` from <https://github.com/nccgroup/blackboxprotobuf/blob/master/mitmproxy/bbpb.py>
3. Run mitmproxy with the addon script:
```
mitmproxy -s mitmproxy/bbpb.py
```
## Usage
### Passive Decoding
The Blackbox Protobuf addon provides a content view to automatically decode
protobuf and gRPC messages and websockets displayed in mitmproxy. The content
view will use saved types if available (eg. to name fields or change types),
but will default to an anonymous decoding.
The content view should also work in mitmweb, but commands for editing messages
and types will not be available.
### Persisting Type Data
The addon will remember edited types and the associated endpoints, but this
data will be lost if mitmproxy is shut down or the addon is reloaded.
The `bbpb_project_file` option in the mitmproxy configuration will cause the
addon to load types from the provided file and then automatically write back
any changes. If you have an extensive number of options, it's recommended to
back up the project file regularly to ensure data isn't overwritten.
The types can also be manually saved or loaded using the `:bbpb.project.save`
and `:bbpb.project.load` commands.
### Commands
Message and type editing functionality is provided via mitmproxy commands.
The commands operate on the currently selected flow. Most commands take a
`flow_part` argument, which indicates where the desired protobuf payload is in the
message. Currently supported flow parts are:
* `request-body`
* `response-body`
* `websocket` (for `bbpb.edit`, not supported with `bbpb.edit_type`)
* `websocket-request` (for `bbpb.edit_type`, not supported for `bbpb.edit`)
* `websocket-response` (for `bbpb.edit_type`, not supported for `bbpb.edit`)
Command arguments support tab completion.
#### `:bbpb.edit`
The `:bbpb.edit` command will open a text editor with the protobuf decoded to
JSON, similar to editing other HTTP messages in mitmproxy. The edited JSON
payload will be re-encoded to protobuf to be replayed or resumed.
The `request-body` and `response-body` flow parts will edit protobuf messages
in the HTTP body. The `websocket` flow parts will edit the last message in the
websocket flow.
The addon does not support protobuf payloads elsewhere in the HTTP message at
the moment, though you should be able to customize the addon for a particular
use case.
#### `:bbpb.edit_type`
The `:bbpb.edit_type` command will open a text editor to edit the type
definition used for the protobuf data. This can be used to add field names or
change a field's type.
If the endpoint is assigned a named type, then the named type is edited and the
changes will be applied to all other endpoints using the same named type. If
the type is not named, then the edited type definition is remembered just for
that endpoint.
#### `:bbpb.new_type`
The `:bbpb.new_type` command will attach a name to the current type and store
it in `known_types`. A named type can then be applied to multiple endpoints and
you can switch between named types at-will without losing the type definition.
#### `:bbpb.apply_type`
The `:bbpb.apply_type` command will apply a previously saved named type to the
endpoint/protobuf payload. Applying the type name of `(clear)` will remove the
saved type for this endpoint, causing the decoding to be reset.
Warning: If the current endpoint has an edited type which is not named, the
edited type definition will be lost when a named type is applied.
If the current endpoint is associated with another named type, the named type
will be safe and can be reapplied later.
#### `:bbpb.del_type`
The `:bbpb.del_type` command can be used to remove a named type. Any endpoints associated with that named type will also be reset.
#### `:bbpb.project.save` and `:bbpb.project.load`
These commands will manually save/load a one-off Blackbox Protobuf project JSON
file. This file contains all edited type definitions and all endpoint to type
definition mappings.
## gRPC Note
This addon has been tested with a python gRPC and client and is able to
intercept messages by default. However, in the past mitmproxy has had some
issues with intercepting gRPC connections from certain applications, and would
register connections as PRI methods, or fail to intercept the request.
The issue appears to be that some gRPC implementations use a unique `grpc-exp`
ALPN value which proxies do not know how to handle. One work around that has
worked in the past is cloning mitmproxy and changing if conditions which check
for `h2` to also work for `grpc-exp`.
Another potential solution is the addon modifications described at
<https://github.com/mitmproxy/mitmproxy/issues/3052#issuecomment-1676020354>,
but this has not been tested with the Blackbox Protobuf addon yet.
|
