1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
|
---
title: Loss Calculators
next: /docs/api-config
---
All loss calculators follow the same API: they're classes that are initialized
with optional settings and have a `get_grad` method returning the gradient of
the loss with respect to the model outputs and a `get_loss` method returning the
scalar loss.
## Loss {#loss tag="base class"}
### Loss.\_\_init\_\_ {#loss-init tag="method"}
Initialize the loss calculator.
| Argument | Type | Description |
| ---------- | ------------ | ---------------------------------------------------------------------------------------- |
| `**kwargs` | <tt>Any</tt> | Optional calculator-specific settings. Can also be provided via the [config](#registry). |
### Loss.\_\_call\_\_ {#loss-call tag="method"}
Calculate the gradient and the scalar loss. Returns a tuple of the results of
`Loss.get_grad` and `Loss.get_loss`.
| Argument | Type | Description |
| ----------- | ------------------------ | ----------------------------- |
| `guesses` | <tt>Any</tt> | The model outputs. |
| `truths` | <tt>Any</tt> | The training labels. |
| **RETURNS** | <tt>Tuple[Any, Any]</tt> | The gradient and scalar loss. |
### Loss.get_grad {#loss-get_grad tag="method"}
Calculate the gradient of the loss with respect with the model outputs.
| Argument | Type | Description |
| ----------- | ------------ | -------------------- |
| `guesses` | <tt>Any</tt> | The model outputs. |
| `truths` | <tt>Any</tt> | The training labels. |
| **RETURNS** | <tt>Any</tt> | The gradient. |
### Loss.get_loss {#loss-get_grad tag="method"}
Calculate the scalar loss. Typically returns a float.
| Argument | Type | Description |
| ----------- | ------------ | -------------------- |
| `guesses` | <tt>Any</tt> | The model outputs. |
| `truths` | <tt>Any</tt> | The training labels. |
| **RETURNS** | <tt>Any</tt> | The scalar loss. |
---
## Loss Calculators {#calculators}
### CategoricalCrossentropy {#categorical_crossentropy tag="class"}
<inline-list>
- **Guesses:** <tt>Floats2d</tt>
- **Truths:** <tt>Union[Ints1d, List[int], List[str], Floats2d]</tt>
- **Gradient:** <tt>Floats2d</tt>
- **Loss:** <tt>float</tt>
</inline-list>
A flexible implementation of the common categorical cross-entropy loss that
works on various data types. The `guesses` should represent probabilities and
are expected to be in the range of `[0, 1]`. They can both represent exclusive
classes from multi-class cross-entropy (generally coming from a `softmax` layer)
or could be classwise binary decisions for multi-label cross-entropy (`sigmoid`
layer). The `truths` are most commonly provided as labels in `Ints1d`,
`List[int]` or `List[str]` format. Alternatively, users can provide `truths` as
a `Floats2d` for example to encode label-confidences.
<grid>
```python
### {small="true"}
from thinc.api import CategoricalCrossentropy
loss_calc = CategoricalCrossentropy()
```
```ini
### config.cfg {small="true"}
[loss]
@losses = "CategoricalCrossentropy.v1"
normalize = true
```
</grid>
| Argument | Type | Description |
| ----------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| _keyword-only_ | | |
| `names` | <tt>List[str]</tt> | Label names. Has to be provided when using with List[str] as truths. |
| `normalize` | <tt>bool</tt> | Normalize and divide by number of examples given. |
| `neg_prefix` | <tt>str</tt> | Prefix used to indicate that a label is negative e.g. "!sci-fi". |
| `missing_value` | <tt>Union[str, int]</tt> | Specific label that indicates the value is missing and should not be considered for training/evaluation purposes, e.g. empty string `""` or `0`. |
| `label_smoothing` | <tt>float</tt> | Smoothing-coefficient for label-smoothing. |
### SequenceCategoricalCrossentropy {#sequence_categorical_crossentropy tag="class"}
<inline-list>
- **Guesses:** <tt>List[Floats2d]</tt>
- **Truths:** <tt>List[Union[Ints1d, List[int], List[str], Floats2d]]</tt>
- **Gradient:** <tt>List[Floats2d]</tt>
- **Loss:** <tt>List[float]</tt>
</inline-list>
This loss runs the `CategoricalCrossentropy` over a `List` of `guesses` and
`truths`.
<grid>
```python
### {small="true"}
from thinc.api import SequenceCategoricalCrossentropy
loss_calc = SequenceCategoricalCrossentropy()
```
```ini
### config.cfg {small="true"}
[loss]
@losses = "SequenceCategoricalCrossentropy.v1"
normalize = true
```
</grid>
| Argument | Type | Description |
| ----------------- | ------------------------ | ------------------------------------------------------------------- |
| _keyword-only_ | | |
| `names` | <tt>List[str]</tt> | Label names. Has to be provided when using with List[str] as truths |
| `normalize` | <tt>bool</tt> | Normalize and divide by number of examples given. |
| `neg_prefix` | <tt>str</tt> | Symbol that indicates that a label is negative e.g. "!sci-fi". |
| `missing_value` | <tt>Union[str, int]</tt> | Symbol for "missing value" among the labels. |
| `label_smoothing` | <tt>float</tt> | Smoothing-coefficient for label-smoothing. |
### L2Distance {#l2distance tag="class"}
<inline-list>
- **Guesses:** <tt>Floats2d</tt>
- **Truths:** <tt>Floats2d</tt>
- **Gradient:** <tt>Floats2d</tt>
- **Loss:** <tt>float</tt>
</inline-list>
<grid>
```python
### {small="true"}
from thinc.api import L2Distance
loss_calc = L2Distance()
```
```ini
### config.cfg {small="true"}
[loss]
@losses = "L2Distance.v1"
normalize = true
```
</grid>
| Argument | Type | Description |
| -------------- | ------------- | ------------------------------------------------- |
| _keyword-only_ | | |
| `normalize` | <tt>bool</tt> | Normalize and divide by number of examples given. |
### CosineDistance {#cosine_distance tag="function"}
<inline-list>
- **Guesses:** <tt>Floats2d</tt>
- **Truths:** <tt>Floats2d</tt>
- **Gradient:** <tt>Floats2d</tt>
- **Loss:** <tt>float</tt>
</inline-list>
<grid>
```python
### {small="true"}
from thinc.api import CosineDistance
loss_calc = CosineDistance(ignore_zeros=False)
```
```ini
### config.cfg {small="true"}
[loss]
@losses = "CosineDistance.v1"
normalize = true
ignore_zeros = false
```
</grid>
| Argument | Type | Description |
| -------------- | ------------- | ------------------------------------------------- |
| _keyword-only_ | | |
| `normalize` | <tt>bool</tt> | Normalize and divide by number of examples given. |
| `ignore_zeros` | <tt>bool</tt> | Don't count zero vectors. |
---
## Usage via config and function registry {#registry}
Defining the loss calculators in the [config](/docs/usage-config) will return
the **initialized object**. Within your script, you can then call it or its
methods and pass in the data.
<grid>
```ini
### config.cfg {small="true"}
[loss]
@losses = "L2Distance.v1"
normalize = true
```
```python
### Usage {small="true"}
from thinc.api import registry, Config
config = Config().from_disk("./config.cfg")
resolved = registry.resolve(config)
loss_calc = resolved["loss"]
loss = loss_calc.get_grad(guesses, truths)
```
</grid>
|
