1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
|
<div align="center">
<img src="https://github.com/tiny-dnn/tiny-dnn/blob/master/docs/logo/TinyDNN-logo-letters-alpha-version.png"><br><br>
</div>
-----------------
[](https://gitter.im/tiny-dnn/users) [](http://tiny-dnn.readthedocs.io/) [](https://raw.githubusercontent.com/tiny-dnn/tiny-dnn/master/LICENSE) [](https://coveralls.io/github/tiny-dnn/tiny-dnn?branch=master)
**tiny-dnn** is a C++11 implementation of deep learning. It is suitable for deep learning on limited computational resource, embedded systems and IoT devices.
| **`Linux/Mac OS`** | **`Windows`** |
|------------------|-------------|
|[](https://travis-ci.org/tiny-dnn/tiny-dnn)|[](https://ci.appveyor.com/project/tinydnn/tiny-dnn)|
## Table of contents
* [Features](#features)
* [Comparison with other libraries](#comparison-with-other-libraries)
* [Supported networks](#supported-networks)
* [Dependencies](#dependencies)
* [Build](#build)
* [Examples](#examples)
* [Contributing](#contributing)
* [References](#references)
* [License](#license)
* [Gitter rooms](#gitter-rooms)
Check out the [documentation](http://tiny-dnn.readthedocs.io/) for more info.
## What's New
- 2016/9/14 [tiny-dnn v1.0.0alpha is released!](https://github.com/tiny-dnn/tiny-dnn/releases/tag/v1.0.0a)
- 2016/8/7 tiny-dnn is now moved to organization account, and rename into tiny-dnn :)
- 2016/7/27 [tiny-dnn v0.1.1 released!](https://github.com/tiny-dnn/tiny-dnn/releases/tag/v0.1.1)
## Features
- reasonably fast, without GPU
- with TBB threading and SSE/AVX vectorization
- 98.8% accuracy on MNIST in 13 minutes training (@Core i7-3520M)
- portable & header-only
- Run anywhere as long as you have a compiler which supports C++11
- Just include tiny_dnn.h and write your model in C++. There is nothing to install.
- easy to integrate with real applications
- no output to stdout/stderr
- a constant throughput (simple parallelization model, no garbage collection)
- work without throwing an exception
- [can import caffe's model](https://github.com/tiny-dnn/tiny-dnn/tree/master/examples/caffe_converter)
- simply implemented
- be a good library for learning neural networks
## Comparison with other libraries
||tiny-dnn|[caffe](https://github.com/BVLC/caffe)|[Theano](https://github.com/Theano/Theano)|[TensorFlow](https://www.tensorflow.org/)|
|---|---|---|---|---|
|Prerequisites|__Nothing__(Optional:TBB,OpenMP)|BLAS,Boost,protobuf,glog,gflags,hdf5, (Optional:CUDA,OpenCV,lmdb,leveldb etc)|Numpy,Scipy,BLAS,(optional:nose,Sphinx,CUDA etc)|numpy,six,protobuf,(optional:CUDA,Bazel)|
|Modeling By|C++ code|Config File|Python Code|Python Code|
|GPU Support|No|Yes|Yes|Yes|
|Installing|Unnecessary|Necessary|Necessary|Necessary|
|Windows Support|Yes|No*|Yes|No*|
|Pre-Trained Model|Yes(via caffe-converter)|Yes|No*|No*|
*unofficial version is available
## Supported networks
### layer-types
- core
- fully-connected
- dropout
- linear operation
- power
- convolution
- convolutional
- average pooling
- max pooling
- deconvolutional
- average unpooling
- max unpooling
- normalization
- contrast normalization (only forward pass)
- batch normalization
- split/merge
- concat
- slice
- elementwise-add
### activation functions
* tanh
* sigmoid
* softmax
* rectified linear(relu)
* leaky relu
* identity
* exponential linear units(elu)
### loss functions
* cross-entropy
* mean squared error
* mean absolute error
* mean absolute error with epsilon range
### optimization algorithms
* stochastic gradient descent (with/without L2 normalization and momentum)
* adagrad
* rmsprop
* adam
## Dependencies
Nothing. All you need is a C++11 compiler.
## Build
tiny-dnn is header-ony, so *there's nothing to build*. If you want to execute sample program or unit tests, you need to install [cmake](https://cmake.org/) and type the following commands:
```
cmake .
```
Then open .sln file in visual studio and build(on windows/msvc), or type ```make``` command(on linux/mac/windows-mingw).
Some cmake options are available:
|options|description|default|additional requirements to use|
|-----|-----|----|----|
|USE_TBB|Use [Intel TBB](https://www.threadingbuildingblocks.org/) for parallelization|OFF<sup>1</sup>|[Intel TBB](https://www.threadingbuildingblocks.org/)|
|USE_OMP|Use OpenMP for parallelization|OFF<sup>1</sup>|[OpenMP Compiler](http://openmp.org/wp/openmp-compilers/)|
|USE_SSE|Use Intel SSE instruction set|ON|Intel CPU which supports SSE|
|USE_AVX|Use Intel AVX instruction set|ON|Intel CPU which supports AVX|
|USE_NNPACK|Use NNPACK for convolution operation|OFF|[Acceleration package for neural networks on multi-core CPUs](https://github.com/Maratyszcza/NNPACK)|
|USE_OPENCL|Enable/Disable OpenCL support (experimental)|OFF|[The open standard for parallel programming of heterogeneous systems](https://www.khronos.org/opencl/)|
|USE_LIBDNN|Use Greentea LinDNN for convolution operation with GPU via OpenCL (experimental)|OFF|[An universal convolution implementation supporting CUDA and OpenCL](https://github.com/naibaf7/libdnn)|
|USE_SERIALIZER|Enable model serialization|ON<sup>2</sup>|-|
|BUILD_TESTS|Build unit tests|OFF<sup>3</sup>|-|
|BUILD_EXAMPLES|Build example projects|OFF|-|
|BUILD_DOCS|Build documentation|OFF|[Doxygen](http://www.doxygen.org/)|
<sup>1</sup> tiny-dnn use c++11 standard library for parallelization by default
<sup>2</sup> If you don't use serialization, you can switch off to speedup compilation time.
<sup>3</sup> tiny-dnn uses [Google Test](https://github.com/google/googletest) as default framework to run unit tests. No pre-installation required, it's automatically downloaded during CMake configuration.
For example, type the following commands if you want to use intel TBB and build tests:
```bash
cmake -DUSE_TBB=ON -DBUILD_TESTS=ON .
```
## Customize configurations
You can edit include/config.h to customize default behavior.
## Examples
construct convolutional neural networks
```cpp
#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;
using namespace tiny_dnn::layers;
void construct_cnn() {
using namespace tiny_dnn;
network<sequential> net;
// add layers
net << conv<tan_h>(32, 32, 5, 1, 6) // in:32x32x1, 5x5conv, 6fmaps
<< ave_pool<tan_h>(28, 28, 6, 2) // in:28x28x6, 2x2pooling
<< fc<tan_h>(14 * 14 * 6, 120) // in:14x14x6, out:120
<< fc<identity>(120, 10); // in:120, out:10
assert(net.in_data_size() == 32 * 32);
assert(net.out_data_size() == 10);
// load MNIST dataset
std::vector<label_t> train_labels;
std::vector<vec_t> train_images;
parse_mnist_labels("train-labels.idx1-ubyte", &train_labels);
parse_mnist_images("train-images.idx3-ubyte", &train_images, -1.0, 1.0, 2, 2);
// declare optimization algorithm
adagrad optimizer;
// train (50-epoch, 30-minibatch)
net.train<mse>(optimizer, train_images, train_labels, 30, 50);
// save
net.save("net");
// load
// network<sequential> net2;
// net2.load("net");
}
```
construct multi-layer perceptron(mlp)
```cpp
#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;
using namespace tiny_dnn::layers;
void construct_mlp() {
network<sequential> net;
net << fc<sigmoid>(32 * 32, 300)
<< fc<identity>(300, 10);
assert(net.in_data_size() == 32 * 32);
assert(net.out_data_size() == 10);
}
```
another way to construct mlp
```cpp
#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;
void construct_mlp() {
auto mynet = make_mlp<tan_h>({ 32 * 32, 300, 10 });
assert(mynet.in_data_size() == 32 * 32);
assert(mynet.out_data_size() == 10);
}
```
more sample, read examples/main.cpp or [MNIST example](https://github.com/tiny-dnn/tiny-dnn/tree/master/examples/mnist) page.
## Contributing
Since deep learning community is rapidly growing, we'd love to get contributions from you to accelerate tiny-dnn development!
For a quick guide to contributing, take a look at the [Contribution Documents](docs/developer_guides/How-to-contribute.md).
## References
[1] Y. Bengio, [Practical Recommendations for Gradient-Based Training of Deep Architectures.](http://arxiv.org/pdf/1206.5533v2.pdf)
arXiv:1206.5533v2, 2012
[2] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, [Gradient-based learning applied to document recognition.](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf)
Proceedings of the IEEE, 86, 2278-2324.
other useful reference lists:
- [UFLDL Recommended Readings](http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Recommended_Readings)
- [deeplearning.net reading list](http://deeplearning.net/reading-list/)
## License
The BSD 3-Clause License
## Gitter rooms
We have a gitter rooms for discussing new features & QA.
Feel free to join us!
<table>
<tr>
<td><b> developers </b></td>
<td> https://gitter.im/tiny-dnn/developers </td>
</tr>
<tr>
<td><b> users </b></td>
<td> https://gitter.im/tiny-dnn/users </td>
</tr>
</table>
|
