File: README.md

package info (click to toggle)
golang-github-bmatsuo-lmdb-go 1.8.0+git20170215.a14b5a3-2
  • links: PTS, VCS
  • area: main
  • in suites: bullseye, buster, sid
  • size: 840 kB
  • sloc: ansic: 8,247; makefile: 19
file content (267 lines) | stat: -rw-r--r-- 11,332 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
#lmdb-go [![releases/v1.8.0](https://img.shields.io/badge/release-v1.8.0-375eab.svg)](releases) [![C/v0.9.19](https://img.shields.io/badge/C-v0.9.19-555555.svg)](https://github.com/LMDB/lmdb/blob/mdb.RE/0.9/libraries/liblmdb/CHANGES) [![Build Status](https://travis-ci.org/bmatsuo/lmdb-go.svg?branch=master)](https://travis-ci.org/bmatsuo/lmdb-go)

Go bindings to the OpenLDAP Lightning Memory-Mapped Database (LMDB).

## Packages

Functionality is logically divided into several packages.  Applications will
usually need to import **lmdb** but may import other packages on an as needed
basis.

Packages in the `exp/` directory are not stable and may change without warning.
That said, they are generally usable if application dependencies are managed
and pinned by tag/commit.

Developers concerned with package stability should consult the documentation.

####lmdb [![GoDoc](https://godoc.org/github.com/bmatsuo/lmdb-go/lmdb?status.svg)](https://godoc.org/github.com/bmatsuo/lmdb-go/lmdb) [![stable](https://img.shields.io/badge/stability-stable-brightgreen.svg)](#user-content-versioning-and-stability)

```go
import "github.com/bmatsuo/lmdb-go/lmdb"
```

Core bindings allowing low-level access to LMDB.

####lmdbscan [![GoDoc](https://godoc.org/github.com/bmatsuo/lmdb-go/lmdbscan?status.svg)](https://godoc.org/github.com/bmatsuo/lmdb-go/lmdbscan) [![stable](https://img.shields.io/badge/stability-stable-brightgreen.svg)](#user-content-versioning-and-stability)

```go
import "github.com/bmatsuo/lmdb-go/lmdbscan"
```

A utility package for scanning database ranges. The API is inspired by
[bufio.Scanner](https://godoc.org/bufio#Scanner) and the python cursor
[implementation](https://lmdb.readthedocs.org/en/release/#cursor-class).

####exp/lmdbpool [![GoDoc](https://godoc.org/github.com/bmatsuo/lmdb-go/exp/lmdbpool?status.svg)](https://godoc.org/github.com/bmatsuo/lmdb-go/exp/lmdbpool) [![experimental](https://img.shields.io/badge/stability-experimental-red.svg)](#user-content-versioning-and-stability)


```go
import "github.com/bmatsuo/lmdb-go/exp/lmdbpool"
```

A utility package which facilitates reuse of lmdb.Txn objects using a
sync.Pool.  Naively storing lmdb.Txn objects in sync.Pool can be troublesome.
And the lmdbpool.TxnPool type has been defined as a complete pooling solution
and as reference for applications attempting to write their own pooling
implementation.

The lmdbpool package is relatively new.  But it has a lot of potential utility.
And once the lmdbpool API has been ironed out, and the implementation hardened
through use by real applications it can be integrated directly into the lmdb
package for more transparent integration.  Please test this package and provide
feedback to speed this process up.

####exp/lmdbsync [![GoDoc](https://godoc.org/github.com/bmatsuo/lmdb-go/exp/lmdbsync?status.svg)](https://godoc.org/github.com/bmatsuo/lmdb-go/exp/lmdbsync) [![experimental](https://img.shields.io/badge/stability-experimental-red.svg)](#user-content-versioning-and-stability)


```go
import "github.com/bmatsuo/lmdb-go/exp/lmdbsync"
```

An experimental utility package that provides synchronization necessary to
change an environment's map size after initialization.  The package provides
error handlers to automatically manage database size and retry failed
transactions.

The **lmdbsync** package is usable but the implementation of Handlers are
unstable and may change in incompatible ways without notice.  The use cases of
dynamic map sizes and multiprocessing are niche and the package requires much
more development driven by practical feedback before the Handler API and the
provided implementations can be considered stable.

## Key Features

###Idiomatic API

API inspired by [BoltDB](https://github.com/boltdb/bolt) with automatic
commit/rollback of transactions.  The goal of lmdb-go is to provide idiomatic
database interactions without compromising the flexibility of the C API.

**NOTE:** While the lmdb package tries hard to make LMDB as easy to use as
possible there are compromises, gotchas, and caveats that application
developers must be aware of when relying on LMDB to store their data.  All
users are encouraged to fully read the
[documentation](https://godoc.org/github.com/bmatsuo/lmdb-go/lmdb) so they are
aware of these caveats.

Where the lmdb package and its implementation decisions do not meet the needs
of application developers in terms of safety or operational use the lmdbsync
package has been designed to wrap lmdb and safely fill in additional
functionality.  Consult the
[documentation](https://godoc.org/github.com/bmatsuo/lmdb-go/exp/lmdbsync) for
more information about the lmdbsync package.

###API coverage

The lmdb-go project aims for complete coverage of the LMDB C API (within
reason).  Some notable features and optimizations that are supported:

- Idiomatic subtransactions ("sub-updates") that allow the batching of updates.

- Batch IO on databases utilizing the `MDB_DUPSORT` and `MDB_DUPFIXED` flags.

- Reserved writes than can save in memory copies converting/buffering into
  `[]byte`.

For tracking purposes a list of unsupported features is kept in an
[issue](https://github.com/bmatsuo/lmdb-go/issues/1).

###Zero-copy reads

Applications with high performance requirements can opt-in to fast, zero-copy
reads at the cost of runtime safety.  Zero-copy behavior is specified at the
transaction level to reduce instrumentation overhead.

```
err := lmdb.View(func(txn *lmdb.Txn) error {
    // RawRead enables zero-copy behavior with some serious caveats.
    // Read the documentation carefully before using.
    txn.RawRead = true

    val, err := txn.Get(dbi, []byte("largevalue"), 0)
    // ...
})
```

###Documentation

Comprehensive documentation and examples are provided to demonstrate safe usage
of lmdb.  In addition to [godoc](https://godoc.org/github.com/bmatsuo/lmdb-go)
documentation, implementations of the standand LMDB commands (`mdb_stat`, etc)
can be found in the [cmd/](cmd/) directory and some simple experimental
commands can be found in the [exp/cmd/](exp/cmd) directory.  Aside from
providing minor utility these programs are provided as examples of lmdb in
practice.

##LMDB compared to BoltDB

BoltDB is a quality database with a design similar to LMDB.  Both store
key-value data in a file and provide ACID transactions.  So there are often
questions of why to use one database or the other.

###Advantages of BoltDB

- Nested databases allow for hierarchical data organization.

- Far more databases can be accessed concurrently.

- Operating systems that do not support sparse files do not use up excessive
  space due to a large pre-allocation of file space.  The exp/lmdbsync package
  is intended to resolve this problem with LMDB but it is not ready.

- As a pure Go package bolt can be easily cross-compiled using the `go`
  toolchain and `GOOS`/`GOARCH` variables.

- Its simpler design and implementation in pure Go mean it is free of many
  caveats and gotchas which are present using the lmdb package.  For more
  information about caveats with the lmdb package, consult its
  [documentation](https://godoc.org/github.com/bmatsuo/lmdb-go/lmdb).

###Advantages of LMDB

- Keys can contain multiple values using the DupSort flag.

- Updates can have sub-updates for atomic batching of changes.

- Databases typically remain open for the application lifetime.  This limits
  the number of concurrently accessible databases.  But, this minimizes the
  overhead of database accesses and typically produces cleaner code than
  an equivalent BoltDB implementation.

- Significantly faster than BoltDB.  The raw speed of LMDB easily surpasses
  BoltDB.  Additionally, LMDB provides optimizations ranging from safe,
  feature-specific optimizations to generally unsafe, extremely situational
  ones.  Applications are free to enable any optimizations that fit their data,
  access, and reliability models.

- LMDB allows multiple applications to access a database simultaneously.
  Updates from concurrent processes are synchronized using a database lock
  file.

- As a C library, applications in any language can interact with LMDB
  databases.  Mission critical Go applications can use a database while Python
  scripts perform analysis on the side.

##Build

There is no dependency on shared libraries.  So most users can simply install
using `go get`.

`go get github.com/bmatsuo/lmdb-go/lmdb`

On FreeBSD 10, you must explicitly set `CC` (otherwise it will fail with a
cryptic error), for example:

    CC=clang go test -v ./...

Building commands and running tests can be done with `go` or with `make`

    make bin
    make test
    make check
    make all

On Linux, you can specify the `pwritev` build tag to reduce the number of syscalls
required when committing a transaction. In your own package you can then do

    go build -tags pwritev .

to enable the optimisation.

##Documentation

###Go doc

The `go doc` documentation available on
[godoc.org](https://godoc.org/github.com/bmatsuo/lmdb-go) is the primary source
of developer documentation for lmdb-go.  It provides an overview of the API
with a lot of usage examples.  Where necessary the documentation points out
differences between the semantics of methods and their C counterparts.

###LMDB

The LMDB [homepage](http://symas.com/mdb/) and mailing list
([archives](http://www.openldap.org/lists/openldap-technical/)) are the
official source of documentation regarding low-level LMDB operation and
internals.

Along with an API reference LMDB provides a high-level
[summary](http://symas.com/mdb/doc/starting.html) of the library.  While
lmdb-go abstracts many of the thread and transaction details by default the
rest of the guide is still useful to compare with `go doc`.

###Versioning and Stability

The lmdb-go project makes regular releases with IDs `X.Y.Z`.  All packages
outside of the `exp/` directory are considered stable and adhere to the
guidelines of [semantic versioning](http://semver.org/).

Experimental packages (those packages in `exp/`) are not required to adhere to
semantic versioning.  However packages specifically declared to merely be
"unstable" can be relied on more for long term use with less concern.

The API of an unstable package may change in subtle ways between minor release
versions.  But deprecations will be indicated at least one release in advance
and all functionality will remain available through some method.

##License

Except where otherwise noted files in the lmdb-go project are licensed under
the BSD 3-clause open source license.

The LMDB C source is licensed under the OpenLDAP Public License.

##Links

####[github.com/bmatsuo/raft-mdb](https://github.com/bmatsuo/raft-mdb) ([godoc](https://godoc.org/github.com/bmatsuo/raft-mdb))

An experimental backend for
[github.com/hashicorp/raft](https://github.com/hashicorp/raft) forked from
[github.com/hashicorp/raft-mdb](https://github.com/hashicorp/raft-mdb).

####[github.com/bmatsuo/cayley/graph/lmdb](https://github.com/bmatsuo/cayley/tree/master/graph/lmdb) ([godoc](https://godoc.org/github.com/bmatsuo/cayley/graph/lmdb))

Experimental backend quad-store for
[github.com/google/cayley](https://github.com/google/cayley) based off of the
BoltDB
[implementation](https://github.com/google/cayley/tree/master/graph/bolt).