1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
|
Buffer
==========
[](https://godoc.org/github.com/djherbis/buffer)
[](https://github.com/djherbis/buffer/releases/latest)
[](LICENSE.txt)
[](https://travis-ci.org/djherbis/buffer)
[](https://coveralls.io/r/djherbis/buffer?branch=master)
[](https://goreportcard.com/report/github.com/djherbis/buffer)
Usage
------------
The following buffers provide simple unique behaviours which when composed can create complex buffering strategies. For use with github.com/djherbis/nio for Buffered io.Pipe and io.Copy implementations.
For example:
```go
import (
"github.com/djherbis/buffer"
"github.com/djherbis/nio"
"io/ioutil"
)
// Buffer 32KB to Memory, after that buffer to 100MB chunked files
buf := buffer.NewUnboundedBuffer(32*1024, 100*1024*1024)
nio.Copy(w, r, buf) // Reads from r, writes to buf, reads from buf writes to w (concurrently).
// Buffer 32KB to Memory, discard overflow
buf = buffer.NewSpill(32*1024, ioutil.Discard)
nio.Copy(w, r, buf)
```
Supported Buffers
------------
#### Bounded Buffers ####
Memory: Wrapper for bytes.Buffer
File: File-based buffering. The file never exceeds Cap() in length, no matter how many times its written/read from. It accomplishes this by "wrapping" around the fixed max-length file when the data gets too long but there is available freed space at the beginning of the file. The caller is responsible for closing and deleting the file when done.
```go
import (
"ioutil"
"os"
"github.com/djherbis/buffer"
)
// Create a File-based Buffer with max size 100MB
file, err := ioutil.TempFile("", "buffer")
if err != nil {
return err
}
defer os.Remove(file.Name())
defer file.Close()
buf := buffer.NewFile(100*1024*1024, file)
// A simpler way:
pool := buffer.NewFilePool(100*1024*1024, "") // "" -- use temp dir
buf, err := pool.Get() // allocate the buffer
if err != nil {
return err
}
defer pool.Put(buf) // close and remove the allocated file for the buffer
```
Multi: A fixed length linked-list of buffers. Each buffer reads from the next buffer so that all the buffered data is shifted upwards in the list when reading. Writes are always written to the first buffer in the list whose Len() < Cap().
```go
import (
"github.com/djherbis/buffer"
)
mem := buffer.New(32*1024)
file := buffer.NewFile(100*1024*1024, someFileObj)) // you'll need to manage Open(), Close() and Delete someFileObj
// Buffer composed of 32KB of memory, and 100MB of file.
buf := buffer.NewMulti(mem, file)
```
#### Unbounded Buffers ####
Partition: A queue of buffers. Writes always go to the last buffer in the queue. If all buffers are full, a new buffer is "pushed" to the end of the queue (generated by a user-given function). Reads come from the first buffer, when the first buffer is emptied it is "popped" off the queue.
```go
import (
"github.com/djherbis/buffer"
)
// Create 32 KB sized-chunks of memory as needed to expand/contract the buffer size.
buf := buffer.NewPartition(buffer.NewMemPool(32*1024))
// Create 100 MB sized-chunks of files as needed to expand/contract the buffer size.
buf = buffer.NewPartition(buffer.NewFilePool(100*1024*1024, ""))
```
Ring: A single buffer which begins overwriting the oldest buffered data when it reaches its capacity.
```go
import (
"github.com/djherbis/buffer"
)
// Create a File-based Buffer with max size 100MB
file := buffer.NewFile(100*1024*1024, someFileObj) // you'll need to Open(), Close() and Delete someFileObj.
// If buffered data exceeds 100MB, overwrite oldest data as new data comes in
buf := buffer.NewRing(file) // requires BufferAt interface.
```
Spill: A single buffer which when full, writes the overflow to a given io.Writer.
-> Note that it will actually "spill" whenever there is an error while writing, this should only be a "full" error.
```go
import (
"github.com/djherbis/buffer"
"github.com/djherbis/nio"
"io/ioutil"
)
// Buffer 32KB to Memory, discard overflow
buf := buffer.NewSpill(32*1024, ioutil.Discard)
nio.Copy(w, r, buf)
```
#### Empty Buffer ####
Discard: Reads always return EOF, writes goto ioutil.Discard.
```go
import (
"github.com/djherbis/buffer"
)
// Reads will return io.EOF, writes will return success (nil error, full write) but no data was written.
buf := buffer.Discard
```
Custom Buffers
------------
Feel free to implement your own buffer, just meet the required interface (Buffer/BufferAt) and compose away!
```go
// Buffer Interface used by Multi and Partition
type Buffer interface {
Len() int64
Cap() int64
io.Reader
io.Writer
Reset()
}
// BufferAt interface used by Ring
type BufferAt interface {
Buffer
io.ReaderAt
io.WriterAt
}
```
Installation
------------
```sh
go get github.com/djherbis/buffer
```
|
