1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
|
# breader
[](https://godoc.org/github.com/shenwei356/breader)
[](https://goreportcard.com/report/github.com/shenwei356/breader)
breader (Buffered File Reader), asynchronous parsing and pre-processing while
reading file. Safe cancellation is also supported.
## Example
1). Simple example with default parameters (`ChunkSize`: 100;
`BufferSize`: #. of CPUs, `ProcessFunc`: trimming new-line symbol)
```go
import "github.com/shenwei356/breader"
reader, err := breader.NewDefaultBufferedReader(file)
checkErr(err)
for chunk := range reader.Ch {
checkError(chunk.Err)
for _, data := range chunk.Data {
line := data.(string)
fmt.Println(line)
}
}
```
2). Example with custom pre-processing function: splitting line to slice.
**Note the processing of interface{} containing slice,
using a custom struct is recommended**.
```go
type Slice []string // custom type
fn := func(line string) (interface{}, bool, error) {
line = strings.TrimRight(line, "\n")
if line == "" || line[0] == '#' { // ignoring blank line and comment line
return "", false, nil
}
items := strings.Split(line, "\t")
if len(items) != 2 {
return items, false, nil
}
return Slice(items), true, nil
}
reader, err := breader.NewBufferedReader(file, runtime.NumCPU(), 100, fn)
checkErr(err)
for chunk := range reader.Ch {
checkError(chunk.Err)
for _, data := range chunk.Data {
// do not simply use: data.(slice)
fmt.Println(data.(Slice))
}
}
```
3). Example with custom pre-processing function: creating object from line data.
```go
type string2int struct {
id string
value int
}
fn := func(line string) (interface{}, bool, error) {
line = strings.TrimRight(line, "\n")
if line == "" || line[0] == '#' {
return nil, false, nil
}
items := strings.Split(line, "\t")
if len(items) != 2 {
return nil, false, nil
}
if items[0] == "" || items[1] == "" {
return nil, false, nil
}
id := items[0]
value, err := strconv.Atoi(items[1])
if err != nil {
return nil, false, err
}
return string2int{id, value}, true, nil
}
reader, err := breader.NewBufferedReader(file, runtime.NumCPU(), 100, fn)
checkErr(err)
for chunk := range reader.Ch {
checkError(chunk.Err)
for _, data := range chunk.Data {
obj := data.(string2int)
// handle of the string2int object
}
}
```
4). Example of cancellation. **Note that `range chanel` is buffered, therefore,
`for-select-case` is used.**
```go
reader, err := breader.NewBufferedReader(testfile, 0, 1, breader.DefaultFunc)
checkErr(err)
// note that range is bufferd. using range will be failed
// for chunk := range reader.Ch {
LOOP:
for {
select {
case chunk := <-reader.Ch:
if chunk.Err != nil {
t.Log(chunk.Err)
return
}
reader.Cancel()
break LOOP
default:
}
}
```
## License
[MIT License](https://github.com/shenwei356/breader/blob/master/LICENSE)
|
