1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
# JSON Package Design
Author: John Doak(jdoak@microsoft.com)
## Why?
This project needs a special type of marshal/unmarshal not directly supported
by the encoding/json package.
The need revolves around a few key wants/needs:
- unmarshal and marshal structs representing JSON messages
- fields in the messgage not in the struct must be maintained when unmarshalled
- those same fields must be marshalled back when encoded again
The initial version used map[string]interface{} to put in the keys that
were known and then any other keys were put into a field called AdditionalFields.
This has a few negatives:
- Dual marshaling/unmarshalling is required
- Adding a struct field requires manually adding a key by name to be encoded/decoded from the map (which is a loosely coupled construct), which can lead to bugs that aren't detected or have bad side effects
- Tests can become quickly disconnected if those keys aren't put
in tests as well. So you think you have support working, but you
don't. Existing tests were found that didn't test the marshalling output.
- There is no enforcement that if AdditionalFields is required on one struct, it should be on all containers
that don't have custom marshal/unmarshal.
This package aims to support our needs by providing custom Marshal()/Unmarshal() functions.
This prevents all the negatives in the initial solution listed above. However, it does add its own negative:
- Custom encoding/decoding via reflection is messy (as can be seen in encoding/json itself)
Go proverb: Reflection is never clear
Suggested reading: https://blog.golang.org/laws-of-reflection
## Important design decisions
- We don't want to understand all JSON decoding rules
- We don't want to deal with all the quoting, commas, etc on decode
- Need support for json.Marshaler/Unmarshaler, so we can support types like time.Time
- If struct does not implement json.Unmarshaler, it must have AdditionalFields defined
- We only support root level objects that are \*struct or struct
To faciliate these goals, we will utilize the json.Encoder and json.Decoder.
They provide streaming processing (efficient) and return errors on bad JSON.
Support for json.Marshaler/Unmarshaler allows for us to use non-basic types
that must be specially encoded/decoded (like time.Time objects).
We don't support types that can't customer unmarshal or have AdditionalFields
in order to prevent future devs from forgetting that important field and
generating bad return values.
Support for root level objects of \*struct or struct simply acknowledges the
fact that this is designed only for the purposes listed in the Introduction.
Outside that (like encoding a lone number) should be done with the
regular json package (as it will not have additional fields).
We don't support a few things on json supported reference types and structs:
- \*map: no need for pointers to maps
- \*slice: no need for pointers to slices
- any further pointers on struct after \*struct
There should never be a need for this in Go.
## Design
## State Machines
This uses state machine designs that based upon the Rob Pike talk on
lexers and parsers: https://www.youtube.com/watch?v=HxaD_trXwRE
This is the most common pattern for state machines in Go and
the model to follow closesly when dealing with streaming
processing of textual data.
Our state machines are based on the type:
```go
type stateFn func() (stateFn, error)
```
The state machine itself is simply a struct that has methods that
satisfy stateFn.
Our state machines have a few standard calls
- run(): runs the state machine
- start(): always the first stateFn to be called
All state machines have the following logic:
* run() is called
* start() is called and returns the next stateFn or error
* stateFn is called
- If returned stateFn(next state) is non-nil, call it
- If error is non-nil, run() returns the error
- If stateFn == nil and err == nil, run() return err == nil
## Supporting types
Marshalling/Unmarshalling must support(within top level struct):
- struct
- \*struct
- []struct
- []\*struct
- []map[string]structContainer
- [][]structContainer
**Term note:** structContainer == type that has a struct or \*struct inside it
We specifically do not support []interface or map[string]interface
where the interface value would hold some value with a struct in it.
Those will still marshal/unmarshal, but without support for
AdditionalFields.
## Marshalling
The marshalling design will be based around a statemachine design.
The basic logic is as follows:
* If struct has custom marshaller, call it and return
* If struct has field "AdditionalFields", it must be a map[string]interface{}
* If struct does not have "AdditionalFields", give an error
* Get struct tag detailing json names to go names, create mapping
* For each public field name
- Write field name out
- If field value is a struct, recursively call our state machine
- Otherwise, use the json.Encoder to write out the value
## Unmarshalling
The unmarshalling desin is also based around a statemachine design. The
basic logic is as follows:
* If struct has custom marhaller, call it
* If struct has field "AdditionalFields", it must be a map[string]interface{}
* Get struct tag detailing json names to go names, create mapping
* For each key found
- If key exists,
- If value is basic type, extract value into struct field using Decoder
- If value is struct type, recursively call statemachine
- If key doesn't exist, add it to AdditionalFields if it exists using Decoder
|
