1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
|
#<cldoc:Usage>
Configuration and execution of `yrmcdsd`.
`yrmcdsd` is the server program of yrmcds. Its built-in configurations are
reasonably good to run as a stand-alone server. For replication, you need
to specify a virtual IP address and configure a clustering software such as
[keepalived][].
Files
-----
Usually, yrmcds is installed under `/usr/local`.
* /usr/local/etc/yrmcds.conf
The default configuration file.
* /usr/local/sbin/yrmcdsd
The program.
Configuration (memcache)
------------------------
These options are to configure memcache protocol:
* `user` (Default: none)
If set, the program will try to `setuid` to the given user.
* `group` (Default: none)
If set, the program will try to `setgid` to the given group.
* `virtual_ip` (Default: 127.0.0.1)
The master's virtual IP address.
Both IPv4 and IPv6 addresses are supported.
* `port` (Default: 11211)
memcache protocol port number.
* `repl_port` (Default: 11213)
The replication protocol port number.
* `bind_ip` (Default: none)
Space delimited IP addresses to listen on.
Default is to listen on all addresses.
Note that `virtual_ip` will be bound in addition to `bind_ip`.
* `max_connections` (Default: 0)
Maximum number of client connections. 0 means unlimited.
* `temp_dir` (Default: /var/tmp)
Directory to store temporary files for large objects.
* `log.threshold` (Default: info)
Log threshold. Possible values: `error`, `warning`, `info`, `debug`.
* `log.file` (Default: standard error)
Logs will be written to this file.
* `buckets` (Default: 1000000)
Hash table size.
* `max_data_size` (Default: 1M)
The maximum object size.
* `heap_data_limit` (Default: 256K)
Objects larger than this will be stored in temporary files.
* `repl_buffer_size` (Default: 30)
The replication buffer size. Unit is MiB.
* `secure_erase` (Default: false)
If `true`, object memory will be cleared as soon as the object is removed.
* `lock_memory` (Default: false)
If `true`, prevent memory from being swapped using `mlockall`.
* `memory_limit` (Default: 1024M)
The amount of memory allowed for yrmcdsd.
* `workers` (Default: 8)
The number of worker threads.
* `gc_interval` (Default: 10)
The interval between garbage collections in seconds.
* `slave_timeout` (Default: 10)
slave_timeout specifies how many seconds to wait for heartbeats from slaves before the connection is forcibly closed.
Configuration (counter)
------------------------
yrmcds has an extra protocol called **counter** in addition to memcache
protocol. The counter protocol can be used to manage resource usage
dynamically just like (distributed) semaphores.
The counter protocol has these configuration options:
* `counter.enable` (Default: `false`)
If `true`, the counter extension is enabled.
* `counter.port` (Default: 11215)
TCP port number used for the counter protocol.
* `counter.max_connections` (Default: 0)
The maximum number of connections for the counter protocol.
0 means unlimited.
* `counter.buckets` (Default: 1000000)
The size of the counter hash table.
* `counter.stat_interval` (Default: 86400)
The interval of measuring the maximum resource consumption.
The default is 86400 seconds = 1 day.
Running yrmcdsd
---------------
Just run `yrmcdsd`.
The program does not run as traditional UNIX daemons.
Instead, use [upstart][] or [systemd][] to run `yrmcdsd` as a background service.
Do not forget to increase the maximum file descriptors.
A sample upstart script is available at [etc/upstart](https://github.com/cybozu/yrmcds/blob/master/etc/upstart).
Replication
-----------
The replication of yrmcds servers is based on a virtual IP address.
To assign a virtual IP address to a live server, a clustering software
such as [keepalived][] or [Pacemaker][pacemaker] is used.
Our recommendation is to use [keepalived][].
### keepalived
For stability, keepalived should be configured in non-preemptive mode.
A sample configuration file is available at [etc/keepalived.conf](https://github.com/cybozu/yrmcds/blob/master/etc/keepalived.conf).
### Replication status
`yrmcdsd`, either it is master or slave, dumps the current replication
status to the log file when it receives `SIGUSR1` signal.
```
$ kill -USR1 <pid of yrmcdsd>
```
### How replication works
`yrmcdsd` **periodically watches** if the server owns the virtual IP address
or not. If `yrmcdsd` finds it holds the virtual IP address, that server
will promote itself as the new master. All others become slaves.
Slaves connect to the master through the virtual IP address. If the
connection resets, `yrmcdsd` waits some seconds to see if the virtual IP
address is assigned. If assigned, the slave becomes the new master. If
not, the slave drops all objects then try to connect to the new master.
Since the master node is elected dynamically by [keepalived][], each
`yrmcdsd` should have the same configuration parameters.
### counter protocol
Resource counters are not replicated.
<a name="secure_erase" />
Erasing confidential data in memory
-----------------------------------
If `secure_erase` configuration option is enabled, `yrmcdsd` will erase
memory used by objects as soon as they are removed or their lifetime
expire.
This ensures confidential data such as crypto keys will never be leaked
after expiration.
Consider setting `max_data_size` equal to `heap_data_limit` when you
enable this option to avoid writing confidential data into persistent
storage.
Locking memory
--------------
If `lock_memory` configuration option is enabled, `yrmcdsd` will lock
memory with `mlockall` to prevent them from being swapped.
**Use of this option may cause troubles**; locking enough memory for
yrmcds will require the process having `CAP_IPC_LOCK` capability or
sufficiently large `RLIMIT_MEMLOCK` resource limit.
See [man page][mlockall] for details.
[keepalived]: http://www.keepalived.org/
[pacemaker]: http://clusterlabs.org/wiki/Main_Page
[upstart]: http://upstart.ubuntu.com/
[systemd]: http://www.freedesktop.org/wiki/Software/systemd/
[mlockall]: http://manpages.ubuntu.com/manpages/trusty/en/man2/mlockall.2.html
|
