1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
|
<!DOCTYPE html>
<html lang='en'>
<head>
<meta charset="utf-8">
<meta name="author" content="Philip Semanchuk">
<meta name="copyright" content="All contents © 2021 Philip Semanchuk">
<meta name="keywords" content="python sysv system v ipc semaphore shared memory message queue">
<title>System V IPC for Python - Semaphores, Shared Memory and Message Queues</title>
<style>
dt {
font-family: monospace;
font-weight: bold;
padding-bottom: .33em;
margin-top: 1em;
}
span[lang] { font-style: italic; }
span.param {
font-family: monospace;
font-style: italic;
}
span.book_title {
text-decoration: underline;
}
pre { margin-left: 2em; }
li { margin-top: 1em; margin-bottom: 1em; }
fieldset, legend {
background-color: #d0d0a9;
}
/* This style is only present on the local version of the readme.
In the online version, the RSS feed is displayed. */
div.rss { display: none; }
</style>
</head>
<body>
<h2>System V IPC for Python - Semaphores, Shared Memory and Message Queues</h2>
<p>This describes the <code>sysv_ipc</code> module which gives Python access
to System V inter-process semaphores, shared memory and message queues
on most (all?) *nix flavors. Examples include macOS/OS X, Linux, FreeBSD,
OpenSolaris 2008.11, HPUX, and AIX.
It might also work under Windows with a library like
<a href="http://www.cygwin.com/">Cygwin</a>.
</p>
<p>It works with Python ≥ 3.6. It's released
under a <a href="http://creativecommons.org/licenses/BSD/">BSD license</a>.
</p>
<p>You can <strong>download
<a href="sysv_ipc-1.1.0.tar.gz">sysv_ipc version 1.1.0</a>
</strong>
(<a href="sysv_ipc-1.1.0.md5.txt">[md5 sum]</a>,
<a href="sysv_ipc-1.1.0.sha1.txt">[sha1 sum]</a>,
<a href="sysv_ipc-1.1.0.sha256.txt">[sha256 sum]</a>)
which contains the source code, setup.py, installation instructions and
<a href="#samples">sample code</a>. You can read about
<a href="history.html#current">all of the changes in this version</a>.
</p>
<p>
You might also want to read
about some <a href="#bugs">known bugs</a>.
</p>
<p>You might be interested in the very similar module
<a href="/philip/posix_ipc/"><code>posix_ipc</code></a>
which provides Python access to POSIX IPC primitives. POSIX IPC is a little
easier to use than SysV IPC, but not all operating systems support it
completely.
</p>
<h2>Module <code>sysv_ipc</code></h2>
<p>Jump to <a href="#semaphore">semaphores</a>,
<a href="#shared_memory">shared memory</a>, or
<a href="#message_queue">message queues</a>.</p>
<h3>Module Functions</h3>
<dl>
<dt>attach(id, [address = None, [flags = 0]])</dt>
<dd>Attaches the (existing) shared memory that has the given <code>id</code> and
returns a new SharedMemory object. See
<a href="#attach">SharedMemory.attach()</a> for details on the
<code>address</code> and <code>flags</code> parameters.
<p>This method is useful only under fairly unusual circumstances.
You probably don't need it.
</p>
</dd>
<dt>ftok(path, id, [silence_warning = False])</dt>
<dd>Calls <code>ftok(path, id)</code>. Note that
<a href="#ftok_weakness"><code>ftok()</code> has limitations</a>, and this
function will issue a warning to that effect unless
<code>silence_warning</code> is True.
</dd>
<dt>remove_semaphore(id)</dt>
<dd>Removes the semaphore with the given <code>id</code>.</dd>
<dt>remove_shared_memory(id)</dt>
<dd>Removes the shared memory with the given <code>id</code>.</dd>
<dt>remove_message_queue(id)</dt>
<dd>Removes the message queue with the given <code>id</code>.</dd>
</dl>
<h3>Module Constants</h3>
<dl>
<dt>IPC_CREAT, IPC_EXCL and IPC_CREX</dt>
<dd><code>IPC_CREAT</code> and <code>IPC_EXCL</code> are flags used when
creating IPC objects. They're
bitwise unique and can be ORed together. <code>IPC_CREX</code> is
shorthand for <code>IPC_CREAT | IPC_EXCL</code>.
<p>When passed to an IPC object's constructor, <code>IPC_CREAT</code> indicates
that you want to create a new object or open an existing one. If you want
the call to fail if an object with that key already exists, specify
the <code>IPC_EXCL</code> flag, too.
</p>
</dd>
<dt>IPC_PRIVATE</dt>
<dd>This is a special value that can be passed in place of a key. It implies that
the IPC object should be available only to the creating process or its
child processes (e.g. those created with <code>fork()</code>).
</dd>
<dt>KEY_MIN and KEY_MAX</dt>
<dd>Denote the range of keys that this module accepts. Your OS might
limit keys to a smaller range depending on the typedef of
<code>key_t</code>.
<p>Keys randomly generated by this module are in the range
<code>1 ≤ key ≤ SHRT_MAX</code>.
That's type-safe unless your OS has a very bizarre
definition of <code>key_t</code>.
</p>
</dd>
<dt>SEMAPHORE_VALUE_MAX</dt>
<dd>The maximum value of a semaphore.
</dd>
<dt>PAGE_SIZE</dt>
<dd>The operating system's memory page size, in bytes. It's probably a good
idea to make shared memory segments some multiple of this size.
</dd>
<dt>SEMAPHORE_TIMEOUT_SUPPORTED</dt>
<dd>True if the platform supports timed semaphore waits, False otherwise.</dd>
<dt>SHM_RDONLY</dt>
<dd>Pass this flag to <code>SharedMemory.attach()</code> to attach the segment
read-only.</dd>
<dt>SHM_RND</dt>
<dd>You probably don't need this, but it can be used when attaching shared
memory to force the address to be
rounded down to SHMLBA. See your system's man page for <code>shmat()</code>
for more information.
</dd>
<dt>SHM_HUGETLB, SHM_NORESERVE and SHM_REMAP</dt>
<dd>You probably don't need these. They're Linux-specific flags that can
be passed to the SharedMemory
constructor, or to the <code>.attach()</code> function in the case of
SHM_REMAP. See your system's man page for <code>shmget()</code>
and <code>shmat()</code> for more information.
</dd>
</dl>
<h3>Module Errors</h3>
<p>In addition to standard Python errors (e.g. <code>ValueError</code>),
this module raises custom errors. These errors cover
situations specific to IPC.
</p>
<dl>
<dt>Error</dt>
<dd>The base error class for all the custom errors in this module. This
error is occasionally raised on its own but you'll almost
always see a more specific error.
</dd>
<dt>InternalError</dt>
<dd>Indicates that something has gone very wrong in the module code. Please
report this to the maintainer.
</dd>
<dt>PermissionsError</dt>
<dd>Indicates that you've attempted something that the permissions on the
IPC object don't allow.
</dd>
<dt>ExistentialError</dt>
<dd>Indicates an error related to the existence or non-existence of
an IPC object.
</dd>
<dt>BusyError</dt>
<dd>Raised when a semaphore call to <code>.P()</code> or <code>.Z()</code> either times out
or would be forced to wait when its <code>block</code> attribute is False.
</dd>
<dt>NotAttachedError</dt>
<dd>Raised when a process attempts to read from or write to a shared memory
segment to which it is not attached.
</dd>
</dl>
<h3 id="semaphore">The Semaphore Class</h3>
<p>This is a handle to a semaphore.</p>
<h4>Methods</h4>
<dl>
<dt>Semaphore(key, [flags = 0, [mode = 0600, [initial_value = 0]]])</dt>
<dd>Creates a new semaphore or opens an existing one.
<p><span class="param">key</span> must be <code>None</code>,
<code>IPC_PRIVATE</code> or
an integer > <code>KEY_MIN</code> and ≤ <code>KEY_MAX</code>. If the key
is <code>None</code>, the module chooses a random unused key.
</p>
<p>The <span class="param">flags</span> specify whether you want to create a
new semaphore or open an existing one.
</p>
<ul>
<li>With <span class="param">flags</span> set to the <strong>default</strong> of <code>0</code>, the module attempts
to <strong>open an existing</strong> semaphore identified by <span class="param">key</span> and raises
a <code>ExistentialError</code> if that semaphore doesn't exist.
</li>
<li>With <span class="param">flags</span> set to <code>IPC_CREAT</code>, the module
<strong>opens</strong> the semaphore identified by
<span class="param">key</span> <strong>or creates</strong> a new
one if no such semaphore exists. Using <code>IPC_CREAT</code> by itself
is not recommended. (See <a href="#sem_init">Semaphore Initialization</a>.)
</li>
<li>With <span class="param">flags</span> set to
<code>IPC_CREX</code> (<code>IPC_CREAT | IPC_EXCL</code>),
the module
<strong>creates a new semaphore</strong> identified by <span class="param">key</span>. If a
semaphore with that key already exists, the call raises an
<code>ExistentialError</code>.
<strong>The <span class="param">initial_value</span> is ignored unless
both of these flags are specified</strong> or
if the semaphore is read-only.
</li>
</ul>
<p>When opening an existing semaphore, <span class="param">mode</span> is ignored.
</p>
</dd>
<dt>acquire([timeout = None, [delta = 1]])</dt>
<dd>Waits (conditionally) until the semaphore's value is > 0 and then
returns, decrementing the semaphore.
<p>The <span class="param">timeout</span> (which can be a float) specifies how
many seconds this call should wait, if at all.
</p>
<p>The semantics of the timeout <a href="#v0_3">changed a little in
version 0.3</a>.
</p>
<ul>
<li>A <span class="param">timeout</span> of None (the default)
implies no time limit. The call will not return until its wait
condition is satisfied.
</li>
<li>When <span class="param">timeout</span> is 0, the call
raises a <code>BusyError</code> if it can't immediately
acquire the semaphore. Since it will
return immediately if <em>not</em> asked to wait, this can be
thought of as "non-blocking" mode.
</li>
<li>When the <span class="param">timeout</span> is > 0, the call
will wait no longer than <span class="param">timeout</span>
seconds before either returning (having acquired the semaphore)
or raising a <code>BusyError</code>.
</li>
</ul>
<p>When the call returns, the semaphore's value decreases by
<span class="param">delta</span>
(or more precisely, <code>abs(<span class="param">delta</span>)</code>)
which defaults to 1.
</p>
<p>On platforms that don't support the <code>semtimedop()</code> API call,
all timeouts (including zero) are treated as infinite. The call
will not return until its wait condition is satisfied.
</p>
<p>Most platforms provide <code>semtimedop()</code>. macOS is a
notable exception. The module's Boolean constant
<code>SEMAPHORE_TIMEOUT_SUPPORTED</code>
is True on platforms that support <code>semtimedop()</code>.
</p>
</dd>
<dt>release([delta = 1])</dt>
<dd>
Releases (increments) the semaphore.
<p>The semaphore's value increases by <span class="param">delta</span>
(or more precisely, <code>abs(<span class="param">delta</span>)</code>)
which defaults to 1.
</p>
</dd>
<dt>P()</dt>
<dd>A synonym for <code>.acquire()</code> that takes the same parameters.
<p>"P" stands for
<span lang="nl">prolaag</span> or <span lang="nl">probeer te verlagen</span>
(try to decrease), the original name given by
<a href="http://en.wikipedia.org/wiki/Semaphore_(programming)">Edsger Dijkstra</a>.
</p>
</dd>
<dt>V()</dt>
<dd>A synonym for <code>.release()</code> that takes the same parameters.
<p>"V" stands for
<span lang="nl">verhoog</span> (increase), the original name given by
<a href="http://en.wikipedia.org/wiki/Semaphore_(programming)">Edsger Dijkstra</a>.
</p>
</dd>
<dt>Z([timeout = None])</dt>
<dd>Blocks until zee zemaphore is zero.
<p><span class="param">Timeout</span> has
the same meaning as described in <code>.acquire()</code>.
</p>
</dd>
<dt>remove()</dt>
<dd>
Removes (deletes) the semaphore from the system.
<p>As far as I can tell, the effect of deleting a semaphore that
other processes are still using is OS-dependent. Check your system's
man pages for <code>semctl(IPC_RMID)</code>.
</p>
</dd>
</dl>
<h4>Attributes</h4>
<dl>
<dt>key (read-only)</dt>
<dd>The key passed in the call to the constructor.</dd>
<dt>id (read-only)</dt>
<dd>The id assigned to this semaphore by the OS.</dd>
<dt>value</dt>
<dd>The integer value of the semaphore.</dd>
<dt>undo</dt>
<dd>Defaults to False.
<p>When True, operations that change the
semaphore's value will be undone (reversed) when
the process exits. Note that when a process exits, an undo operation
may imply that a semaphore's value should become negative or
exceed its maximum.
Behavior in this case is system-dependent, which means that
<strong>using this flag can make your code non-portable</strong>.
</p>
</dd>
<dt>block</dt>
<dd>
Defaults to True, which means that calls to <code>acquire()</code> and
<code>release()</code> will not return
until their wait conditions are satisfied.
<p>When False, these calls
will not block but will instead raise an error if they are unable
to return immediately.
</p>
</dd>
<dt>mode</dt>
<dd>The semaphore's permission bits.
<p>Tip: the following Python code will display
the mode in octal:<br>
<code>print int(str(my_sem.mode), 8)</code>
</p>
</dd>
<dt>uid</dt>
<dd>The semaphore's user id.</dd>
<dt>gid</dt>
<dd>The semaphore's group id.</dd>
<dt>cuid (read-only)</dt>
<dd>The semaphore creator's user id.</dd>
<dt>cgid (read-only)</dt>
<dd>The semaphore creator's group id.</dd>
<dt>last_pid (read-only)</dt>
<dd>The PID of the process that last called <code>semop()</code> (<code>.P()</code>,
<code>.V()</code> or <code>.Z()</code>) on this semaphore.
<p>Linux and macOS also set this when the semaphore's value is changed,
although doing so disagrees with the POSIX specification.
See <a href='https://bugzilla.kernel.org/show_bug.cgi?id=112271'>Linux kernel bug 112271</a>.
</p>
</dd>
<dt>waiting_for_nonzero (read-only)</dt>
<dd>The number of processes waiting for the value of the semaphore to become
non-zero (i.e. the number waiting in a call to <code>.P()</code>).
</dd>
<dt>waiting_for_zero (read-only)</dt>
<dd>The number of processes waiting for the value of the semaphore to become
zero (i.e. the number waiting in a call to <code>.Z()</code>).
</dd>
<dt>o_time (read-only)</dt>
<dd>The last time <code>semop()</code> (i.e. <code>.P()</code>, <code>.V()</code> or
<code>.Z()</code>) was called on this semaphore.
</dd>
</dl>
<h4>Context Manager Support</h4>
<p>These semaphores provide <code>__enter__()</code> and <code>__exit__()</code>
methods so they can be used in context managers. For instance --
</p>
<pre>
with sysv_ipc.Semaphore(name) as sem:
# Do something...
</pre>
<p>Entering the context acquires the semaphore, exiting the context releases
the semaphore. See <code>demo4/child.py</code> for a complete example.
</p>
<h3 id="shared_memory">The SharedMemory Class</h3>
<p>This is a handle to a shared memory segment. </p>
<h4>Buffer Protocol Support</h4>
<p>
In addition to the
methods and attributes described below, <code>SharedMemory</code> supports
Python's buffer protocol, which means you can create <code>bytearray</code>
and <code>memoryview</code> objects based on a <code>SharedMemory</code> segment.
See <code>demos/buffer_protocol</code> for an example.
</p>
<h4>Methods</h4>
<dl>
<dt>SharedMemory(key, [flags = 0, [mode = 0600, [size = 0 or PAGE_SIZE, [init_character = ' ']]]])</dt>
<dd>Creates a new shared memory segment or opens an existing one.
The memory is automatically attached.
<p><span class="param">key</span> must be <code>None</code>,
<code>IPC_PRIVATE</code> or
an integer > <code>0</code> and ≤ <code>KEY_MAX</code>. If the key
is <code>None</code>, the module chooses a random unused key.
</p>
<p>The <span class="param">flags</span> specify whether you want to create a
new shared memory segment or open an existing one.
</p>
<ul>
<li>With <span class="param">flags</span> set to the
<strong>default</strong> of <code>0</code>, the module attempts
to <strong>open an existing</strong> shared memory segment identified by
<span class="param">key</span> and raises
a <code>ExistentialError</code> if it doesn't exist.
</li>
<li>With <span class="param">flags</span> set to <strong><code>IPC_CREAT</code></strong>, the module
<strong>opens</strong> the shared memory segment identified
by <span class="param">key</span> <strong>or
creates</strong> a new one if no such segment exists.
Using <code>IPC_CREAT</code> by itself
is not recommended. (See <a href="#mem_init">Memory Initialization</a>.)
</li>
<li>With <span class="param">flags</span> set to
<strong><code>IPC_CREX</code></strong> (<code>IPC_CREAT | IPC_EXCL</code>),
the module
<strong>creates</strong> a new shared memory segment identified by
<span class="param">key</span>. If
a segment with that key already exists, the call raises
a <code>ExistentialError</code>.
<p>When both <code>IPC_CREX</code> is specified
and the caller has write permission, each byte in the new memory segment will be
initialized to the value of <span class="param">init_character</span>.
</p>
</li>
</ul>
<p>The value of <span class="param">size</span> depends on whether
one is opening an existing segment or creating a new one.
</p>
<ul>
<li>When opening an existing segment, <span class="param">size</span>
must be ≤ the existing segment's size. Zero is
always valid.
</li>
<li>When creating an new segment,
many (most? all?) operating systems insist on a <span class="param">size</span>
> <code>0</code>.
In addition, some round the size
up to the next multiple of PAGE_SIZE.
</li>
</ul>
<p>This module supplies a default
<span class="param">size</span> of <code>PAGE_SIZE</code> when
<code>IPC_CREX</code> is specified and <code>0</code> otherwise.
</p>
</dd>
<dt id="attach">attach([address = None, [flags = 0]])</dt>
<dd>
Attaches this process to the shared memory. The memory must be attached
before calling <code>.read()</code> or <code>.write()</code>. Note that the
constructor automatically attaches the memory
so you won't need to call this method unless you explicitly detach it
and then want to use it again.
<p>The address parameter allows one to specify (as a Python long) a memory
address at which to attach the segment. Passing None (the default)
is equivalent to passing NULL to <code>shmat()</code>. See that
function's man page for details.
</p>
<p>The flags are mostly only relevant if one specifies a specific address.
One exception is the flag <code>SHM_RDONLY</code> which, surprisingly,
attaches the segment read-only.
</p>
<p>Note that on some (and perhaps all) platforms, each call to <code>.attach()</code>
increments the system's "attached" count. Thus, if each call to
<code>.attach()</code> isn't paired with a call to <code>.detach()</code>,
the system's "attached" count for the shared memory segment will not
go to zero when the process exits. As a result, the shared memory
segment may not disappear even when its creator calls <code>.remove()</code>
and exits.
</p>
</dd>
<dt>detach()</dt>
<dd>Detaches this process from the shared memory.</dd>
<dt>read([byte_count = 0, [offset = 0]])</dt>
<dd>Reads up to <span class="param">byte_count</span> bytes from the
shared memory segment starting at <span class="param">offset</span>
and returns them as a bytes object (which is the same as a <code>str</code>
under Python 2).
<p>If <span class="param">byte_count</span> is zero (the default) the
entire buffer is returned.
</p>
<p>This method will never attempt to read past the end of the shared
memory segment, even when
<span class="param">offset</span> + <span class="param">byte_count</span>
exceeds the memory segment's size. In that case, the bytes
from <span class="param">offset</span> to the end of the segment are returned.
</p>
</dd>
<dt>write(some_bytes, [offset = 0])</dt>
<dd>Writes bytes (i.e. <code>str</code> in Python 2) to the shared memory,
starting at <span class="param">offset</span>. Passing a Unicode object
may work, but doing so is unsupported and may be explicitly deprecated
in a future version.
<p>If the offset + data would write outside of the segment,
this function raises <code>ValueError</code>.
</p>
<p>The bytes may contain embedded NULL bytes (<code>'\0'</code>).
</dd>
<dt>remove()</dt>
<dd>Removes (destroys) the shared memory. Note that actual destruction of the
segment only occurs when all processes have detached.
</dd>
</dl>
<h4>Attributes</h4>
<dl>
<dt>key (read-only)</dt>
<dd>The key provided in the constructor.</dd>
<dt>id (read-only)</dt>
<dd>The id assigned to this segment by the OS.</dd>
<dt>size (read-only)</dt>
<dd>The size of the segment in bytes.</dd>
<dt>address (read-only)</dt>
<dd>The address of the segment as Python long.</dd>
<dt>attached (read-only)</dt>
<dd>If True, this segment is currently attached.</dd>
<dt>last_attach_time (read-only)</dt>
<dd>The last time a process attached this segment.</dd>
<dt>last_detach_time (read-only)</dt>
<dd>The last time a process detached this segment.</dd>
<dt>last_change_time (read-only)</dt>
<dd>The last time a process changed the uid, gid or mode on this segment.</dd>
<dt>creator_pid (read-only)</dt>
<dd>The PID of the process that created this segment.</dd>
<dt>last_pid (read-only)</dt>
<dd>The PID of the most last process to attach or detach this segment.</dd>
<dt>number_attached (read-only)</dt>
<dd>The number of processes attached to this segment.</dd>
<dt>uid</dt>
<dd>The segment's user id.</dd>
<dt>gid</dt>
<dd>The segment's group id.</dd>
<dt>mode</dt>
<dd>The shared memory's permission bits.
<p>Tip: the following Python code will display
the mode in octal:<br>
<code>print int(str(my_mem.mode), 8)</code>
</p>
</dd>
<dt>cuid (read-only)</dt>
<dd>The segment creator's user id.</dd>
<dt>cgid (read-only)</dt>
<dd>The segment creator's group id.</dd>
</dl>
<h3 id="message_queue">The MessageQueue Class</h3>
<p>This is a handle to a FIFO message queue.</p>
<h4>Methods</h4>
<dl>
<dt>MessageQueue(key, [flags = 0, [mode = 0600, [max_message_size = 2048]]])</dt>
<dd>Creates a new message queue or opens an existing one.
<p><span class="param">key</span> must be <code>None</code>,
<code>IPC_PRIVATE</code> or
an integer > <code>0</code> and ≤ <code>KEY_MAX</code>. If the key
is <code>None</code>, the module chooses a random unused key.
</p>
<p>The <span class="param">flags</span> specify whether you want to create a
new queue or open an existing one.
</p>
<ul>
<li>With <span class="param">flags</span> set to the
<strong>default</strong> of <code>0</code>, the module attempts
to <strong>open an existing</strong> message queue identified by
<span class="param">key</span> and raises
a <code>ExistentialError</code> if it doesn't exist.
</li>
<li>With <span class="param">flags</span> set to <strong><code>IPC_CREAT</code></strong>, the module
<strong>opens</strong> the message queue identified
by <span class="param">key</span> <strong>or
creates</strong> a new one if no such queue exists.
</li>
<li>With <span class="param">flags</span> set to
<strong><code>IPC_CREX</code></strong> (<code>IPC_CREAT | IPC_EXCL</code>),
the module
<strong>creates</strong> a new message queue identified by
<span class="param">key</span>. If
a queue with that key already exists, the call raises
a <code>ExistentialError</code>.
</li>
</ul>
<p>The <span class="param">max_message_size</span> can be increased
from the default, but be aware of the issues discussed in
<a href="#message_queue_limits">Message Queue Limits</a>.
</p>
</dd>
<dt>send(message, [block = True, [type = 1]])</dt>
<dd>Puts a message on the queue.
<p>The <span class="param">message</span> should be a bytes object
(a.k.a. <code>str</code> in Python 2) and can contain embedded
NULLs (ASCII <code>0x00</code>). Passing a Unicode object
may work, but doing so is unsupported and may be explicitly deprecated
in a future version.
</p>
<p>The <span class="param">block</span> flag specifies whether or
not the call should wait if the message can't be sent (if, for
example, the queue is full). When <span class="param">block</span>
is <code>False</code>, the call will raise a <code>BusyError</code> if
the message can't be sent immediately.
</p>
<p>The <span class="param">type</span> is
associated with the message and is relevant when calling
<code>receive()</code>. It must be > 0.
</p>
</dd>
<dt>receive([block = True, [type = 0]])</dt>
<dd>
Receives a message from the queue, returning a tuple of
<code>(message, type)</code>. The message is a bytes object
(a.k.a. <code>str</code> in Python 2).
<p>The <span class="param">block</span> flag specifies whether or
not the call should wait if there's no messages of the
specified type to retrieve. When <span class="param">block</span>
is <code>False</code>, the call will raise a <code>BusyError</code> if
a message can't be received immediately.
</p>
<p>The <span class="param">type</span> permits some control over
which messages are retrieved.
</p>
<ul>
<li>When <span class="param">type</span> <code>== 0</code>, the call
returns the first message on the queue regardless of its
type.
</li>
<li>When <span class="param">type</span> <code>> 0</code>, the call
returns the first message of that type.
</li>
<li>When <span class="param">type</span> <code>< 0</code>, the call
returns the first message of the lowest type that is ≤ the
absolute value of <span class="param">type</span>.
</li>
</ul>
</dd>
<dt>remove()</dt>
<dd>Removes (deletes) the message queue.</dd>
</dl>
<h4>Attributes</h4>
<dl>
<dt>key (read-only)</dt>
<dd>The key provided in the constructor.</dd>
<dt>id (read-only)</dt>
<dd>The id assigned to this queue by the OS.</dd>
<dt id="queue_max_size">max_size</dt>
<dd>The maximum size of the queue in bytes. Only a process with
"appropriate privileges" can increase this value, and on some
systems even that won't work. See
<a href="#message_queue_limits">Message Queue Limits</a> for details.
</dd>
<dt>last_send_time (read-only)</dt>
<dd>The last time a message was placed on the queue.</dd>
<dt>last_receive_time (read-only)</dt>
<dd>The last time a message was received from the queue.</dd>
<dt>last_change_time (read-only)</dt>
<dd>The last time a process changed the queue's attributes.</dd>
<dt>last_send_pid (read-only)</dt>
<dd>The id of the most recent process to send a message.</dd>
<dt>last_receive_pid (read-only)</dt>
<dd>The id of the most recent process to receive a message.</dd>
<dt>current_messages (read-only)</dt>
<dd>The number of messages currently in the queue.</dd>
<dt>uid</dt>
<dd>The queue's user id.</dd>
<dt>gid</dt>
<dd>The queue's group id.</dd>
<dt>mode</dt>
<dd>The queue's permission bits.
<p>Tip: the following Python code will display
the mode in octal:<br>
<code>print int(str(my_mem.mode), 8)</code>
</p>
</dd>
<dt>cuid (read-only)</dt>
<dd>The queue creator's user id.</dd>
<dt>cgid (read-only)</dt>
<dd>The queue creator's group id.</dd>
</dl>
<h3>Usage Tips</h3>
<h4 id='samples'>Sample Code</h4>
<p>This module comes with four sets of demonstration code in the
directory <code>demos</code>.</p>
<ul>
<li><code>sem_and_shm</code> demonstrates using semaphores and
shared memory to share data between processes.
</li>
<li><code>message_queues</code> demonstrates using message queues
to share data between processes.
</li>
<li><code>buffer_protocol</code> demonstrates creating <code>bytearray</code> and
<code>memoryview</code> objects atop shared memory.
</li>
<li><code>semaphore_context_manager</code> demonstrates using
semaphore instances in a context manager (<code>with</code> statement).
</li>
</ul>
<h4 id="ftok_weakness">The Weakness of <code>ftok()</code></h4>
<p>
Most System V IPC sample code recommends <code>ftok()</code> for generating an
integer key that's more-or-less random.
It does not, however, guarantee that the key it generates is unused. If
<code>ftok()</code> gives your application a key that some other application is
already using,
your app is in trouble unless it has a reliable second mechanism for generating
a key. And if that's the case, why not just abandon <code>ftok()</code> and use the
second mechanism exclusively?
</p>
<p>This is the weakness of <code>ftok()</code> -- it isn't guaranteed to give you
what you want. The <a href="http://www.unix.com/man-page/FreeBSD/3/ftok/">BSD
man page for <code>ftok</code></a> says it is "quite possible for the routine to
return duplicate keys". The term "quite possible" isn't quantified, but suppose
it means one-tenth of one percent. Who wants to have 1-in-1000 odds of a
catastrophic failure in their program, or even 1-in-10000?
</p>
<p>This module obviates the need for <code>ftok()</code> by generating random
keys for you. If your application can't use <code>sysv_ipc</code>'s automatically
generated keys because it needs to know the key in advance, hardcoding a
random number like 123456 in your app might be no worse than using
<code>ftok()</code> and has the advantage of not hiding its limitations.
</p>
<p>This module provides <code>ftok()</code> in case you want to experiment with it.
However, to emphasize its weakness, this version of <code>ftok()</code> raises a
warning with every call unless you explicitly pass a flag to silence it.
</p>
<p>This package also provides <code>ftok_experiment.py</code> so that you can observe
how often <code>ftok()</code> generates duplicate keys on your system.
</p>
<h4 id="sem_init">Semaphore Initialization</h4>
<p>When a System V sempahore is created at the C API level, the OS is not required
to initialize the semaphore's value. (This per
<a href="http://www.opengroup.org/onlinepubs/009695399/functions/semget.html">the
SUSv3 standard for <code>semget()</code></a>.)
Some (most? all?) operating systems initialize it to zero, but this behavior
is non-standard and therefore can't be relied upon.
</p>
<p>If sempahore creation happens in an predictable, orderly fashion, this isn't a
problem. But a
race condition arises when multiple processes vie to create/open the same semaphore. The
problem lies in the fact that when an application calls <code>semget()</code> with only
the <code>IPC_CREAT</code> flag, the caller can't tell whether or not he has
created a new semaphore or opened an existing one.
<strong>This makes it
difficult to create reliable code without using <code>IPC_EXCL</code>.</strong>
W. Richard Stevens' <span class='book_title'>Unix Network Programming Volume 2</span>
calls this "a fatal flaw in the design of System V semaphores" (p 284).
</p>
<p>
For instance, imagine processes P1 and P2. They're executing the same code,
and that code intends to share a binary semaphore.
Consider the following sequence of events at the startup of P1 and P2 –
</p>
<ol>
<li>P1 calls <code>semget(IPC_CREAT)</code> to create the semaphore S.</li>
<li>P2 calls <code>semget(IPC_CREAT)</code> to open S.</li>
<li>P1 initializes the semaphore's value to 1.</li>
<li>P1 calls <code>acquire()</code>, decrementing the value to 0.</li>
<li>P2, assuming S is a newly-created semaphore that needs to be initialized,
incorrectly sets the semaphore's value to 1.</li>
<li>P2 calls <code>acquire()</code>, decrementing the value to 0. Both processes
now think they own the lock.</li>
</ol>
<p>W. Richard Stevens' solution for this race condition is to check the value of
<code>sem_otime</code> (an element in the <code>semid_ds</code> struct that's
populated on the call to <code>semctl(IPC_STAT)</code> and which is exposed to
Python by this module) which
is initialized to zero when the semaphore is created and otherwise holds
the time of the last
call to <code>semop()</code> (which is called by <code>P()</code>/<code>acquire()</code>,
<code>V()</code>/<code>release()</code>, and <code>Z()</code>).
</p>
<p>In Python, each process would run something like this:
<pre>
try:
sem = sysv_ipc.Semaphore(42, sysv_ipc.IPC_CREX)
except sysv_ipc.ExistentialError:
# One of my peers created the semaphore already
sem = sysv_ipc.Semaphore(42)
# Waiting for that peer to do the first acquire or release
while not sem.o_time:
time.sleep(.1)
else:
# Initializing sem.o_time to nonzero value
sem.release()
# Now the semaphore is safe to use.
</pre>
<h4 id="mem_init">Shared Memory Initialization</h4>
<p>With shared memory,
using the <code>IPC_CREAT</code> flag without <code>IPC_EXCL</code>
is problematic <em>unless you know the size of the segment
you're potentially opening</em>.
</p>
<p>Why? Because when creating a new segment,
many (most? all?) operating systems demand a non-zero size. However,
when opening an existing segment, zero is the only guaranteed safe value
(again, assuming one doesn't know the size of the segment in advance).
Since <code>IPC_CREAT</code>
can open or create a segment, there's no safe value for the size under
this circumstance.
</p>
<p>As a (sort of) side note, the
<a href="http://www.opengroup.org/onlinepubs/009695399/functions/shmget.html">SUSv3
specification for <code>shmget()</code></a> says only that the size of a new
segment must not be less than "the system-imposed minimum". I
gather that at one time, some systems set the minimum at zero despite the
fact that it doesn't make much sense to create a zero-length shared memory
segment. I think most modern systems do the sensible thing and insist on
a minimum length of 1.
</p>
<h4 id="message_queue_limits">Message Queue Limits</h4>
<p>Python programmers can usually remain blissfully ignorant of memory
allocation issues. Unfortunately, a combination of factors makes them
relevant when dealing with System V message queues.
</p>
<p><strong>Some implementations impose extremely stingy limits.</strong>
For instance, many BSDish systems (macOS, FreeBSD,
<a href="http://fxr.watson.org/fxr/source/sys/msg.h?v=NETBSD">NetBSD</a>, and
<a href="http://fxr.watson.org/fxr/source/sys/msg.h?v=OPENBSD">OpenBSD</a>)
limit queues to 2048 bytes. Note that that's the <em>total
queue size</em>, not the message size. Two 1k messages would fill the queue.
</p>
<p><strong>Those limits can be very difficult to change.</strong> At best,
only privileged processes can increase the limit. At worst, the limit
is a kernel parameter and requires a kernel change via a tunable or
a recompile.
</p>
<p><strong>This module can't figure out what the limits are</strong>, so
it can't cushion them or even report them to you.
On some systems the limits are expressed in header files, on others
they're available through kernel interfaces (like FreeBSD's <code>sysctl</code>).
Under macOS and to some extent OpenSolaris I can't figure out where they're
defined and what I report here is the result of experimentation and educated
guesses formed by Googling.
</p>
<p>The good news is that this module will still behave as advertised no
matter what these limits are. Nevertheless you might be surprised when a
call to <code>.send()</code> get stuck because a queue is full even though you've
only put 2048 bytes of messages in it.
</p>
<p>Here are the limits I've been able to find under my test operating
systems, ordered from best (most generous) to worst (most stingy).
<strong>This information was current as of 2009</strong> when I wrote the
message queue code. It's getting pretty stale now. I hope the situation has
improved over the 2009 numbers I describe below.
</p>
<p>Under <strong>OpenSolaris 2008.05</strong> each queue's maximum size defaults
to 64k. A privileged process (e.g. root) can change this through the
<code>max_size</code> attribute of a <code>sysv_ipc.MessageQueue</code> object.
I was able to increase it to 16M and successfully sent sixteen 1M messages to
the queue.
</p>
<p>Under <strong>Ubuntu 8.04</strong> (and perhaps other Linuxes) each
queue's maximum size defaults to 16k. As with OpenSolaris, I was able to
increase this to 16M, but only for a privileged process.
</p>
<p>Under <strong>FreeBSD 7</strong> and I think NetBSD and OpenBSD, each
queue's maximum size defaults to 2048 bytes. Furthermore, one can (as root)
set <code>max_size</code> to something larger and FreeBSD doesn't complain, but
it also ignores the change.
</p>
<p><strong>macOS</strong> is the worst of the lot. Each queue is limited
to 2048 bytes and macOS silently ignores attempts to increase this (just like
FreeBSD). To add insult to injury, there appears to be no way to increase
this limit short of recompiling the kernel.
I'm guessing at this based on the
<a href="http://www.google.com/search?q=site%3Aopensource.apple.com+%22msg.h%22">Darwin
message queue limits</a>.
</p>
<p>If you want
to search for these limits on your operating system, the key constants are
<code>MSGSEG</code>, <code>MSGSSZ</code>, <code>MSGTQL</code>, <code>MSGMNB</code>,
<code>MSGMNI</code> and <code>MSGMAX</code>. Under BSD, <code>sysctl kern.ipc</code>
should tell you what you need to know and may allow you to change these
parameters.
</p>
<h4>Nobody Likes a Mr. Messy</h4>
<p>Semaphores and especially shared memory are a little different from most Python objects
and therefore require a little more care on the part of the programmer. When a
program creates a semaphore or shared memory object, it creates something that
resides <em>outside of its own process</em>, just like a file on a hard drive. It
won't go away when your process ends unless you explicitly remove it.
</p>
<p>In short, remember to clean up after yourself.</p>
<h4>Consult Your Local <code>man</code> Pages</h4>
<p>The sysv_ipc module is just a wrapper around your system's API. If your
system's implementation has quirks, the <code>man</code> pages for <code>semget, semctl, semop
shmget, shmat, shmdt</code> and <code>shmctl</code> will probably cover them.
</p>
<h4>Interesting Tools</h4>
<p>Many systems (although not some older versions of OS X) come
with <code>ipcs</code> and <code>ipcrm</code>.
The former shows existing shared memory, semaphores and message queues on your system and
the latter allows you to remove them.
</p>
<h4>Last But Not Least</h4>
<p>For Pythonistas –</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=Xe1a1wHxTyo">A meditation on the inaccuracy
of shared memories</a>
</li>
</ul>
<h3 id="bugs">Known Bugs</h3>
<p>Bugs? My code never has bugs! There are, however, some suboptimal anomalies...</p>
<ul>
<li>This module can't report the exact min and max values for a key.
It turns out
that <a href="http://groups.google.com/group/comp.lang.c/browse_thread/thread/14a1cdbfb111f4eb">it's
really difficult to determine the maximum value that a
typedef-ed variable can hold</a>.
</li>
</ul>
<h3>Future Features/Changes</h3>
<p>These are features that may or may not be added depending on technical
difficulty, user interest and so forth.
</p>
<ul>
<li>Update this documentation with a list of platforms that support semtimedop().</li>
<li>Find a way to make <code>SEMAPHORE_VALUE_MAX</code> more accurate.</li>
</ul>
<p>I don't plan on adding support for semaphore sets.</p>
</body>
</html>
|
