1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
|
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Foolscap Schemas</title>
<link href="stylesheet-unprocessed.css" type="text/css" rel="style" />
</head>
<body>
<h1>Foolscap Schemas</h1>
<em>NOTE! This is all preliminary and is more an exercise in semiconscious
protocol design than anything else. Do not believe this document. This
sentence is lying. So there.</em>
<h2>Existing <code>Constraint</code> classes</h2>
<table border="" width="">
<tr><td>class name</td><td>shortcut</td><td></td></tr>
<tr>
<td><code>Any()</code></td>
<td></td>
<td>accept anything</td>
</tr>
<tr>
<td><code>StringConstraint(maxLength=1000)</code></td>
<td><code>str</code></td>
<td>string of up to maxLength characters (maxLength=None means
unlimited), or a VOCAB sequence of any length</td>
</tr>
<tr>
<td><code>IntegerConstraint(maxBytes=-1)</code></td>
<td><code>int</code></td>
<td>integer. maxBytes=-1 means s_int32_t, =N means LONGINT which can be
expressed in N or fewer bytes (i.e. abs(num) < 2**(8*maxBytes)),
=None means unlimited. NOTE: shortcut 'long' is like shortcut 'int' but
maxBytes=1024.</td>
</tr>
<tr>
<td><code>NumberConstraint(maxBytes=1024)</code></td>
<td><code>float</code></td>
<td>integer or float. Integers are limited by maxBytes as in
<code>IntegerConstraint</code>, floats are fixed size.</td>
</tr>
<tr>
<td><code>BooleanConstraint(value=None)</code></td>
<td><code>bool</code></td>
<td>True or False. If value=True, only accepts True. If value=False,
only accepts False. NOTE: value= is a very silly parameter.</td>
</tr>
<tr>
<td><code>InterfaceConstraint(iface)</code></td>
<td>Interface</td>
<td>TODO. UNSAFE. Accepts an instance which claims to implement the given
Interface. The shortcut is simply any Interface subclass.</td>
</tr>
<tr>
<td><code>ClassConstraint</code></td>
<td>Class</td>
<td>TODO. UNSAFE. Accepts an instance which claims to be of the given
class name. The shortcut is simply any (old-style) class object.</td>
</tr>
<tr>
<td><code>PolyConstraint(*alternatives)</code></td>
<td>(alt1, alt2)</td>
<td>Accepts any object which obeys at least one of the alternative
constraints provided. Implements a logical OR function of the given
constraints. Also known as <code>ChoiceOf</code>.</td>
</tr>
<tr>
<td><code>TupleConstraint(*elemConstraints)</code></td>
<td></td>
<td>Accepts a tuple of fixed length with elements that obey the given
constraints. Also known as <code>TupleOf</code>.</td>
</tr>
<tr>
<td><code>ListConstraint(elemConstraint, maxLength=30)</code></td>
<td></td>
<td>Accepts a list of up to maxLength items, each of which obeys the
element constraint provided. Also known as <code>ListOf</code>.</td>
</tr>
<tr>
<td><code>DictConstraint(keyConstraint, valueConstraint, maxKeys=30)</code></td>
<td></td>
<td>Accepts a dictionary of up to maxKeys items. Each key must obey
keyConstraint and each value must obey valueConstraint. Also known
as <code>DictOf</code>.</td>
</tr>
<tr>
<td><code>AttributeDictConstraint(*attrTuples, **kwargs)</code></td>
<td></td>
<td>Constrains dictionaries used to describe instance attributes, as used
by RemoteCopy. Each attrTuple is a pair of (attrname, constraint), used
to constraint individual named attributes. kwargs['attributes']
provides the same control. kwargs['ignoreUnknown'] is a boolean flag
which indicates that unknown attributes in inbound state should simply
be dropped. kwargs['acceptUnknown'] indicates that unknown attributes
should be accepted into the instance state dictionary.</td>
</tr>
<tr>
<td><code>RemoteMethodSchema(method=None, _response=None,
__options=[], **kwargs)</code></td>
<td></td>
<td>Constrains arguments and return value of a single remotely-invokable
method. If method= is provided, the <code>inspect</code> module is used
to extract constraints from the method itself (positional arguments are
not allowed, default values of keyword arguments provide constraints for
each argument, the results of running the method provide the return value
constraint). If not, most kwargs items provide constraints for method
arguments, _response provides a constraint for the return value.
__options and additional kwargs keys provide neato whiz-bang future
expansion possibilities.</td>
</tr>
<tr>
<td><code>Shared(constraint, refLimit=None)</code></td>
<td></td>
<td>TODO. Allows objects with refcounts no greater than refLimit (=None
means unlimited). Wraps another constraint, which the object must obey.
refLimit=1 rejects shared objects.</td>
</tr>
<tr>
<td><code>Optional(constraint, default)</code></td>
<td></td>
<td>TODO. Can be used to tag Copyable attributes or (maybe) method
arguments. Wraps another constraint. If an object is provided, it must
obey the constraint. If not provided, the default value will be given in
its place.</td>
</tr>
<tr>
<td><code>FailureConstraint()</code></td>
<td></td>
<td>Constrains the contents of a CopiedFailure.</td>
</tr>
</table>
<pre>
"""
RemoteReference objects should all be tagged with interfaces that they
implement, which point to representations of the method schemas. When a
remote method is called, Foolscap should look up the appropriate method and
serialize the argument list accordingly.
We plan to eliminate positional arguments, so local RemoteReferences use
their schema to convert callRemote calls with positional arguments to
all-keyword arguments before serialization.
Conversion to the appropriate version interface should be handled at the
application level. Eventually, with careful use of twisted.python.context,
we might be able to provide automated tools for helping application authors
automatically convert interface calls and isolate version-conversion code,
but that is probably pretty hard.
"""
class Attributes:
def __init__(self,*a,**k):
pass
X = Attributes(
('hello', str),
('goodbye', int),
# Allow the possibility of multiple or circular references. The default
# is to make multiple copies to avoid making the serializer do extra
# work.
('next', Shared(Narf)),
('asdf', ListOf(Narf, maxLength=30)),
('fdsa', (Narf, String(maxLength=5), int)),
('qqqq', DictOf(str, Narf, maxKeys=30)),
('larg', AttributeDict(('A', int),
('X', Number),
('Z', float))),
Optional("foo", str),
Optional("bar", str, default=None),
ignoreUnknown=True,
)
X = Attributes(
attributes={ 'hello': str, # this form doesn't allow Optional()
'goodbye': int,
},
Optional("foo", str), # but both can be used at once
ignoreUnknown=True)
class Narf(Remoteable):
# step 1
__schema__ = X
# step 2 (possibly - this loses information)
class schema:
hello = str
goodbye = int
class add:
x = Number
y = Number
__return__ = Copy(Number)
class getRemoteThingy:
fooID = Arg(WhateverID, default=None)
barID = Arg(WhateverID, default=None)
__return__ = Reference(Narf)
# step 3 - this is the only example that shows argument order, which we
# _do_ need in order to translate positional arguments to callRemote, so
# don't take the nested-classes example too seriously.
schema = """
int add (int a, int b)
"""
# Since the above schema could also be used for Formless, or possibly for
# World (for state) we can also do:
class remote_schema:
"""blah blah
"""
# You could even subclass that from the other one...
class remote_schema(schema):
dontUse = 'hello', 'goodbye'
def remote_add(self, x, y):
return x + y
def rejuvinate(self, deadPlayer):
return Reference(deadPlayer.bringToLife())
# "Remoteable" is a new concept - objects which may be method-published
# remotely _or_ copied remotely. The schema objects support both method
# / interface definitions and state definitions, so which one gets used
# can be defined by the sending side as to whether it sends a
# Copy(theRemoteable) or Reference(theRemoteable)
# (also, with methods that are explicitly published by a schema there is
# no longer technically any security need for the remote_ prefix, which,
# based on past experience can be kind of annoying if you want to
# provide the same methods locally and remotely)
# outstanding design choice - Referenceable and Copyable are subclasses
# of Remoteable, but should they restrict the possibility of sending it
# the other way or
def getPlayerInfo(self, playerID):
return CopyOf(self.players[playerID])
def getRemoteThingy(self, fooID, barID):
return ReferenceTo(self.players[selfPlayerID])
class RemoteNarf(Remoted):
__schema__ = X
# or, example of a difference between local and remote
class schema:
class getRemoteThingy:
__return__ = Reference(RemoteNarf)
class movementUpdate:
posX = int
posY = int
# No return value
__return__ = None
# Don't wait for the answer
__wait__ = False
# Feel free to send this over UDP
__reliable__ = False
# but send in order!
__ordered__ = True
# use priority queue / stream 3
__priority__ = 3
# allow full serialization of failures
__failure__ = FullFailure
# default: trivial failures, or str or int
__failure__ = ErrorMessage
# These options may imply different method names - e.g. '__wait__ =
# False' might imply that you can't use callRemote, you have to
# call 'sendRemote' instead... __reliable__ = False might be
# 'callRemoteUnreliable' (longer method name to make it less
# convenient to call by accident...)
## (and yes, donovan, we know that TypedInterface exists and we are going to
## use it. we're just screwing around with other syntaxes to see what about PB
## might be different.)
Common banana sequences:
A reference to a remote object.
(On the sending side: Referenceable or ReferenceTo(aRemoteable)
On the receiving side: RemoteReference)
('remote', reference-id, interface, version, interface, version, ...)
A call to a remote method:
('fastcall', request-id, reference-id,
method-name, 'arg-name', arg1, 'arg-name', arg2)
A call to a remote method with extra spicy metadata:
('call', request-id, reference-id, interface,
version, method-name, 'arg-name', arg1, 'arg-name', arg2)
Special hack: request-id of 0 means 'do not answer this call, do not
acknowledge', etc.
Answer to a method call:
('answer', request-id, response)
('error', request-id, response)
Decrement a reference incremented by 'remote' command:
('decref', reference-id)
Broker currently has 9 proto_ methods:
_version(vnum): accept a version number, compare to ours, reject if different
_didNotUnderstand(command): log command, maybe drop connection
_message(reqID, objID, message, answerRequired, netArgs, netKw):
_cachemessage (like _message but finds objID with self.cachedLocallyAs instead
of self.localObjectForID, used by RemoteCacheMethod and
RemoteCacheObserver)
look up objID, invoke it with .remoteMessageReceived(message, args),
send "answer"(reqID, results)
_answer(reqID, results): look up self.waitingForAnswers[reqID] and fire
callback with results
_error(reqID, failure): lookup waitingForAnswers, fire errback
_decref(objID): dec refcount of self.localObjects[objID]. Sent in
RemoteReference.__del__
_decache(objID): dec refcount of self.remotelyCachedObjects[objID]
_uncache(objID): remove obj from self.locallyCachedObjects[objID]
</pre>
<h2>stuff</h2>
<p>A RemoteReference/RemoteCopy (called a Remote for now) has a schema
attached to it. remote.callRemote(methodname, *args) does
schema.getMethodSchema(methodname) to obtain a MethodConstraint that
describes the individual method. This MethodConstraint (or MethodSchema) has
other attributes which are used by either end: what arguments are allowed
and/or expected, calling conventions (synchronous, in-order, priority, etc),
and how the return value should be constrained.</p>
<p>To use the Remote like a RemoteCopy ...</p>
<pre>
Remote:
.methods
.attributes
.getMethodSchema(methodname) -> MethodConstraint
.getAttributeSchema(attrname) -> a Constraint
XPCOM idl specifies methods and attributes (readonly, readwrite). A Remote
object which represented a distant XPCOM object would have a Schema that is
created by parsing the IDL. Its callRemote would do the appropriate
marshalling. Issue1: XPCOM lets methods have in/out/inout parameters.. these
must be detected and a wrapper generated. Issue2: what about attribute
set/get operations? Could add setRemote and getRemote for these.
---
Some of the schema questions come down to how PBRootSlicer should deal with
instances. The question is whether to treat the instance as a Referenceable
(copy-by-reference: create and transmit a reference number, which will be
turned into a RemoteReference on the other end), or as a Copyable
(copy-by-value: collect some instance state and send it as an instance).
This decision could be made by looking at what the instance inherits from:
if isinstance(obj, pb.Referenceable):
sendByReference(obj)
elif isinstance(obj, pb.Copyable):
sendByValue(obj)
else:
raise InsecureJellyError
or by what it can be adapted to:
r = IReferenceable(obj, None)
if r:
sendByReference(r)
else:
r = ICopyable(obj, None)
if r:
sendByValue(r)
else:
raise InsecureJellyError
The decision could also be influenced by the sending schema currently in
effect. Side A invokes a method on side B. A knows of a schema which states
that the 'foo' argument of this method should be a CopyableSpam, so it
requires the object be adaptable to ICopyableSpam (which is then copied by
value) tries to comply when that argument is serialized. B will enforce its
own schema. When B returns a result to A, the method-result schema on B's
side can influence how the return value is handled.
For bonus points, it may be possible to send the object as a combination of
these two. That may get pretty hard to follow, though.
</pre>
<h2>adapters and Referenceable/Copyable</h2>
<p>One possibility: rather than using a SlicerRegistry, use Adapters. The
ISliceable interface has one method: getSlicer(). Slicer.py would register
adapters for basic types (list, dict, etc) that would just return an
appropriate ListSlicer, etc. Instances which would have been pb.Copyable
subclasses in oldpb can still inherit from pb.Copyable, which now implements
ISliceable and produces a Slicer (opentype='instance') that calls
getStateToCopy() (although the subclass-__implements__ handling is now more
important than before). pb.Referenceable implements ISlicer and produces a
Slicer (opentype='reference'?) which (possibly) registers itself in the
broker and then sends the reference number (along with a schema if necessary
(and the other end wants them)).</p>
<p>Classes are also welcome to implement ISlicer themselves and produce
whatever clever (streaming?) Slicer objects they like.</p>
<p>On the receiving side, we still need a registry to provide reasonable
security. There are two registries. The first is the
RootUnslicer.openRegistry, and maps OPEN types to Unslicer factories. It is
used in doOpen().</p>
<p>The second registry should map opentype=instance class names to something
which can handle the instance contents. Should this be a replacement
Unslicer?</p>
</body> </html>
|
