SPREAD: A Reliable Multicast and Group Communication Toolkit ----------------------------------------------------------- /===========================================================================\ | The Spread Group Communication Toolkit. | | Copyright (c) 1993-2004 Spread Concepts LLC | | All rights reserved. | | | | The Spread package is licensed under the Spread Open-Source License. | | You may only use this software in compliance with the License. | | A copy of the license can be found at http://www.spread.org/license | | | | This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF | | ANY KIND, either express or implied. | | | | Spread is developed at Spread Concepts LLC and the Center for Networking | | and Distributed Systems, The Johns Hopkins University. | | | | Creators: | | Yair Amir yairamir@cs.jhu.edu | | Michal Miskin-Amir michal@spreadconcepts.com | | Jonathan Stanton jstanton@gwu.edu | | | | Major Contributors: | | Cristina Nita-Rotaru crisn@cs.purdue.edu - GC security. | | Theo Schlossnagle jesus@omniti.com - Perl, skiplists, autoconf. | | Dan Schoenblum dansch@cnds.jhu.edu - Java interface. | | John Schultz jschultz@d-fusion.net - contribution to process | | group membership. | | | | Contributors: | | Ryan Caudy wyvern@cnds.jhu.edu - Group membership | | Ben Laurie ben@algroup.co.uk - FreeBSD port and warning fixes | | Daniel Rall dlr@finemaltcoding.com - Java & networking fixes, | | configuration improvements | | Marc Zyngier - Windows fixes | | | | Special thanks to the following for providing ideas and/or code: | | Ken Birman, Danny Dolev, Mike Goodrich, Ben Laurie, | | David Shaw, Robbert VanRenesse. | | | | Partial funding provided by the Defense Advanced Research Projects Agency | | (DARPA) and The National Security Agency (NSA) since 2000. The Spread | | toolkit is not necessarily endorsed by DARPA or the NSA. | | | | WWW : http://www.spread.org and http://www.cnds.jhu.edu | | Contact: spread@spread.org | | | | Version 3.17.2 Built 5/March/2004 | \===========================================================================/ March 5, 2004 Ver 3.17.2: -------------- 1) Fix daemon quit when multiple interfaces are configured as "D" daemon interfaces in the spread.conf file. Bug reported by Orit Wasserman. 2) Updated url for Java 'ant' build system. Patch by Daniel Rall. 3) Fix group_id bug that causes incorrect vs_sets. Patch by Ryan Caudy. 4) Fix spread.conf parser so it validates the machine names in segments and forces them to be less then MAX_PROC_NAME. Patch by Mikhail Terekhov. 5) Minor fix to Mac OS X compilation so library softlinks do not fail the second time make is run. 6) Alarm() changes to support priority levels on each Alarm() call. 7) Fix crash by improving packet accounting when a client connected to a singleton daemon sends a large broadcast. Reported by David Shaw. 8) Fix bus errors on Sparc & Alpha for message buffer integer assignment. Reported by Greg Shebert; tested and patched Mikhail Terekhov. 9) Verify daemon names in spread.conf are unique. If non-unique names are provided in spread.conf, configuration will be rejected and daemon will not start. Suggested by Tim Peters. 10) Zero buffer in c library before sending multicast. Reported by Panagiotis Kougiouris. 11) Send fewer lookup probe messages when only a single segment is configured. 12) Remove extra token rotations when no messages are sent. Will decrease network packet overhead. 13) Make mailbox and service in sp.h a typedef instead of a #define. Suggested and patched by Steven Dake. 14) Fix small endianness error in sp.c where the mess_type field may not be correctly converted for different endian platforms when the SP_*_recv calls return a BUFFER_TOO_SHORT or GROUPS_TOO_SHORT error. 15) Change alarm tag for security prints from SEC to SECURITY because of conflict with sys/time.h header. 16) Documentation fix to SP_receive man page to correct fields for self-leave membership messages. 17) Update of email addresses in copyright statements and headers. 18) Windows binary libraries now built as libspread and libtspread like other platforms. SOURCE INSTALL: --------------- The source install uses autoconf and make to support many Unix and unix-like platforms as well as Windows 95+. Windows is supported by a set of Visual Studio desktop and project files in the win32 subdirectory. These files build the daemon, libraries, and user programs. Any not-specifically supported platform that has reasonably close to normal sockets networking should also be easily portable. See the file PORTING for hints on porting. From the directory where you unpacked the Spread source distribution do the following: 1) Run "./configure" 2) Run "make" 3) Run "make install" as a user with rights to write to the selected installation location. (/usr/local/{bin,man,sbin,include,lib} by default) 4) Create a spread.conf file appropriate to your local configuration. For more info on how to do this look at the sample file "sample.spread.conf" or below in the binary install instructions. BINARY INSTALL: -------- We recommend that if you are experimenting with spread you create a special 'spread' directory (for example /usr/local/spread or /opt/spread) and keep all the files together there so it is easy to find stuff. That also makes it easier to run multiple architectures as the binaries for each are in their own subdirectory. This is not necessary though. You can create that directory anywhere (e.g. your own directory). If you are installing spread for active use it is probably easier to just install the correct version of the binaries, headers, man pages, and libraries into your standard locations. The directions below assume you are doing this. 1) Unpack the spread.tar.gz file into a temporary directory 2) Look at the Readme for any updates 3) Select the appropriate architecture: arch-bsdi arch-sgi arch-sunos arch-sunsol arch-pcsol arch-linux 4) Type 'make arch-????' with your architecture as the option to make 5) Now you need to copy the files, I will assume you use /usr/local/{bin,include,lib,man} Replace "ARCH" with the directory for your architecure. cp -p include/* /usr/local/include/ cp -p ARCH/libspread.a /usr/local/lib/ cp -p ARCH/libtspread.a /usr/local/lib/ cp -p ARCH/spread /usr/local/bin/ cp -p ARCH/spmonitor /usr/local/bin/ cp -p ARCH/spuser /usr/local/bin/ cp -p ARCH/sptuser /usr/local/bin/ cp -p ARCH/spflooder /usr/local/bin/ cp -p docs/*.3 /usr/local/man/man3/ cp -p docs/*.1 /usr/local/man/man1/ 6) To run spread you need to modify the 'spread/sample.spread.conf' file for your network configuration. Comments in the sample file explain this. Then cp -p sample.spread.conf /usr/local/bin/spread.conf (Notice the name change. The config file must be named that to be found. Alternatively you can run spread -c ) You can run "spread usage" to see the spread running options. To use the Java classes and examples you need to have a copy of the main 'spread' daemon running. Then the sp.class file gives you the equivelent of the libsp.a as a java class. The user.java, user.html, and user.class files give you a demonstration applet and source code. The tree.html AllNames.html and packages.html give some documentation for the java interface. For Windows (95/NT) systems use the spread.exe daemon and the libsp.lib to link with your programs. OVERVIEW: --------- Spread consists of two parts, an executable daemon which is executed on each machine which is part of the heavyweight 'machine' group, and a library which provides a programming interface for writing group multicast applications and which is linked into the application. The daemon, called "spread", should be run as a non-priveledged user (we created a 'spread' user) and only needs permissions to create a socket in /tmp and read its config file which should be in the same directory as the executable. By default the daemon binds to and runs off the non-priveledged port 4803 unless the config file indicates otherwise. Each daemon can be started independently and if it does not find any other members of its defined configuration currently active it runs as a machine group of 1 machine. When other daemons are started they will join this machine group. The machines which are running a daemon with a common config file form a 'machine group' (in contrast to a 'process group'). The daemons handle multicasting mesages between each other,providing ordering and delivery guarantees, detecting and handling failures of nodes or links, and managing all applications which are connected to each daemon. Each application utilizing the spread system needs to link with the 'libspread.a', 'libtspread.a', 'libtspread.lib' or 'libspread.lib' library and needs to use the calls defined in the 'sp.h' and 'sp_func.h' header files. Documentation for the interface is currenly incomplete, but a basic application must do at least the following: 1) Before using any other spread calls you need to 'connect' to a daemon by calling the SP_connect(...) call. This will set a mbox which is an integer representing your connection to the daemon. You use this when making other spread calls. You CAN connect to daemons multiple times from the same application (and we know of times when this is very useful). 2) If you want to receive messages from a group you need to call SP_join(). Groups are named with standard alphanumeric strings. 3) To send to a group you do NOT need to join it, just call SP_multicast() with an appropriate group name. You will not receive the messages back if you are not a member of the group. 4) To be nice to everyone else you should call SP_disconnect() when your program is finished using the spread system, if you do not your program's disconnection will eventually be detected. Some important observations when writing spread programs: 1) All the messages for a particular connection are received during an SP_receive() call. This includes messages from different groups and membership messages(like group leave and join), so you must demultiplex them yourself. This is a feature. You can opt to avoid getting membership messages all together by indicating that to SP_connect(). 2) Spread handles endian differences correctly for metadata and our headers but does NOT convert your data for you. It DOES tell you in the SP_receive call whether or not the data originated at machine with a different endianness then yours so you can easily use this to convert it yourself if necessary. 3) Each connection to a daemon has a unique 'private group' name which can be used to send a message to a particular, and only to a particular, application connection. 4) Spread supports the following ordering/delivery guarantees for messages: Unreliable (least) Reliable (will get there, no ordering) Fifo (reliable and ordered fifo by source) Causal (reliable and all mesg from any source of this level are causally ordered) Agreed (reliable and all mesg from any source of this level are totally ordered) Safe (Agreed ordering and mesg will not be delivered to application until the mesg has reached ALL receipients' daemons) For more detail, much of which IS important, see papers on Extended Virtual Sychrony, Transis and Totem. UTILITY PROGRAMS: ----------------- Spread comes with several demonstration programs and useful tools. 1) spmonitor: This program has a special interface to spread which allows it to control a machine group. It can terminate all the spread daemons in the group, change flow control parameters, and monitor the stats for one or all of the machines. It needs to know the spread.conf file used for a particular set of spread daemons. 2) spuser: This program is provided with source and uses all the functions of the spread interface. It also acts as a simple client which is useful for testing. Reading the source code to this program should show you how to use all the features of spread, and answer many questions about syntax details. 3) simple_user: This is just about the simplest spread program possible and is provided with source. It sends and receives text strings. 4) spflooder: This is used as one type of performace test. It 'floods' a spread group with data messages. It is provided with source. CONTACT: -------- If you have any questions please feel free to contact: commedia@cnds.jhu.edu yairamir@cs.jhu.edu jonathan@cs.jhu.edu