I'm sure the Gold App Dev Request form for all of this is just delayed
in Campus Mail...

This email note is an epic.  Ed in the role of the protagonist, Doug in
the role of the antagonist (of course).  Save this note, it's the only
documentation you will get for a while...


Here is a summary of the changes that I made to spong.  The new version
of Spong was put into production this morning.  I came down and walked
the day shift through the new features.  I'll leave a note for the 2nd
and 3rd shift.  The changes that I made should not significantly affect
how they use it.

But anyway - first the single, lone, isolated, only, request that I
didn't do.

> --------------------------------------------------
> (*) Change "spong" name to "pong" or something doesn't get confused with
> "sponge",
> --------------------------------------------------

Changing the name would just cause additional confusion, and requires
more effort then what it is worth IMNSHO.  Feel free to make fun of me for
being attached to the name if it makes you feel better...


> --------------------------------------------------
> (*) Change "procs" name to "jobs" to lessen confusion about what is down,
> --------------------------------------------------

Done, the real fix will require the following however.  

  1. You will need to push out a new copy of spong-client to all of the
     machines and restart-spong client.

  2. Once you have that done everywhere, you will need to clean out the
     spong database on dim, so all records of the old procs service is
     removed. 

But since, I know how much you will whine if I try to make you do that,
I also went ahead and added a few little hacks in the spong-scripts that
check for "procs" in various places and replace the string with "jobs"
when it finds it.  So you can do steps 1 and 2 above at your leisure...


> --------------------------------------------------
> (*) Improve flexibility of setting down times for various services,
> --------------------------------------------------

Basically, I just fixed the way it was supposed to work before,  In the
spong.hosts file you can configure a host like the following:

 'strobe.weeg.uiowa.edu' =>  	{ services => 'ftp pop3 http',
				  contact  => 'unix-staff',
				  group    => 'unix-all',
				  down     => [ '*:12:00-14:00',
					        '1:4:00-6:00' ] };

This definition says that strobe is down everyday from noon until 2pm,
and is also down on Monday morning from 4-6am.  You can include as many
different downtimes as you want.  The first part of the field is the day
of the week (0=sunday), or * indicating that it happens everyday.  The
second part is the time range in 24 hour format.

This wasn't working in the version of spong that you are using, so I
have fixed it.  Basically during the times that you define, all of the
services on the machine are "acknowledged" - well not actually, what
really happens is that their services just have a "blue" state, so that
they are not reported as problems to the operators.  I could make them
have a unique color if you want, but I think 5 colors is plenty...

Now, this isn't a full blow cron type thing, but when working I think it
will handle most of the cases.  It doesn't handle the silicon case where
just ftp is down at certain times, but that is addressed later...


> --------------------------------------------------
> (*) Include option (default?) that just shows "down" services,
> --------------------------------------------------

I believe that the main reason that you wanted to do this is just
because it was so damn slow.  Hopefully I've fixed that problem so this
is less of an issue.

If you still want to see just the hosts with problems, basically you
just need to jump to the URL that loads the left frame.  That URL is:

	http://dim.weeg.uiowa.edu/cgi-bin/www-spong/problems/all


> --------------------------------------------------
> (*) Provide clarity in messages so operators page with exact phrasing,
> --------------------------------------------------

I've done the following.  In both the Problem List frame where it has
the name of the group/person to contact, and on the Host and Service
page where there is a link that says "Contact Staff".  Those links are
now "smart", meaning when you click on them - it will take you to the
Operator Paging Page, with the name of the person who is on call for
that machine already selected and a message indicating the problem
already filled out in the message box.  So in most cases the operator
just has to press the send button.

This means that there is an additional file you have to maintain with
the paging cgi script.  There is a file called "hosts" in the directory
with the rest of the paging stuff.  That file contains a list of hosts,
and the group or individual that is responsible for that machine.

The default message contains the host name, and if there is a single
problem it lists the service name and summary information (which is
sometimes redundant).  If there are multiple problems then it will just
list the services that are red.

Again, the operators will still have the ability to change the message
if they want.


> --------------------------------------------------
> Improve loading performance.
> --------------------------------------------------

Well...  Here is what I've done.  I believe it's significantly faster
from what some of you have said (but then again I thought the old one
was just fine).

Got rid of the little dot images, and replaced them with little squares
that are just tables with their background set to different colors.
This seems to get around the problem that Netscape has displaying all
the images.  So once the page gets to your browser, it should display
much faster.

If some day, Netscape gets fixed and you are feeling nostalgic for the
dots, then you can turn them back on in the spong.conf file.

    $WWW_USE_IMAGES = 1;

Also, the exact color of the various boxes can be adjusted as well.  I
have played with a number of variations, and I do think what I have is
pretty good, but if you want to change it, then change the following in
the spong.conf file

    $WWW_COLOR{"red"}    = "#cc0000";
    $WWW_COLOR{"yellow"} = "#ffff00";
    $WWW_COLOR{"green"}  = "#339900";
    $WWW_COLOR{"purple"} = "#990099";
    $WWW_COLOR{"blue"}   = "#0000ff";

> --------------------------------------------------
> Allow others (e.g., Help Desk) to access spong. This will require
> access control (e.g., passwords) as well as configurable "refresh rate"
> option so that only operators and us are allowed auto refresh.
> --------------------------------------------------

Ok, I'll meet you half way on this one.  The access control should come
through the standard web server mechanism.  There are 3 parts to spong:

   www-spong      - which just allows you to view the spong database
   www-spong-ack  - which allows you to update the database
   page.cgi       - which allows the operators to page us

I would suggest that you basically open up the www-spong program to the
people that need it, and tighten down the ack and page programs so that
you don't have the help desk acking problems or paging you.  

The way that you want to set up this access is up to you (host base vs
user names and passwords, etc...)  You are quite familiar with the issues
of maintaining the password files for this type of thing, etc...

Ok, now that I've passed the buck on that, here is what I'll do.  I've
included two new variables in the spong.conf file.  They are:

      @WWW_REFRESH_ALLOW = ( '.*' );
      @WWW_REFRESH_DENY  = ( 'edhill', '128.255.51\.\d+', 'traitor.*' );

These lists would contain regular expressions that are checked.  ALLOW
is checked first, followed by deny.  The following pieces of information
are checked against these regexps - REMOTE_USER (the username if you
protect www-spong using user authentication), REMOTE_HOST (the hostname
of the person connecting), and REMOTE_ADDR (the IP address of the person
connecting).

If no regular expression is matched from either list, then the
auto-refresh is not included in the output.

----------------------------------------------------------------------------

And over the course of trying to do these things, instead of hearing the
kind words of encouragement that I'm used to over in this building, I
instead heard the following additional whining and tried to address it.
These comments have not been through Rex's tone modifier like the
requests above.

> --------------------------------------------------
> I don't know why we use this program anyway, the spong-server seems to
> crash for no good reason at random times.  We try to hack in things
> like getting spong to restart spong child when they die, but if think
> was just written well to begin with, we wouldn't have to do that crap.
> --------------------------------------------------

I have found a repeatable case where I could get spong to crash.  If a
client (either updating or querying spong) would time out (because spong
was too slow) - that client would just shut down the connection.  

Well, the spong-server was still trying to write to a PIPE that was now
closed.  spong-server was told this with a SIG_PIPE, but I wasn't
checking for that signal, and if you don't check for that signal and you
get it, the program seems to exit.

So I'm now checking for it, and it no longer seems to go away.  I've
left your child restarting code just in case (but as you probably know -
that won't help the case where the parent dies - either could happen
with the case that I fixed.)

There could of course be other problems then this, but there is now one
less thing that will make spong die.


> --------------------------------------------------
> The history feature is useless, I can't believe it's actually slower
> then the spong summary.  It's so slow, it never even comes back before
> netscape times it out.  It's so slow, I went off and wrote my own
> little spong-history command because your a bad programmer and blah
> blah blah...
> --------------------------------------------------

Ok, I wrote a script called spong-cleanup.  It should be run every
night.  It does the following:

   * Cleans out any history older then 7 day.  It moves the old history
     for each host into the /local/www/docs/spong/archive directory.
     If you don't think you would ever want to get at that history, then
     you can just change the script so that it is deleted.

   * Removes any acknowledgments that are no longer valid.

   * Removes any services that don't seem to be reported any more (if
     you stop monitoring something on a machine - the old entry will
     still hang around and show up as purple).

Cleaning out the old history brought the load time down from 90 seconds
to about 4 seconds.  That combined with the colored squares instead of
gif images should make retrieving the history via the web usable again.


On another note, I actually have a command line program called "spong"
which I have now included in the spong depot package.  This is a
client/server program that basically reports all the same things that
the www-spong program does (including history).  So you could install
"spong" on your desktop and run:

	spong --history

or if you just want to see the Unix machines:

	spong --history unix-all

It also allows you to view the summary table, problem hosts, individual
hosts, etc...  Type:

	spong --help

So if you wanted to do some type of automatic monitoring of the
automatic monitoring system from your desktop, you can use the command
line program to do it.


> --------------------------------------------------
> The acknowledgment mechanism sucks, we want web-ARS.  Hey, tie spong
> into the directory server. Hey, why you're at it, build us a directory
> server.
> --------------------------------------------------

Ok, you now have the ability to delete existing acknowledgments via the
web interface.  When you click on a host, in the Acknowledgment section
next to the descriptions there is a new link which allows you to delete
an Ack.

If you click on just the generic "Ack" menu item, it will take you to
the same screen as before (allowing you to add a new acknowledgment),
but at the top of the screen is listed all of the pending Acks, so
you can click on them, and either delete them.

Updating an acknowledgment can be done through this interface now.  1)
first you delete the old one, and 2) then you add a new one that was
similar but different then the old one - Walla, updates 8-) (I ran out
of time...)

I'll assume that the interface to all of this is self-explanatory.

Ok, now to the lame solution to the "ftp is down on silicon" problem.
As with the spong command line program, there is also a spong-ack
client/server command line program.  In your script on silicon that
disables ftp, you could add the following command

    spong-ack silicon.weeg.uiowa.edu ftp '+18h' 'its all Taos baby'

You can of course do this in any place that you have a script which is
going to down a service for a period of time.

You can also delete an acknowledgment through this command line
interface, but it is a little more convoluted.  You would run the
following command

    spong-ack --delete silicon.weeg.uiowa.edu-ftp-898185233

The little funky looking thing after the delete is the ack id.  No, you
probably don't normally know what the ID of your acknowledgment is, but
you can find it with the following

    spong --brief --acks

yeah, deleting is not the cleanest via the command line, but you get
what you pay for (and no there is not "updating" via the command line,
because in reality there is no updating period - it's all an illusion).


> --------------------------------------------------
> Well what else is broken with this piece of crap package...
> --------------------------------------------------

Here is one I had not heard of, but noticed this week.  If you had a
host which had a problem with say it's disk, and you acknowledged that
service, but then the host had another problem with say jobs. The host
would not show up in the "Problems list" on the left side of the frame,
so I'm sure the operators would probably not call you about the second
problem.

I fixed that bug...


> --------------------------------------------------
> Well, I bet you screwed up all the patches that we have had to make to
> duct tape this software together since you last worked on it a long
> long time ago.
> --------------------------------------------------

Well, I incorporated all the changes that I could tell including the
following:
	
    * The paging space addition to spong-client
    * Dan's connection to the LDAP server for machine info
    * Dave's various fixes for things like PID file removal, etc...

Basically I incorporated everything that I could tell by the History
file or diffing the various programs.


> --------------------------------------------------
> Why would you get rid of frames you dufus, it was the only thing you
> did right.
> --------------------------------------------------

Jeeze, I was just trying to make things faster.  Relax, it's back to the
frames version...


> --------------------------------------------------
> Why do you need root access on dim?  How about we just give you access
> to the cp and ls commands. All other commands that you want to execute
> will need to be placed in a file called "duh" and submitted for our
> approval.
> --------------------------------------------------

My initial response:  Grrr...

My response after accidently chowning /tmp:  Yes sir, whatever is best
sir...


If there are any problems with any of this, I'm sure you will let me
know.