Freeciv Hacker's Guide
			  
This guide is intended to be a help for developers, wanting to mess with
Freeciv program. 

Here and there, you'll see some comments marked as [...], containing more
personal thoughts on the design, why it looks like it does, and sometimes what 
went wrong. I hope developers will find that interesting too.

To read about the AI, see README.AI

===========================================================================
Basic
===========================================================================
Freeciv is a client/server civilization style of game, written in C.
The client is pretty dumb. Almost all calculations is performed on the
server.

[It wasn't like this always. Originally more code was placed in the
common/ dir, allowing the client to do some of the world updates itself. 
The end_of_turn city-refresh was for example performed both on the server 
and on the client. However things got quite complex, more and more info
was needed on the client-side(security problem). Little by little we moved 
more code to the server, and as of 1.5 the client is quite dumb -PU]

The source code has the following important directories:
common: data structures and code used by both the client and server.
server: (duh)
client: common client code
client/* (fx gui-gtk): a specific gui implementation of the client.
data: graphics, rulesets and stuff
po: translations
ai: the ai, later linked into the server.

===========================================================================
Server
===========================================================================
General:

The server main loop basically looks like:

  while(server_state==RUN_GAME_STATE) { /* looped once per turn */
    do_ai_stuff();   /* do the ai controlled players */
    sniff_packets(); /* get player requests and handle them */
    end_turn();      /* main turn update */
    game_next_year();
  }


Most time is spend in the sniff_packets() function, where a select()
call waits for packets or input on stdin(server-op commands).

===========================================================================
Data Structures
===========================================================================
For variable length list of fx units and cities freeciv uses a genlist,
which is implemented in common/genlist.c. By some macro magic type specific
macros have been defined, avoiding much trouble.
For example a tile struct (the pointer to it we call ptile) has a unit
list, ptile->units; to iterate though all the units on the tile you would
do the following:

unit_list_iterate(ptile->units, punit) {
/* In here we could do something with punit, which is a pointer to a
    unit struct */
} unit_list_iterate_end;

Note that the macro itself declares the variable punit.
Similarly there is a 

city_list_iterate(pplayer->cities, pcity) {
/* Do something with pcity, the pointer to a city struct */
} city_list_iterate_end;

There are other operations than iterating that can be performed on a list;
inserting, deleting, sorting etc. See common/speclist.h
Note that the way the *_list_iterate macro is implemented means you can use
"continue" and "break" in the usual manner.

One thing you should keep in the back of your mind: Say you are iterating
through a unit list, and then somewhere inside the iteration decide to
disband a unit. In the server you would do this by calling
wipe_unit(punit), which would then remove the unit node from all the
relevant unit lists. But by the way unit_list_iterate works, if the removed
unit was the following node unit_list_iterate will already have saved the
pointer, and use it in a moment, with a segfault as the result.
The way to avoid this is to use the function
void wipe_unit_spec_safe(struct unit *punit, struct genlist_iterator *iter,
			 int wipe_cargo)
and as the second argument write "myiter", a variable declared in the macro.
The unit list will then automatically be kept sane.

You can also define your own lists with operations like iterating; read how
in common/speclist.h.

[There was a time before the macros were developed, and raw genlists were used.
Then a unit iteration would look like
{
  struct unit_list *punit_list;
  struct genlist_iterator myiter;
  .....
  
  genlist_iterator_init(&myiter, &punit_list->list, 0);
  for(; ITERATOR_PTR(myiter); ITERATOR_NEXT(myiter)) {
    punit=(struct unit *)ITERATOR_PTR(myiter);
    /* do something with the unit */
  }
}
There are still examples of this in the code]

=========================================================================
Network and Packets
=========================================================================
The basic netcode is located in server/sernet.c and client/clinet.c.

All information passed between the server and clients, must be sent
through the network as serialized packet structures.
These are defined in common/packets.h.

For each 'foo' packet structure, there is one send and one receive function:

int		    send_packet_foo	(struct connection *pc,
					 struct packet_foo *packet);
struct packet_foo * receive_packet_foo	(struct connection *pc);

The send_packet_foo() function serializes a structure into a bytestream
and adds this to the send buffer in the connection struct.
The receive_packet_foo() function de-serializes a bytestream into a
structure and removes the bytestream from the input buffer in the
connection struct.
The connection struct is defined in common/connection.h.

Each structure field in a structure is serialized using architecture
independent functions such as put_int32() and de-serialized with
functions like get_int32().

A packet is constituted by header followed by the serialized structure
data. The header contains the following fields:

uint16	:	length		(the length of the entire packet)
uint8	:	type		(e.g. PACKET_TILE_INFO)

To demonstrate the route for a packet through the system, here's how
a unit disband is performed:

1)  A player disbands a unit.
2)  The client initializes a packet_unit_request structure, and calls the
    packet layer function send_packet_unit_request() with this structure and
    packet type: PACKET_UNIT_DISBAND.
3)  The packet layer serializes the structure, wraps it up in a packet
    containing the packetlength, type and the serialized data. Finally
    the data is send to the server.
4)  On the server the packet is read. Based on the type, the corresponding
    de-serialize function is called is called by get_packet_from_connection(). 
5)  a packet_unit_request is initialized with the bytestream.
6)  Finally the corresponding packet-handler, handle_unit_disband() function,
    is called with the newly constructed structure.
7)  The handler function checks if the disband request is legal (is the sender
    really the owner of the unit) etc.
8)  The unit is disbanded => wipe_unit() => send_remove_unit().
9)  Now an integer, containing the id of the disbanded unit is
    wrapped into a packet along with the type PACKET_REMOVE_UNIT:
    send_packet_generic_integer().
10) The packet is serialized and send across the network.
11) On the client the packet is deserialized into a packet_generic_integer
    structure.
12) The corresponding client handler function is now called 
    handle_remove_unit(), and finally the unit is disbanded.
    
Notice that both packets involved in this scenario were generic packets.
That means the packet structures involved, are used by various requests.
The packet_unit_request() is for example also used for
the packets PACKET_UNIT_BUILD_CITY and PACKET_UNIT_CHANGE_HOMECITY.

When adding a new packet type, check to see if you can reuse some of the
existing packet types. This saves you the trouble of
writing new serialize/deserialize functions.

The Xaw client uses XtAppAddInput() to tell Xt to call the callback
functions, when something happens on the client socket.
The Gtk+ client uses a similar gdk_input_add() call.

[The packet system started out as a pain and it stayed that way all the time. 
After using Java's object serialize system, this handcoded serialize
code really feels like the stoneage. That said, I fear it requires some
advanced pre-processing system etc to automate the task for C structures. -PU]

=========================================================================
Network Improvements
=========================================================================

In previous versions when a connection send buffer in the server got full
we emptied the buffer contents and continued processing. Unfortunately this
caused incomplete packets to be sent to the client, which caused crashes
in either the client or the server, since the client cannot detect this
situation. This has been fixed by closing the client connection when the
buffer is emptied.

We also had (and still have) several problems related to flow control.
Basically the problem is the server can send packets much faster than the
client can process them. This is especially true when in the end of the
turn the AIs move all their units. Unit moves in particular take a long
time for the client to process since by default smooth unit moves is on.

There are 3 ways to solve this problem:
1) We wait for the send buffers to drain before continuing processing.
2) We cut the player's connection and empty the send buffer.
3) We lose packets (this is similar to 2) but can cause an incoherent
   state in the client).

We mitigated the problem by increasing the send buffer size on the server
and making it dynamic. We also added in strategic places in the code calls
to a new flush_packets() function that makes the server stall for some time
draining the send buffers. Strategic places include whenever we send the
whole map. The maximum amount of time spent per flush_packets() call is
specified by the 'netwait' variable.

To disconnect unreachable clients we added two other features: the server
terminates a client connection if it doesn't accept writes for a period
of time (set using the 'tcptimeout' variable). It also pings the client
after a certain time elapses (set using the 'pingtimeout' variable). If
the client doesn't reply its connection is closed.

=========================================================================
Graphics
=========================================================================
Currently the graphics is stored in the xpm fileformat, and accordingly
loaded using libxpm.

If you alter the graphics, then notice that after saving the new xpm
file, you must manually mark the first color as transparent. Look at
the original xpm-files, to see how this is done. Failing to do this
results in crash(the mask-pixmaps will not be generated).

Each terrain tile is drawn in 16 versions, all the combinations with
with a green border in one of the main directions. Hills, mountains,
forests and rivers are treated in special cases.

[IMO we should have gotten rid of libxpm long ago. The graphics should
be drawn in 24bit, and then quantized runtime, if the client is displayed
on a pseudocolor display. 
The current tile system should be converted to something like civ2's.
They get away with drawing way less tiles, and it looks better. -PU]

=========================================================================
Diplomacy
=========================================================================
A few words about the diplomacy system. When a diplomacy meeting is
established, a Treaty structure is created on both of the clients and
on the server. All these structures are updated concurrently as clauses
are added and removed.

=========================================================================
Map structure
=========================================================================
The map is maintained in a pretty straightforward C array, containing
X*Y tiles. You can use the function
struct tile *map_get_tile(x, y)
to find a pointer to a specific tile.
A tile has various fields; see the struct in common/map.h

When operating on tiles you normally iterate over x and y and maybe use
map_get_tile() to get the tile struct.
When fx iterating in a square around a map position (x,y) the naive method

for (x1 = x-1; x1 <= x+1; x1++)
  for (y1 = y-1; y1 <= y+1; y1++)
    /* do something */

would sometimes, fx if (x,y) = (0,0) , give tiles like (-1,0) or (0,-1)
Because the map wraps in the x direction the first position should be
[assuming the map has the size (map.xsize,map.ysize)] (map.xsize-1, 0), while
the second tile (0,-1) is not a real time at all!
This could be solved by the following:

for (x1 = x-1; x1 <= x+1; x1++) {
  for (y1 = y-1; y1 <= y+1; y1++) {
    int abs_x = x1, abs_y = y1;
    if (!normalize_map_pos(&abs_x, &abs_y))
      continue;
    /* do something with abs_x, abs_y */
  }
}

normalize_map_pos() will adjust the values of abs_x, abs_y to fix within
the map, and if the original pos does not correspond to a tile, as (0, -1),
it will return 0.
Alternatively this could have all been done via the macro call

square_iterate(x, y, 1/*radius*/, x1, y1)  {
/* do something */
} square_iterate_end;

which automatically adjust the tile values. The defined macros should be
used whenever possible, the example of a "manual" square iterate was only
included to give people the knowledge of how things work.

Also available is the function real_map_distance(), which takes wrap and
such things into account. It does not however take terrain and roads, etc.
into account. For that you would use a move_cost_map, which is generated
via the function generate_warmap(), and calculates the distances from a
tile to all other tiles on the map. See server/gotohand.c.

Note that if all the code that operate on x and y values used macros, it
would be very simple to convert these, to allow fx a flat world or a map
structure like in civ2 for isometric view.

=========================================================================
Unknown tiles and Fog of War
=========================================================================

In the tile struct there is a field

struct tile {
  ...
  unsigned int known;
  ...
};

On the server the known fields is considered to be a bitvector, one
bit for each player, 0==tile unknown, 1==tile known.
On the client this field contains one of the following 3 values:

enum known_type {
 TILE_UNKNOWN, TILE_KNOWN_FOGGED, TILE_KNOWN
};

The values TILE_UNKNOWN, TILE_KNOWN are straightforward. TILE_FOGGED
is a tile of which the user knows the terrain (inclusive cities, roads,
etc...).

Because of how the tile-drawing system works, the client has to know the
terrain and specials of tiles adjacent to TILE_KNOWN and TILE_KNOWN_FOGGED
tiles. This is done by sending the adjacent tiles with the value
TILE_UNKNOWN. To keep track of which tiles have been sent and for fast
lookup the server has a bitvector sent similar to known. [You could do
this by seeing if the tile in question already have a known tile adjacent,
in which case the value would already would have been sent.]

Fog of war is the fact that even when you have seen a tile once you are
not sent updates unless it is inside the sight range of one of your units
or cities.
We keep track of fog of war by counting the number of units and cities
[and nifty future things like radar outposts] of each client that can
see the tile. This requires a number per player, per tile, so each tile
has a short[]. Every time a unit/city/miscellaneous can observe a tile
1 is added to its player's number at the tile, and when it can't observe
any more (killed/moved/pillaged) 1 is subtracted. In addition to the
initialization/loading of a game this array is manipulated with the
void unfog_area(struct player *pplayer, int x, int y, int len)
and
void fog_area(struct player *pplayer, int x, int y, int len)
functions. "int len" is the radius of the area that should be
fogged/unfogged, i.e. a len of 1 is a normal unit. In addition to keeping
track of fog of war, these functions also make sure to reveal TILE_UNKNOWN
tiles you get near, and send info about TILE_UNKNOWN tiles near that the
client needs for drawing. They then send the tiles to
void send_tile_info(struct player *dest, int x, int y)
which then sets the correct known_type and sends the tile to the client.

[Beware of using map_adjust_y. If you have a 80x50 map and then iterate
around the tile 40,49 you can get values like 40,50, which is not a
tile. if you map_adjust the y value it will return 40,49. If you then
think this should count as a unit seeing tile 40,49 and add a sight
point for it at 40,49 fog of war gets out of sync. This will usually
result in a wrap error on some point, as you unfog a square twice and it
wraps around to 65536. This should get caught by the server and the error
reported. I suggest using the is_real_tile function to test the tiles.
unfog_area and fog_area tests their tiles this way, or using a macro like
square_iterate that does the checking for you. Remember to recode
the is_real_tile function when we switch to isometric view.]

If you want to just show the terrain and cities of the square the
function show_area does this. The tiles remain fogged.
If you play without fog of war all the values of the seen arrays are
initialized to 1. So you are using the exact same code, you just never
get down to 0. As changes in the "fogginess" of the tiles are only sent
to the client when the value shifts between zero and non-zero, no
redundant packages are sent. You can even switch fog of war on/off
in game just by adding/subtracting 1 to all the tiles.

We only send city and terrain updates to the players who can see the
tile. So a city (or improvement) can exist in a square that is known and
fogged and not be shown on the map. Likewise, you can see a city in a
fogged square even if the city doesn't exist (it will be removed when
you see the tile again). This is done by 1) only sending info to players
who can see a tile 2) keeping track of what info has been sent so the
game can be saved. For the purpose of 2) each player has a map on the
server (consisting of player_tile's and dumb_city's) where the relevant
information is kept.

The case where a player p1 gives map info to another player p2: This
requires some extra info. Imagine a tile that neither player sees, but
which p1 have the most recent info on. In that case the age of the players'
info should be compared which is why the player tile has a last_updated
field.
This field is not kept up to date as long as the player can see the tile
and it is unfogged, but when the tile gets fogged the date is updated.

[An alternative solution would be to give each tile a list
of the units and cities that observe it. IMO this would not be any
easier than just counting, and would have no benefits. The current
solution also gives the possibility to reveal squares as you like,
say near a radar tower tile special. Very flexible.]

[The barbarians and the ai take their map info directly from the server,
so they can currently ignore fog of war, and they do so. I really think
that the ideal AI wouldn't be cheating like this.]

There is now a shared vision feature, meaning that if p1 gives shared
vision to p2, every time a function like show_area, fog_area, unfog_area
or give_tile_info_from_player_to_player is called on p1 p2 also gets the
info. Note that if p2 gives shared info to p3, p3 also gets the info.
This is controlled by p1's really_gives_vision bitvector, where the
dependencies will be kept.

If there is anything I have explained inadequately in this section you
can ask me on <thue@diku.dk>.
-Thue

=========================================================================
Client GUI- Athena 
=========================================================================
One client GUI is written using athena-widgets. A few comments on this 
could prove useful for anyone wishing to write new dialogs or improve
on the current ones.

Widgets:
--------
When you create new widgets for a dialog, like:

  players_form = XtVaCreateManagedWidget("playersform", 
				       formWidgetClass, 
				       players_dialog_shell, NULL);

then put the widget properties in the app-default file 'Freeciv', instead
of hardcoding them. For the widget created above, the following entries
in the app-default file applies:

*playersform.background:          lightblue
*playersform.resizable:           true
*playersform.top:                 chainTop
*playersform.bottom:              chainBottom
*playersform.left:                chainLeft
*playersform.right:               chainRight

Pixcomm and Canvas:
-------------------
The Pixcomm is a subclassed Command-widget, which can displays a Pixmap
instead of a string, on top of a button(command). The Pixcomm widget
should be used all places where this kind of high-level functionality
is required. 

The Canvas widget is more low-level. One have to write an expose(redraw)
event-handler for each widget. The widget generates events on resize
and mousebuttons.

[Reading any Xt documentation, will tell you how powerful widget
subclassing is. So I went trough great troubles subclassing the
command widget. It was not before long I got mails from unhappy Xaw3d
(and derives) users, that the client keeps crashing on them. Turns
out that subclassing from any widgets but Core, chains the new
widgets to libXaw. In hindsight I should just subclassed the Canvas
widget and add more highlevel functionality. -PU]

===========================================================================
Misc - The idea trashcan 
===========================================================================
[Currently all of the major entities - units, cities, players, contains
an unique id. This id is really only required when a reference to an entity
is to be serialized(saved or distributed over the net). However in the
current code, the id is also used for comparing, looking up and in general
referencing entities. This results in a lot of mess and unnecessary duplicate
functions. Often when programming, one wonders if some function needs
the id or a pointer when referring to an entity. -PU]

===========================================================================

Player-related entities in Freeciv - by rp@win.tue.nl 
+ by dwp@mso.anu.edu.au

Freeciv is confused about the meaning of 'player'.  As a participant in
Freeciv games, you'll notice that the participants are called 'players'.
At the same time, players seem to be identified with civilizations.
On the other hand, civilizations seem to be identified by 'race':
every player chooses a race at the start of the game.

In the data structures, a 'player' identifies a civilization, not a user.

----
  THE PLAN:

There are four player-related entities:

+ player
  A civilization, with a capital, cities, units, an income, etc.

+ race
  A type of civilization (except that very little actually depends on
  race, and the oddity exists that each player must be of different race)

+ user
  Identifies 'someone out there', usually a human user running a civclient
  (this is the plan; it is not yet implemented).

+ connection
  Records a client connection; like a user, but disappears when the user 
  disconnects, whereas for real users we may want to remember them between 
  connections.  See Connections section below.

Where do these entities exist?

Races aren't actually used for annything that matters; for them,
so the question isn't very interesting.

Players (more aptly named, 'civilizations'), exist in games.  Except in
the context of a running game, the entity makes no sense.  Players and
their status are part of savefiles.  A game can be saved and restarted
on a different server; the players will be the same.  A new game will
have new players.  Players exist in common/ (even games do) but a
client only has one instantiated player.

The reason to introduce users is client-side server commands.  It must
be possible to assign different levels of access to commands to different
users.  Attaching it to players is not good enough: the information must
survive the addition and removal of other players, the start or restart
of a game, reconnections by the same user even from different computers,
or transferral of the game to a different server.  However, a user
may have different levels of access in different games.

While they last, connections are sufficient identification for users.
The user entity will allow users to be identified when they reconnect.

Ideally, users would be identified with unique global ids, handed out
by a 'registry service' similar to the metaserver, but this would be
too cumbersome in practice.  So the plan is to make users persist in
a server session (even whan a game is started, or restarted when that
option is added) and make them persist across games (when a saved
game is loaded in a different server session).

Users will be created when they first connect to a server, remembered by
the running server and in savefiles.  Upon connecting, the client will
be sent a unique session id, generated when the server started, plus a
fresh user id; it will store them in a ~/.civcookie file, and send it
back when trying to reconnect.  This will allow the identity of users
to be protected.  'Protected' players will only allow the same user to
reconnect; 'unprotected' ones allow anyone to connect; server commands
and probably client options will be available to control this.

Player names will be assigned to users, not players.

The server maintains a default access level, which is used for new
users and unprotected ones.

----
  THE PRESENT IMPLEMENTATION:

Currently access levels are stored in the connection struct.  This allows 
access levels to be assigned to each individual connected player, which 
would not be the case if they were directly assigned to the player struct 
(due to the fact that the players array changes when players are added or 
removed).

But that's it.

Players are still created before the game is started, and player names
still belong to players.  Access levels exist in client and server, 
but only the server uses them.  User ids are not yet implemented;
Server ids do not exist at all.

Commands to protect/unprotect users do not yet exist; they would serve 
no useful purpose.

Access levels can set for individual users, including AI players with 
a connected observer, but only while someone is connected; they will not 
be remembered when the user disconnects.

===========================================================================
Connections
===========================================================================

The code is currently transitioning from 1 or 0 connections per player
only, to allowing multiple connections for each player (recall
'player' means a civilization, see above), where each connection may
be either an "observer" or "controller".

This discussion is mostly about connection in the server.  The client
only has one real connection: 'aconnection' - its connection to the
server, though it does use some other connection structs (currently
pplayer->conn) to store information about other connected clients
(eg, capability strings).

In the old paradigm, server code would usually send information to a
single player, or to all connected players (usually represented by
destination being a NULL player pointer).  With multiple connections
per player things become more complicated.  Sometimes information
should be sent to a single connection, or to all connections for a
single player, or to all (established) connections, etc.  To handle
this, "destinations" should now be specified as a pointer to a struct
conn_list (list of connections).  For convenience the following
commonly applicable lists are maintained:
   game.all_connections   -  all connections
   game.est_connections   -  established connections
   game.game_connections  -  connections observing and/or involved in game
   pplayer->connections   -  connections for specific player
   pconn->self            -  single connection (as list)

Connections can be classified as follows:  (first match applies)

1. (pconn->used == 0) Not a real connection (closed/unused), should
   not exist in any list of have any information sent to it.

(All following cases exist in game.all_connections.)

2. (pconn->established == 0) TCP connection has been made, but initial
   Freeciv packets have not yet been negotiated (join_game etc).
   Exists in game.all_connections only.  Should not be sent any
   information except directly as result of join_game etc packets, 
   or server shutdown, or connection close, etc.

(All following cases exist in game.est_connections.)

3. (pconn->player == NULL) Connection has been established, but is not
   yet associated with a player.  Currently this is not possible, but
   the plan is to allow this in future, so clients can connect and
   then see list of players to choose from, or just control the server
   or observe etc. Two subcases:

   3a. (pconn->observer == 0) Not observing the game.  Should receive
       information about other clients, game status etc, but not map,
       units, cities, etc.

(All following cases exist in game.game_connections.)

   3b. (pconn->observer == 1) Observing the game.  Exists in
       game.game_connections.  Should receive game information about
       map, units, cities, etc.

4. (pconn->player != NULL) Connected to specific player, either as
   "observer" or "controller".  (But observers not yet implemented.)
   Exists in game.game_connections, and in pconn->player->connections.

===========================================================================
Mini Style Guide
===========================================================================

If you want to hack Freeciv, and want your patches to be accepted, it
helps to follow some simple style rules:

- Use K&R indentation style with indentation 2 (when in doubt, use
  "indent -kr -i2").  However, do not re-indent areas of code you are
  not modifying or creating.

- Order include files consistently:  First state all system include
  files with <> in alphabetic order, then all Freeciv include files
  with "", sorted by directory (common, server, ...) and then by
  alphabetic order, with a blank line between the sections.  This helps
  to avoid adding unnecessary or duplicated include files.

- If you use a system specific feature, don't add #ifdef __CRAY__ or
  something like that.  Rather write a check for that feature for
  configure.in, and use a meaningful macro name in the source.

- Always prototype global functions in the appropriate header file.
  Local functions should always be declared as static.

- Don't use C++-style comments (i.e., // comments).

- Comments to translators are stored before the N_(), _() or Q_() marked
  string, and are preceded by "TRANS:".  These comments are copied to the
  translators file.  Use them whenever you think the translators may need
  some more information.
  Example:
    /* TRANS: Don't translate "commandname". */
    printf(_("commandname <arg> [-o <optarg>]"));

- If you send patches, use "diff -u" (or "diff -r -u").  For further
  information, see <http://www.freeciv.org/contribute.html>.  Also,
  name patch files descriptively (e.g. "fix-foo-bug-0.diff" is good,
  but "freeciv.diff" is not).

- When doing a "diff" for a patch, be sure to exclude unnecessary files
  by using the "-X" argument to "diff".  E.g.:

    % diff -ruN -Xdiff_ignore freeciv_cvs freeciv >/tmp/fix-foo-bug-0.diff

  A suggested "diff_ignore" file is included in the Freeciv distribution.

===========================================================================