#! /bin/sh /usr/share/dpatch/dpatch-run ## 06-translated-files.dpatch by Dario Minnucci (midget) ## ## All lines beginning with `## DP:' are a description of the patch. ## DP: Translation into English for README.jis and RELEASE.jis files. ## DP: Thanks to Fumihito YOSHIDA . @DPATCH@ diff -urNad ftpmirror-1.96+dfsg~/README ftpmirror-1.96+dfsg/README --- ftpmirror-1.96+dfsg~/README 1970-01-01 01:00:00.000000000 +0100 +++ ftpmirror-1.96+dfsg/README 2007-07-16 03:20:03.689778896 +0200 @@ -0,0 +1,616 @@ + FTPMIRROR + +Introduction + + Ftpmirror is an utility to copy directory hierarchy (this is called + ``mirror'') with FTP. A transfer is triggered only when modified, it can + be efficient transferring directory trees. + + It is userful for ftp-servers maintenance, contentns sync with web server, + system backupping and so on. + + A similar perl script exists, whose name is `mirror', this implementation + is fully comparing directory lists of sources and distination, it take a + heavy memmory load. If use on a large directory trees, so eat a lot of + memory(or swap) spaces. When I use ftp-server with 16MB memory, 'mirror' + software is not usable :-p, so I wrote software from scratch, that is + frpmirror. For example, full mirroring of FreeBSD distribution + (ftp://ftp.FreeBSD.ORG/pub/FreeBSD/*) needs 8~9 MB. *(Notice by + translator) Original document wrote at old(1998). May be this results has + obsolete.* + + You need perl-5.003 or later to use ftpmirror. Please ensure that + perl-5.003 or later was installed in your environment, at first. But, perl + 5.004 has a bug of memory leak. It eat a lot of memory spaces when + mirroring directory tree. I test with perl5.005, it has small memory leak + too, it is smaller than perl 5.004, it is not critical for real usage. I + suggest you using Perl 5.005. + +How to use ftpmirror + + 1. Install ftpmirror first. The newest version of ftpmirror is available + from ftp.intec.co.jp. + + Please use these URLs. + * ftp://ftp.intec.co.jp/pub/utils/ftpmirror/ : release + * ftp://ftp.intec.co.jp/pub/utils/ftpmirror/beta/ : beta + + Since ftpmirror is a archive of instal files, you must only extract + the archive. Do one of following: + + % cd /usr/local/src + % gzip -cd < /tmp/ftpmirror-x.y.tar.gz | tar xf - + + So create a directory named ftpmirror-x.y. Change directory and + execute configure: + Notice: If you need define specific perl verions, please set perls + path for enviroment variable "PERL". + + % cd ftpmirror-x.y + % ./configure + + Or, when you need specific perl version: + + % cd ftpmirror-x.y + % env PERL=/usr/local/bin/perl5.005 ./configure + + Execute make: + + % make + + When finished make with no errors. So install: + + % su + # make install + + In finished this procedure, so installed files are below paths: + + /usr/local/bin/ftpmirror + /usr/local/bin/rotate + /usr/local/etc/ftpmirror.cf-sample + /usr/local/lib/perl5/site_perl/Fan.pm + /usr/local/lib/perl5/site_perl/Fan/... (and some dirs) + /usr/local/lib/perl5/site_perl/auto/Fan/... (and some dirs) + + * Notice: libraries install directory can changes + with perl's installed settings. + + And copying and editing a configuration files, please add difinition + what you needs. + + # cd /usr/local/etc + # cp ftpmirror.cf-sample ftpmirror.cf + # vi ftpmirror.cf + +Configuration + + You can define parameters for ftpmirror with the default configuration + file, package specific configuration file, and run-time options + (commands arguments). In addition, configuration files are set + /usr/local/etc/ftpmirror.cf by defauls, it can change by command + argument "--load-config=/hoge/local.cf" if you need. And argument + "--load-config=+/hogehoge/local.cf" is used, this is append reading + after adove defaults reading. + + Configuration parameters have follwoing types. + + Default parametes: + Configuration files settings. They are written before + Serverss and Packages parameters. + + Server parameters: + Difinition of server specific(timeouts, gateways and + so on..). They are provided at configuration files, + with 'server' parameters. + + Package parameters: + Difinition of packages specific(direcotry, transfer + modes and so on). They are provided at configuration files, + with 'package' parameters. + + Option parameters: + It provide by command args. + + Their parameters are settings by adove order(when exists). Option + parameters are most prefferd. + + How to settings parameters at configuration files: + That has two syntaxr, + + param-name = value + or + param-name += value + + = is set param-name by value. + += is adding value(strings) for param-name. + + How to settings parameters at commands args: + When using by commands args,any parameters must be prepended by "--" + in command line options. That use --(two minus) before options. For + example, following syntax is same as adove configuration settings. + + % ftpmirror --param-name=value ... + % ftpmirror --param-name+=value ... + +Parameters + + There are many acceptable parameters as follows: + * Parameters for system and behavior + * todo:(string, default: full-mirror) + Settings of command behavior. Currently ftpmirror supports + these behavior. + * scan-remote : listup remote sides files. + * scan-local : listup local sides files. + * update-master : update local sides master index. + * synch-remote : synchronize index files. + * step-mirror : mirroring as slave mode. + * full-mirror : execute basic mirroring(default). + In defaults, it set basic mirroring behavior "full-mirror". + *-regexp filters are enabled at first, any behavior works + with these filters. + * verbose: (boolean, default: no) + When set yes, transfer logs ouput verbosey. If set yes, logs + output about non-transfererd/non-modified files. + * log-mask: (comma separated list, default: none) + Settings about output logs. These difinition separates by + perl libraries. For example, log-mask = Fan=6,Fan::FTP=7 are + set, a output log include filelist of non transfererd/ non + modified, and ftp's verbosey log. + * test-mode (boolean, default: no) + dry-run. When set yes, only show filelist of transfer + targets without real transfer. + * Parameters for FTP session + * ftp-server: (string, default: none) + The server name to connect FTP session. This must be + specified for mirroring. + * ftp-gateway: (string, default: none) + Specify this parameter if you use FTP proxy gateway such as + TIS Internet FireWall ToolKit. For example, "ftp-user = + anonymous", "ftp-server = ring.etl.go.jp", "ftp-gateway = + proxy.intec.co.jp" are set, real-connection goes to + proxy.intec.co.jp, and username is anynymou@ring.etl.go.jp + (ftp-pass is through simple). You may define this with + server parameter. + * ftp-port: (service, default: ftp) ftp port definition. If + you need change a port number, please set. When nomarlly + usage, it is not necessary. + * ftp-bindaddr: (ip address, default: none) ftp local sides IP + address difinitition. This can set a specific difinition of + local ip address. When using multi-homed hosts(The hosts + have multiple ip addresses), this settings provides a clear + log files. When nomarlly usage, it is not necessary. + * ftp-user: (string, default: anonymous) + Login name for FTP session. If you want to use anonymous + FTP, please specify "anonymous" as this parameter. + * ftp-pass: (string, default: `whoami`@`hostname`) + Password for FTP session. + * ftp-group: (string, default: none) + If you want change group id after logged to the server, + specify the group name. This must be specified with + ftp-gpass. + * ftp-gpass: (string, default: none) + If you want change group id after logged to the server, + specify a password to change group. This must be specified + with ftp-group. + * ftp-passive: (boolean, default: no) + Set this parameter to yes if passive connection(PASV command + in FTP) is required. This helps you if you use ftpmirror + over a SYN/ACK style firewall. + * ftp-idle: (numeric, default: 0) + If this parameter is not zero, ftpmirror tries to set idle + timer of FTP session to this value. FTP server must support + SITE IDLE command. + * ftp-max-idle: (boolean, default: 0) + If this parameter is not zero, ftpmirror tries to set idle + timer of FTP session to maximum value. FTP server must + support SITE IDLE command. + * ftp-list-method: (LIST or STAT or STAT-A or STAT-A or + STAT-AT default: STAT) + You can specify which of STAT or LIST command should be used + to get server's directory entries. Since STAT command + returns server's directory entries over control connection, + it is little faster than LIST command. Some ftp server + software does not returns directory entry which is required + by ftpmirror, in that case you must speficy as + ftp-list-method = list. + * ftp-timeout: (numeric, default: 300) + Timeout of connection. Unit is second. + * ftp-login-retry: (numeric, default: none) + *This function is not implemented yet.* + * ftp-login-delay: (numeric, default: 60) + * ftp-stats: (boolean, default: no) + If it set yes, when mirroring is finished, total transaction + staistics is shown. It appear "control connection in/out", + and "data connection in/out" by octet units. + * http-Proxy: (string, default: none) http-proxy settings when + you need accessing ftp server with http. But, http cannot + get direcroty information, so use ls-LR file, or using slave + mode. + For example, + * package = FreeBSD + * ftp-server = ftp.tokyonet.ad.jp + * http-proxy = proxy.my.company.com + * remote-directory = /pub/FreeBSD + * local-directory = ~ftp/pub/FreeBSD + * lslR-file = /pub/FreeBSD/ls-lR.Z + Notice: HTTP is usgin per file transfer TCP connection. So + cause large session logs, and heavy latency. + * Parameters for a package + * server: (boolean, default none) + This is used to define FTP server specific parameters. After + this parameter, any parameter are treated as the server + specific parameter, until package or (next)server parameter + will be found. Especially, it's reasonable to specify + * Parameters for a mirroring + * unlink:(boolean, default: no) + When set yes, local directory trees are copied to remote + hosts. For example, + * package = webcopy + * ftp-server = www.intec.co.jp + * ftp-user = webadmin + * ftp-pass = nan-jara-hoi + * ftp-passive = yes + * put-mode = yes + * remote-directory = /usr/local/etc/httpd/htdocs + * local-directory = ~ikuo/web/htdocs + * unlink: (boolean or 'rename' default: yes) + Set this parameter "yes" or 1 if you want to + unlink a file which is removed from servers + directory. You may set this parameter as "rename", + in this case, ftpmirror renames a file with a + trailing Tilda('~'). + * unlink-limit: (size value, default: 0) + ** !!! This is not support current version. This + is obsolete.** + If this parameter is defined, ftpmirror will not + remove large files or large directories. This may + help you when server side directory hierachy is + dramatically changed. For example, if unlink-limit + is defined as 30M, any file or directory who is + larger than 30Mbytes will not be removed. You may + also specify 500K or 3G as the maximum size. When + unlink-limit is defined as 100 (e.g. no size + suffix), ftpmirror will not remote the directory + who has more than 100 entries. + * ftp-force-mtime:(boolean, default: no) + When it set yes, time comparing is using MDTM + commands.It set no, MDTM commands does not call + using only file transfering. + * ignore-mtime: (boolean, default: no) + * check-mtime: (default: 1) + By default, ftpmirror try to check modified times + of local and remote files. If this parameter is + zero, ftpmirror does NOT check modified time. In + this case, if local and remote files have same + size ftpmirror will only change local file + timestamp to match remote one. + * temp-directory: (pathname, default: /tmp) + Ftpmirror stores temporary files to the directory + specified by this parameter. Currently, ls-lR or + dirinfo files may be stored in this directory. + * lock-directory: (pathname, default: /tmp) + Ftpmirror use lock mechanism to avoid invoking + multiple ftpmirrors for a package. lock-directory + parameter means the directory to create lock file. + local-db + * create-directory : (boolean, default: yes) When + local-directory, temp-directory, lock-directory is + set, but target directories dont exists, case of + set yes to mkdir, set no to not and error + aborting. + * remote-directory: (boolean, default: none) + You must specify the directory on the FTP server + where you want to mirror as this parameter. Any + mirror is recursively performed from this + directory. + * local-directory: (pathname, default: none) + You must set this parameter to the directory in + local-side host, e.g. the directory paired with + remote-directory. Next example shows that + ftpmirror will mirror "/pub/FreeBSD" in the FTP + server to "/ftp/pub/FreeBSD" in local. + + remote-directory = /pub/FreeBSD + local-directory = /ftp/pub/FreeBSD + + If home-directory is defined (or ftp user's home + directory exists), you can specify as follows: + + remote-directory = /pub/FreeBSD + local-directory = ~/pub/FreeBSD + + * master-db-directory: (pathname, default: none) + ftpmirror can store localhosts filelist and last + mirroring datas, if use this features, set + master-db-directory and create storing directory. + For example, following esettings will create + filelist of ~ftp/pub/FreeBSD files. In this case, + "n" is index files revision. And cycle a + mirroring, ~ftp/db/FreeBSD/step.m is created too. + "m" is step files revisions. step.m is diff of + index.m => index.m+1. + + package = FreeBSD + ftp-server = ftp.FreeBSD.ORG + remote-directory = /pub/%s + local-directory = ~ftp/pub/%s + master-db-directory = ~ftp/db/%s + + Addition, when dont use mirroring, + master-db-direcoty can manage directory tree. + Example is below. + + package = original-tools + local-directory = ~ftp/pub/%s + master-db-directory = ~ftp/db/%s + + For update database only please execute: + + % ftpmirror --todo=update-master original-tools + + If remote server has a master-db-directory, it can + use "slave mode" mirroring. Please set + remote-db-directory and local-db-cirectory + parameters. + * remote-db-directory: (pathname, default: none) + * local-db-directory: (pathname, default: none) + When remote server has a remote-db-directory, it + can efficiency mirroring. This called "slave + mode". "slave mode" can be use setting + remote-db-directory and local-db-directory. If you + use it, please create directory that + local-db-directory specified. A parameters of + master-db-directory is must set remote sides + master-db-direcotry. + These parameters can %s conversion(same + local-directry parameters) And, local-db-directory + supports tilda(~) directory conversion. When + remote-db-directory and local-db-directory set, + ftpmirror's behavior is below. At first, starts + sync master-db-directory, second, start mirroring + with db information. In this case, all file + informations are provide by db, so without LIST, + STAT commands. And, database syncronization is not + fully copy, this synthethis a old index file and + step files. Full copy is only apper at two case + one is first copy, one of db files are broken. For + example, using "slave mode" with + adove(master-db-directory's example) server, + sample configuration is below. + + package = FreeBSD + ftp-server = ftp.master.server.host + remote-directory = /pub/%s + local-directory = ~ftp/pub/%s + remote-db-directory = /db/%s + local-db-directory = ~ftp/db/%s + + For advence, you can cumulative mirroring with + step files. Configuration is same adove, use: + + % ftpmirror --todo=step-mirror FreeBSD + + * lslR-file: (pathname, default: none) + You can use ls-lR(or similar) on the FTP server + rather than directory scan with STAT/LIST. Specify + server side filename of ls-lR(or similar) as this + parameter. If lslR-file has an extension of .gz or + .Z, gunzip or uncompress will be automatically + invoked. For example, if you mirror + ftp.freebsd.org:/pub/FreeBSD with ls-lR.gz, you + may specify as next: + + ftp-server = ftp.freebsd.org + remote-directory = /pub/FreeBSD + local-directory = ~/pub/FreeBSD + lslR-file = /pub/FreeBSD/ls-lR.gz + + * lslR-copy: (pathname, default: none) + When you need archive copying lslR-file on + localhost, set this parameter. If you set a + relative path, a path leading from + local-directory. And, relative path set + override-file parameters automatically. So + lslR-copy is prevent from overwrite with + mirroring. This setting is set, when lslR-file is + transfered, and chacheing to lslR-copy file, then + prevent needless transfer. The check of size/last + modified are non-transferring with lslR-file. + Addition, when you set this parameters, you can + use efficient mirroring with ls-lR file without + lslR-file parameters. + * transfer-file-regexp: (regexp, default: matches + all) + Regular expressions(perl's regular expression) to + match filenames to transfer. Ftpmirror tries to + transfer only files on the server which matches to + this expression. Any filename on the FTP server is + represented as relative path (from + remote-directory) beginning with "./". That is, if + remote-directory is configured as "/pub/FreeBSD", + the file "/pub/FreeBSD/2.1.5-RELEASE/README.TXT" + on the server is represented as + "./2.1.5-RELEASE/README.TXT". + + Some examples follow: + + transfer-file-regexp += !/~$/ + transfer-file-regexp += !/\/\.in/ + transfer-file-regexp += !/\/\.nfs/ + + With this configuration, ftpmirror does NOT + transfer files end with "~" and begin with ".in" + or ".nfs". + + (Notes: "=" means replacing a parameter value, and + "+=" means appending a parameter string) + + transfer-file-regexp += /\/bash-/ + transfer-file-regexp += /\/gcc-/ + transfer-file-regexp += ! + + With adobe configuration, ftpmirror tries to + transfer bash-* and gcc-* files, but not others. + The last line in this example means ftpmirror does + NOT transfer by default. + * transfer-directory-regexp: (default: matches all) + Regular expressions(perl's regular expression) to + match directory names to scan in a FTP server. + Note that any directory in a FTP server is + represented with trailing slash ('/'). For + example, next shows configuration to avoid + ftpmirror NOT to scan(off course, this means NOT + to transfer) directories whose names are + "lost+found", or directories ended with '~'. + + transfer-directory-regexp += !/\/lost+found\/$/ + transfer-directory-regexp += !/~\/$/ + + * override-file-regexp: (default: matches all) + Regular expressions(perl's regular expression) to + match files you can override in local directories. + In actual mirror session, ftpmirror tries to + transfer files which match transfer-file-regexp, + and also match override-file-regexp. Any file + which does not match override-file-regexp will not + be modified by ftpmirror. In the next example, + ftpmirror tries to transfer bash-* files, but not + modified any other files. This is useful when you + want to get files from multiple FTP servers to a + single local-side directory. + + transfer-file-regexp += /\/bash-/ + transfer-file-regexp += ! + override-file-regexp += /\/bash-/ + override-file-regexp += ! + + * override-directory-regexp: (default: matches all) + Regular expressions(perl's regular expression) to + match directories ftpmirror will modify. You can + transfer directories from multiple FTP servers to + a single local-side directory with this parameter. + Following example shows that you can transfer the + FreeBSD-nonUS into the FreeBSD directory. + + package = FreeBSD + ftp-server = ftp.tokyonet.ad.jp + remote-directory = /pub/FreeBSD + local-directory = ~/pub/FreeBSD + override-directory-regexp += !/\.\/FreeBSD-nonUS\// + + package = FreeBSD-nonUS + ftp-server = ftp.waseda.ac.jp + remote-directory = /pub/FreeBSD-nonUS + local-directory = ~/pub/FreeBSD/FreeBSD-nonUS + + * load-local-dirinfo: (default: 0) + Set this parameter to 1 if you want to load local + ".dirinfo" files. A ".dirinfo" file contains + directory information described below. + * store-local-dirinfo: (default: 0) + Set this parameter to 1 if you want to generate + ".dirinfo" files in local-side directories after + each mirror session. A ".dirinfo" file contains + directory information described below. + * load-remote-dirinfo: (default: 0) Set this + parameter to 1 if you want to load and use + directory information stored in the FTP server. + Using directory information, you can mirror files + without STAT/LIST/SIZE/MDTM, and ftpmirror can + check MD5 checksum of files. + * override-file-uid: (default: 0) + You can specify owner of files/directories in + local. Any files and directories which match + override-*-regexp, will be owned by this user id. + If ftpmirror is invoked by non privileged user, + this parameter is ignored. + * override-file-gid: (default: 0) + Same as override-file-uid, but this parameter is + for group id. Note that if one of + override-file-uid and override-file-gid is + defined, any file which matches override-*-regexp + (even if it is not transfered) will be modified + with chown. + * override-file-mode: (default: root -> 0444, other + -> 0644) + You can specify local-side file mode. Ftpmirror + applies chmod to any file which matches + override-*-regexp if this parameter is specified. + * override-directory-mode: (default: 0755) + Same as override-file-mode, but this parameter is + for directories. + * default-file-uid: (default: 0) + Undef override-file-uid, and set this parameter to + specify default owner of local-side files. + Ftpmirror decides file owner in order: + override-file-uid, owner in FTP server, + default-file-uid. + * default-file-gid: (default: 0) + Same as default-file-uid, but this parameter is + for group id. + * default-file-mode: (default: root -> 0444, other + -> 0644) + Undef override-file-mode and you can specify + default file mode. Ftpmirror tries to change file + mode in order: override-file-mode, file mode in + FTP server, default-file-mode. Note that since + file mode is masked with 0777, you can not set + setuid/setgid file modes. + * default-directory-mode: (default: 0755) + Same as default-file-mode, but this parameter is + for a directory mode. + Ftpmirror tries to compare files in local and remote + with these entries. For example, ftpmirror compares as: + 1. file file-type + Are sizes same? and are md5checksums same? + 2. symlink file-type + Do they point same file? + 3. directory file-type + Is the begin-time of last update of local + directory is before the end-time of last update of + remote directory? + Especially, comparing directory update times are + effective, if no file is changed in the remote + directory, because ftpmirror do nothing about the + directory (even directory scan is skipped). + + Ftpmirror generate ".dirinfo" file if + store-local-dirinfo parameter is 1. You can also + generate this file with mkdirinfo: + + # mkdirinfo /ftp/pub/utils + +Mailing list for ftpmirror + + Technical questions, suggestions, bug reports are + welcome. If you are interesting to ftpmirror, there is + a mailing-list to discuss about ftpmirror. To + subscribe, send a message + + subscribe ftpmirror + + to majordomo@ftp.intec.co.jp. + +Credits + + I would like to thank to anyone who helped me by + testing, bug-reports, suggestions, sending patches. + Especially, members in RingProject helped me by heavy + testing :-) and many suggestions. + +Miscellaneous... + + Please read COPYRIGHT for copyright notice. Any + comments, bug fixes, suggestions are welcome. Please + contact to Ikuo Nakagawa . + +-------------------------------------------------------------------------------- + + Last updated: 1997/01/27 by Ikuo Nakagawa + This is community translated file. This is not original document. + +-------------------------------------------------------------------------------- + + Translated into English on 2007-07-07 by Fumihito YOSHIDA . + +-------------------------------------------------------------------------------- diff -urNad ftpmirror-1.96+dfsg~/README.html ftpmirror-1.96+dfsg/README.html --- ftpmirror-1.96+dfsg~/README.html 1970-01-01 01:00:00.000000000 +0100 +++ ftpmirror-1.96+dfsg/README.html 2007-07-16 03:16:41.750271019 +0200 @@ -0,0 +1,732 @@ + +ftpmirror - mirroring directory hierarchy with FTP + +

FTPMIRROR

+ +

Introduction

+ +

+Ftpmirror is an utility to copy directory hierarchy +(this is called ``mirror'') with FTP. + +A transfer is triggered only when modified, it can be efficient +transferring directory trees. +

+It is userful for ftp-servers maintenance, contentns sync with +web server, system backupping and so on. +

+ +A similar perl script exists, whose name is `mirror', this +implementation is fully comparing directory lists of sources +and distination, it take a heavy memmory load. If use on a +large directory trees, so eat a lot of memory(or swap) spaces. + +When I use ftp-server with 16MB memory, 'mirror' software is +not usable :-p, so I wrote software from scratch, that is +frpmirror. + +For example, full mirroring of FreeBSD distribution +(ftp://ftp.FreeBSD.ORG/pub/FreeBSD/*) needs 8~9 MB. +*(Notice by translator) Original document wrote at + old(1998). May be this results has obsolete.* +

+You need perl-5.003 or later to use ftpmirror. +Please ensure that perl-5.003 or later was installed +in your environment, at first. + +But, perl 5.004 has a bug of memory leak. It eat a +lot of memory spaces when mirroring directory tree. +I test with perl5.005, it has small memory leak too, +it is smaller than perl 5.004, it is not critical for +real usage. I suggest you using Perl 5.005. + +

How to use ftpmirror

+ +
    +

    +

  1. Install ftpmirror first. + The newest version of ftpmirror is available from + ftp.intec.co.jp. + +

    + Please use these URLs. +

      +
    • ftp://ftp.intec.co.jp/pub/utils/ftpmirror/ : release +
    • ftp://ftp.intec.co.jp/pub/utils/ftpmirror/beta/ : beta +
    +

    + Since ftpmirror is a archive of instal files, + you must only extract the archive. + Do one of following: + 

    +  % cd /usr/local/src
+% gzip -cd < /tmp/ftpmirror-x.y.tar.gz | tar xf -
+
+
    + +So create a directory named ftpmirror-x.y. Change directory +and execute configure:
    +Notice: If you need define specific perl verions, please set +perls path for enviroment variable "PERL". +
    +% cd ftpmirror-x.y
+% ./configure 
+
    + +Or, when you need specific perl version: +
    +% cd ftpmirror-x.y
+% env PERL=/usr/local/bin/perl5.005 ./configure 
+
    + +Execute make: + +
    +% make
+
    + +When finished make with no errors. So install: +
    +% su
+# make install
+
    + +In finished this procedure, so installed files are below paths: +
    +/usr/local/bin/ftpmirror
+/usr/local/bin/rotate
+/usr/local/etc/ftpmirror.cf-sample
+/usr/local/lib/perl5/site_perl/Fan.pm
+/usr/local/lib/perl5/site_perl/Fan/...   (and some dirs)
+/usr/local/lib/perl5/site_perl/auto/Fan/...  (and some dirs) 
+
+* Notice: libraries install directory can changes 
+    with perl's installed settings.
+
    + +And copying and editing a configuration files, please add +difinition what you needs. +
    +# cd /usr/local/etc
+# cp ftpmirror.cf-sample ftpmirror.cf
+# vi ftpmirror.cf
+
+
    + +

    Configuration

    + +You can define parameters for ftpmirror with +the default configuration file, +package specific configuration file, +and run-time options (commands arguments). + +In addition, configuration files are set /usr/local/etc/ftpmirror.cf +by defauls, it can change by command argument +"--load-config=/hoge/local.cf" if you need. +And argument "--load-config=+/hogehoge/local.cf" is used, +this is append reading after adove defaults reading. + +

    +Configuration parameters have follwoing types. + +

    +Default parametes:
+Configuration files settings. They are written before
+Serverss and Packages parameters. 
+
+Server parameters:
+Difinition of server specific(timeouts, gateways and
+so on..). They are provided at configuration files, 
+with 'server' parameters.
+
+Package parameters:
+Difinition of packages specific(direcotry, transfer 
+modes and so on). They are provided at configuration files, 
+with 'package' parameters.
+
+Option parameters:
+It provide by command args.
+
    + +Their parameters are settings by adove order(when exists). +Option parameters are most prefferd. +

    + +How to settings parameters at configuration files:
    + +That has two syntaxr, + +

    +param-name = value
+or 
+param-name += value 
+
+=  is set param-name by value.
++= is adding value(strings) for param-name.
+
    +

    +How to settings parameters at commands args:
    + +When using by commands args,any parameters must be +prepended by "--" in command line options. +That use --(two minus) before options. + +For example, following syntax is same as adove +configuration settings. + +

    +% ftpmirror --param-name=value ...
+% ftpmirror --param-name+=value ...
+
    + +

    Parameters

    + +

    +There are many acceptable parameters as follows: +

      +
    • Parameters for system and behavior +
        + +
      • todo:(string, default: full-mirror)
        +Settings of command behavior. Currently ftpmirror +supports these behavior. +
          +
        • scan-remote : listup remote sides files. + +
        • scan-local : listup local sides files. +
        • update-master : update local sides master index. +
        • synch-remote : synchronize index files. +
        • step-mirror : mirroring as slave mode. +
        • full-mirror : execute basic mirroring(default). +
        +In defaults, it set basic mirroring behavior "full-mirror". +*-regexp filters are enabled at first, any behavior +works with these filters. + +
      • verbose: (boolean, default: no)
        +When set yes, transfer logs ouput verbosey. If set yes, +logs output about non-transfererd/non-modified files. + +
      • log-mask: (comma separated list, +default: none)
        +Settings about output logs. These difinition separates by +perl libraries. For example, + log-mask = Fan=6,Fan::FTP=7 +are set, a output log include filelist of non transfererd/ +non modified, and ftp's verbosey log. + +
      • test-mode (boolean, default: no)
        + +dry-run. When set yes, only show filelist of transfer targets +without real transfer. +
      + +
    • Parameters for FTP session +
        + +
      • ftp-server: (string, default: none)
        +The server name to connect FTP session. This must be specified +for mirroring. + +
      • ftp-gateway: (string, default: none)
        +Specify this parameter if you use FTP proxy gateway such as + +TIS Internet FireWall ToolKit. +For example, "ftp-user = anonymous", "ftp-server = ring.etl.go.jp", +"ftp-gateway = proxy.intec.co.jp" are set, real-connection +goes to proxy.intec.co.jp, and username is anynymou@ring.etl.go.jp +(ftp-pass is through simple). +You may define this with server parameter. + + +
      • ftp-port: (service, default: ftp) +ftp port definition. If you need change a port number, please set. +When nomarlly usage, it is not necessary. + +
      • ftp-bindaddr: (ip address, default: none) +ftp local sides IP address difinitition. This can set a +specific difinition of local ip address. When using multi-homed +hosts(The hosts have multiple ip addresses), this settings provides +a clear log files. +When nomarlly usage, it is not necessary. + +
      • ftp-user: (string, default: anonymous)
        +Login name for FTP session. If you want to use anonymous FTP, +please specify "anonymous" as this parameter. + +
      • ftp-pass: (string, default: `whoami`@`hostname`)
        +Password for FTP session. + +
      • ftp-group: (string, default: none)
        +If you want change group id after logged to the server, +specify the group name. +This must be specified with ftp-gpass. + + +
      • ftp-gpass: (string, default: none)
        +If you want change group id after logged to the server, +specify a password to change group. +This must be specified with ftp-group. + +
      • ftp-passive: (boolean, default: no)
        +Set this parameter to yes if passive connection(PASV command in FTP) +is required. This helps you if you use ftpmirror over +a SYN/ACK style firewall. + +
      • ftp-idle: (numeric, default: 0)
        +If this parameter is not zero, ftpmirror tries to set idle timer +of FTP session to this value. +FTP server must support SITE IDLE command. + + +
      • ftp-max-idle: (boolean, default: 0)
        +If this parameter is not zero, ftpmirror tries to set idle timer +of FTP session to maximum value. +FTP server must support SITE IDLE command. + +
      • ftp-list-method: (LIST or STAT or STAT-A or STAT-A or STAT-AT default: STAT)
        +You can specify which of STAT or LIST command should be used to get +server's directory entries. Since STAT command returns server's +directory entries over control connection, it is little faster than +LIST command. Some ftp server software does not returns directory +entry which is required by ftpmirror, in that case you must +speficy as ftp-list-method = list. + +
      • ftp-timeout: (numeric, default: 300)
        + +Timeout of connection. Unit is second. + +
      • ftp-login-retry: (numeric, default: none)
        +*This function is not implemented yet.* + +
      • ftp-login-delay: (numeric, default: 60)
        + +
      • ftp-stats: (boolean, default: no)
        +If it set yes, when mirroring is finished, total transaction +staistics is shown. It appear "control connection in/out", and +"data connection in/out" by octet units. + +
      • http-Proxy: (string, default: none) +http-proxy settings when you need accessing ftp server with +http. But, http cannot get direcroty information, so use +ls-LR file, or using slave mode.
        +For example, + +
          +
        • package = FreeBSD +
        • ftp-server = ftp.tokyonet.ad.jp +
        • http-proxy = proxy.my.company.com +
        • remote-directory = /pub/FreeBSD +
        • local-directory = ~ftp/pub/FreeBSD +
        • lslR-file = /pub/FreeBSD/ls-lR.Z +
        +Notice: HTTP is usgin per file transfer TCP connection. +So cause large session logs, and heavy latency. +
      + +
    • Parameters for a package +
        +
      • server: (boolean, default none)
        +This is used to define FTP server specific parameters. +After this parameter, any parameter are treated as the server +specific parameter, +until package or (next)server parameter will be found. +Especially, it's reasonable to specify +
      + +
    • Parameters for a mirroring +
        +
      • unlink:(boolean, default: no)
        +When set yes, local directory trees are copied to remote hosts. +For example, + +
          +
        • package = webcopy +
        • ftp-server = www.intec.co.jp +
        • ftp-user = webadmin +
        • ftp-pass = nan-jara-hoi +
        • ftp-passive = yes +
        • put-mode = yes +
        • remote-directory = /usr/local/etc/httpd/htdocs +
        • local-directory = ~ikuo/web/htdocs + +
            + +
          • unlink: (boolean or 'rename' default: yes)
            +Set this parameter "yes" or 1 if you want to unlink a file +which is removed from servers directory. +You may set this parameter as "rename", +in this case, ftpmirror renames a file with a trailing Tilda('~'). + +
          • unlink-limit: (size value, default: 0)
            +** !!! This is not support current version. This is obsolete.**
            +If this parameter is defined, ftpmirror will not remove +large files or large directories. +This may help you when server side directory hierachy is +dramatically changed. +For example, if unlink-limit is defined as 30M, +any file or directory who is larger than 30Mbytes will not +be removed. You may also specify 500K or 3G as the maximum size. +When unlink-limit is defined as 100 (e.g. no size suffix), + +ftpmirror will not remote the directory who has more than +100 entries. + +
          • ftp-force-mtime:(boolean, default: no)
            + When it set yes, time comparing is using MDTM commands.It set no, + MDTM commands does not call using only file transfering. + +
          • ignore-mtime: (boolean, default: no)
            +
          • check-mtime: (default: 1)
            +By default, ftpmirror try to check modified times of local and +remote files. If this parameter is zero, ftpmirror does NOT +check modified time. +In this case, if local and remote files have same size ftpmirror + +will only change local file timestamp to match remote one. + +
          • temp-directory: (pathname, default: /tmp)
            +Ftpmirror stores temporary files to the directory specified by +this parameter. Currently, ls-lR or dirinfo files may be stored +in this directory. + +
          • lock-directory: (pathname, default: /tmp)
            +Ftpmirror use lock mechanism +to avoid invoking multiple ftpmirrors for a package. +lock-directory parameter means the directory to create lock file. +local-db +
          • create-directory : (boolean, default: yes) +When local-directory, temp-directory, lock-directory is set, but +target directories dont exists, case of set yes to mkdir, set no +to not and error aborting. + + +
          • remote-directory: (boolean, default: none)
            +You must specify the directory on the FTP server where you want to mirror +as this parameter. +Any mirror is recursively performed from this directory. + +
          • local-directory: (pathname, default: none)
            +You must set this parameter to the directory in local-side host, +e.g. the directory paired with +remote-directory. +Next example shows that ftpmirror will mirror +"/pub/FreeBSD" in the FTP server +to "/ftp/pub/FreeBSD" in local. +
            +remote-directory = /pub/FreeBSD
+local-directory = /ftp/pub/FreeBSD
            +If home-directory is defined (or ftp user's home directory exists), +you can specify as follows: + +
            +remote-directory = /pub/FreeBSD
+local-directory = ~/pub/FreeBSD
            + +
          • master-db-directory: (pathname, default: none)
            +ftpmirror can store localhosts filelist and last mirroring datas, +if use this features, set master-db-directory and create storing +directory. + +For example, following esettings will create filelist of ~ftp/pub/FreeBSD +files. In this case, "n" is index files revision. And cycle a mirroring, +~ftp/db/FreeBSD/step.m is created too. "m" is step files revisions. +step.m is diff of index.m => index.m+1. +
            +	package = FreeBSD
+	 ftp-server = ftp.FreeBSD.ORG
+	 remote-directory = /pub/%s
+	 local-directory = ~ftp/pub/%s
+	 master-db-directory = ~ftp/db/%s
+
            +Addition, when dont use mirroring, master-db-direcoty can manage +directory tree. Example is below. + +
            +	package = original-tools
+	 local-directory = ~ftp/pub/%s
+	 master-db-directory = ~ftp/db/%s
+
            + +For update database only please execute: +
            +	% ftpmirror --todo=update-master original-tools
+
            + +If remote server has a master-db-directory, it can use +"slave mode" mirroring. Please set remote-db-directory +and local-db-cirectory parameters. + +
          • remote-db-directory: (pathname, default: none) +
          • local-db-directory: (pathname, default: none)
            +When remote server has a remote-db-directory, it can efficiency mirroring. +This called "slave mode". "slave mode" can be use setting +remote-db-directory and local-db-directory. If you use it, +please create directory that local-db-directory specified. +A parameters of master-db-directory is must set remote sides +master-db-direcotry.
            + +These parameters can %s conversion(same local-directry parameters) +And, local-db-directory supports tilda(~) directory conversion. + +When remote-db-directory and local-db-directory set, ftpmirror's +behavior is below. At first, starts sync master-db-directory, +second, start mirroring with db information. In this case, +all file informations are provide by db, so without LIST, STAT +commands. + +And, database syncronization is not fully copy, this synthethis +a old index file and step files. Full copy is only apper at +two case one is first copy, one of db files are broken. + +For example, using "slave mode" with adove(master-db-directory's +example) server, sample configuration is below. + +
            +
+	package = FreeBSD
+ 	 ftp-server = ftp.master.server.host
+	 remote-directory = /pub/%s
+	 local-directory = ~ftp/pub/%s
+	 remote-db-directory = /db/%s
+	 local-db-directory = ~ftp/db/%s
+
            + +For advence, you can cumulative mirroring with step files. +Configuration is same adove, use: +
            +	 % ftpmirror --todo=step-mirror FreeBSD
+
            + +
          • lslR-file: (pathname, default: none)
            +You can use ls-lR(or similar) on the FTP server rather than +directory scan with STAT/LIST. Specify server side filename +of ls-lR(or similar) as this parameter. +If lslR-file has an extension of .gz or .Z, +gunzip or uncompress will be automatically invoked. +For example, if you mirror ftp.freebsd.org:/pub/FreeBSD with ls-lR.gz, +you may specify as next: + +
            +ftp-server = ftp.freebsd.org
+remote-directory = /pub/FreeBSD
+local-directory = ~/pub/FreeBSD
+lslR-file = /pub/FreeBSD/ls-lR.gz
            + +
          • lslR-copy: (pathname, default: none)
            +When you need archive copying lslR-file on localhost, set this +parameter. If you set a relative path, a path leading from +local-directory. And, relative path set override-file parameters +automatically. So lslR-copy is prevent from overwrite with mirroring. +This setting is set, when lslR-file is transfered, and chacheing to +lslR-copy file, then prevent needless transfer. +The check of size/last modified are non-transferring with lslR-file. +
            +Addition, when you set this parameters, you can use efficient mirroring +with ls-lR file without lslR-file parameters. + +
          • transfer-file-regexp: (regexp, default: matches all)
            + Regular expressions(perl's regular expression) to match filenames + to transfer. + Ftpmirror tries to transfer only files on the server + which matches to this expression. + Any filename on the FTP server is represented as relative path + (from remote-directory) beginning with "./". + That is, if remote-directory is configured as "/pub/FreeBSD", + the file "/pub/FreeBSD/2.1.5-RELEASE/README.TXT" on the server + is represented as "./2.1.5-RELEASE/README.TXT".
            + +

            + Some examples follow: + 

              transfer-file-regexp += !/~$/
+  transfer-file-regexp += !/\/\.in/
+  transfer-file-regexp += !/\/\.nfs/
            + With this configuration, ftpmirror does NOT transfer files + end with "~" and begin with ".in" or ".nfs". +

            + + (Notes: "=" means replacing a parameter value, + and "+=" means appending a parameter string) + 

              transfer-file-regexp += /\/bash-/
+  transfer-file-regexp += /\/gcc-/
+  transfer-file-regexp += !
            + + With adobe configuration, ftpmirror tries to transfer bash-* and + gcc-* files, but not others. + The last line in this example means + ftpmirror does NOT transfer by default. +
          • transfer-directory-regexp: (default: matches all)
            + Regular expressions(perl's regular expression) to match directory + names to scan in a FTP server. + Note that any directory in a FTP server is represented with trailing + slash ('/'). + For example, next shows configuration to avoid ftpmirror + NOT to scan(off course, this means NOT to transfer) directories + whose names are "lost+found", or directories ended with '~'. + 
              transfer-directory-regexp += !/\/lost+found\/$/
+  transfer-directory-regexp += !/~\/$/
            + +
          • override-file-regexp: (default: matches all)
            + Regular expressions(perl's regular expression) to match files + you can override in local directories. + In actual mirror session, + ftpmirror tries to transfer files which match + transfer-file-regexp, + and also match override-file-regexp. + Any file which does not match override-file-regexp + will not be modified by ftpmirror. + In the next example, ftpmirror tries to transfer bash-* files, + but not modified any other files. This is useful when you want + to get files from multiple FTP servers + to a single local-side directory. + 
              transfer-file-regexp += /\/bash-/
+  transfer-file-regexp += !
+  override-file-regexp += /\/bash-/
+  override-file-regexp += !
            + +
          • override-directory-regexp: (default: matches all)
            + Regular expressions(perl's regular expression) to match directories + ftpmirror will modify. + You can transfer directories from multiple FTP + servers to a single local-side directory with this parameter. + Following example shows that you can transfer the FreeBSD-nonUS + into the FreeBSD directory. + 
              package = FreeBSD
+   ftp-server = ftp.tokyonet.ad.jp
+   remote-directory = /pub/FreeBSD
+   local-directory = ~/pub/FreeBSD
+   override-directory-regexp += !/\.\/FreeBSD-nonUS\//
+
+  package = FreeBSD-nonUS
+   ftp-server = ftp.waseda.ac.jp
+   remote-directory = /pub/FreeBSD-nonUS
+   local-directory = ~/pub/FreeBSD/FreeBSD-nonUS
            + +
          • load-local-dirinfo: (default: 0)
            + Set this parameter to 1 if you want to load local ".dirinfo" files. + A ".dirinfo" file contains + directory information described below. + + +
          • store-local-dirinfo: (default: 0)
            + Set this parameter to 1 if you want to generate ".dirinfo" files + in local-side directories after each mirror session. + A ".dirinfo" file contains + directory information described below. + +
          • load-remote-dirinfo: (default: 0) +Set this parameter to 1 if you want to load and use directory information stored in the FTP server. Using directory information, you can mirror files without STAT/LIST/SIZE/MDTM, and ftpmirror can check MD5 checksum of files. + +
          • override-file-uid: (default: 0)
            + You can specify owner of files/directories in local. Any files + and directories which match override-*-regexp, + will be owned by this user id. + If ftpmirror is invoked by non privileged user, + this parameter is ignored. + +
          • override-file-gid: (default: 0)
            + + Same as override-file-uid, + but this parameter is for group id. + Note that if one of override-file-uid + and override-file-gid is defined, + any file which matches override-*-regexp (even if it is + not transfered) will be modified with chown. + +
          • override-file-mode: (default: root -> 0444, other -> 0644)
            + + You can specify local-side file mode. + Ftpmirror applies chmod to any file which matches + override-*-regexp if this parameter is specified. + +
          • override-directory-mode: (default: 0755)
            + Same as override-file-mode, + but this parameter is for directories. + +
          • default-file-uid: (default: 0)
            + + Undef override-file-uid, + and set this parameter to specify default owner of local-side files. + Ftpmirror decides file owner in order: + override-file-uid, owner in FTP server, + default-file-uid. + +
          • default-file-gid: (default: 0)
            + + Same as default-file-uid, but this parameter is for group id. + +
          • default-file-mode: (default: root -> 0444, other -> 0644)
            + + Undef override-file-mode + and you can specify default file mode. + Ftpmirror tries to change file mode + in order: override-file-mode, + file mode in FTP server, default-file-mode. + Note that since file mode is masked with 0777, you can not set + setuid/setgid file modes. + +
          • default-directory-mode: (default: 0755)
            + + Same as default-file-mode, + but this parameter is for a directory mode. +
          + + Ftpmirror tries to compare files in local + and remote with these entries. + For example, ftpmirror compares as: +
            +
          1. file file-type
            + Are sizes same? and are md5checksums same? +
          2. symlink file-type
            + + Do they point same file? +
          3. directory file-type
            + Is the begin-time of last update of local directory is before + the end-time of last update of remote directory? +
          + Especially, comparing directory update times are effective, + if no file is changed in the remote directory, because ftpmirror + do nothing about the directory (even directory scan is skipped). +

          + Ftpmirror generate ".dirinfo" file + if store-local-dirinfo parameter + is 1. You can also generate this file with mkdirinfo: + 

          +
+ # mkdirinfo /ftp/pub/utils
          + +

          Mailing list for ftpmirror

          + + Technical questions, suggestions, bug reports are welcome. + If you are interesting to ftpmirror, there is a mailing-list + to discuss about ftpmirror. + To subscribe, send a message + 
          + subscribe ftpmirror
          + + to + majordomo@ftp.intec.co.jp. + +

          Credits

          + + I would like to thank to anyone who helped me by testing, + bug-reports, suggestions, sending patches. + Especially, members in + RingProject helped me by + heavy testing :-) and many suggestions. + + +

          Miscellaneous...

          + + Please read COPYRIGHT for copyright notice. + Any comments, bug fixes, suggestions are welcome. + Please contact to + Ikuo Nakagawa + <ikuo@intec.co.jp>. + +
          +Last updated: 1997/01/27 by Ikuo Nakagawa
          +This is community translated file. This is not original document. +Translated on 2007-07-07 by Fumihito YOSHIDA (hito@kugutsu.org). Thank you. + + diff -urNad ftpmirror-1.96+dfsg~/RELEASE ftpmirror-1.96+dfsg/RELEASE --- ftpmirror-1.96+dfsg~/RELEASE 1970-01-01 01:00:00.000000000 +0100 +++ ftpmirror-1.96+dfsg/RELEASE 2007-07-16 03:20:27.995163981 +0200 @@ -0,0 +1,159 @@ +---- This is translated document for your convinience. --- + +Guidance of ftpmirror-1.96 + +* Introduction + +One year or more has passed for the first time the +development of ftpmirror-2.0 branch. +It is reworking of ftpmirror-1.2 branch from scrach. + +Im busy, and doesn't make progress. However, I want +to fix the bug, new version of 1.96 is release now. + +Some bugs still remains, I will fix them. + +* About Changes + +Changelog from 1.2k is below (But it is not included always, +because new version have a lot of changes). + +o Mirroring function supports PUT mode. + (When it is allowed, execute chmod at remote side) + +o Most parts were coding as module. + +o MD5 support is currently wrote by XS, and direct + execution is support now. + +o Mirroring function supports http proxy. +(restriction: this func need dirinfo or ls-lR, and so on..) + +o Using AutoSplit. It can efficiency loading. + +o New feature (work in progress) : "INDEX mode". + It will could usefull mirroring including diff management. + +o Directory name conversion support for easy configuration. + For example, ~ftp/pub and ~ftp/pub/%s and so on. + +o resource checking with getrusage command. + +o In FTP mode, login retry and reconnect when timed out + is supports now. + +Restriction: +1.2k's functions of "unlink-limit" and "follow-symlink-regexp" +and some functions are not implemented yet. +These will come back at future release. + +* Bug fix information + +Newly version(1.96)'s fixlist is below. I will have corrected them. + +o Fix: problem of STAT commands, it with wu-ftpd and so on. + +o Fix: mirroring fail when include '_' in owner and group strings. + +* About version strings + +Currently version is 1.96. I will release incremetal versions +(1.97, 1.98, 1.99...) in beta-phase, product release is using 2.0. +But, if heavy bug remains, and/or in features enhancement, +they have a possibility of branch 1.99 --> 1.991 ;) + +* How to install + +There is configurable from this version. + +Please type: + +% ./configure + +% make + +% su + +# make install + +so installing /usr/local/bin/ftpmirror and perl library. + +perl library is installed for site-addon, for example +"/usr/local/lib/perl5/site_perl". + +appendix modules are: + +Fan.pm + +Fan/Attrib.pm + +Fan/Cool.pm + +Fan/DIR.pm + +Fan/FTP.pm + +Fan/Farm.pm + +Fan/HTTP.pm + +Fan/Loader.pm + +Fan/MD5.pm + +Fan/Param.pm + +Fan/Scan.pm + +Fan/TCP.pm + +Fan/Usage.pm + +and their derivation objects. + + +In default, samples configuration file + +/usr/local/etc/ftpmirror.cf-sample + +is dropped at installation. Please copy this file, +and edit for your configuration. + +# cd /usr/local/etc + +# cp ftpmirror.cf-sample ftpmirror.cf + +# vi ftpmirror.cf + +This sample is include basic settings, please wrote your +archive places and so on. Configuration syntax is wrote +at README files. + +* Copyrights + +Omitted in translation, please see "COPYRIGHT" files. + +* Special thanks + +Thanks for RingProject members, and ftpmirror mailing +list members. You are provide very important informations +and helpful adivices. Thank you sincerely. + +# Notice: Development will start the most severe part :-p +# please cooperate further. > ALL + +* epilogue + +If you have comments, ideas, bug reports, + please send message to ikuo@jp.freebsd.org . + +Dec 27, 1999, Ikuo Nakagawa + +---- This is translated document for your convinience. --- + +-------------------------------------------------------------------------------- + + Translated into English on 2007-07-07 by Fumihito YOSHIDA . + +-------------------------------------------------------------------------------- +