1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
|
2008-10-12 - SkyTools 2.1.8 - "Perestroika For Your Data"
= Fixes =
* deb: Make debian package accept skytools-8.3 too.
* add skylog.ini as installalble file (David Fetter)
* Londiste:
- Document the fkeys, triggers, restore-triggers commands.
- Run ANALYZE after table is copied over.
- Fix "provider add-seq --all"
* pgq.SerialConsumer (used by Londiste) - fix position check,
which got confused when sequence numbers did large jump.
* PgQ database functions:
- pgq.maint_rotate_step1() function removed tick #1
even if consumer was registered on it.
- pgq triggers: import cache invalidation from HEAD.
Now pgq.sqltriga() and pgq.logutriga() should be preferable
to pgq.logtriga().
- uninstall_pgq.sql: Correct syntax for DROP SCHEMA CASCADE.
* skytools.DBScript, used by all Python scripts:
- Don't try to stay running if MemoryError was thrown.
- Detect stale pidfile and ignore it.
- Stop hooking SIGTERM anymore. Python signal handling and
libpq polling did not interact well.
* scriptmgr:
- Don't crash on missing pidfile.
- Fix few typos on scriptmgr (Martin Pihlak)
* walmgr (Martin Pihlak):
- improved error messages on startup if no config file specified.
- walmgr.py stop now also stops syncdaemon
- pg_auth file is copied to slave as part of archiving. restored during boot.
- cleanup in remote_walmgr() to always pass config file to slave walmgr.
- master_backup() now reports exceptions.
= Features =
* londiste -s/-k now kill "copy" process with SIGTERM.
Previously the process was left running. On startup
"replay" will check if "copy" process had died before
finishing and launches one if needed. (Dimitri Fontaine)
* New skytools.signal_pidfile() function.
* Londiste: New lock_timeout parameter to limit time
locks are held on provider. Applies to operations:
provider add/remove, compare, repair.
* Londiste: on psycopg2 v2.0.6+ use new .copy_expert() API,
which does not crash on tables with large number of columns.
2008-06-03 - SkyTools 2.1.7 - "Attack of the Werebugs"
= Fixes =
* Fix pgq trigger compilation with Postgres 8.2 (Marcin Stępnicki)
* Replace `make` calls with $(MAKE) in Makefiles (Pierre-Emmanuel André)
* londiste: Fix incompatibility with Python 2.3 (Dimitri Fontaine)
* walmgr: Fix typo in config symlinking code. (pychecker)
* bulk_loader: Fix typo in temp table check. (pychecker)
* Install upgrade .sql files. Otherwise skytools_upgrade.py could
be used only from source directory.
* pgq.Consumer: Fix bug in retry/failed event handling. (pychecker)
* pgq: Fix pgq.maint_retry_events() - it could create double events when
amount of event to be moved back into main queue was more than 10.
= Features =
* Quoting of table and column names in Londiste and dispatcher scripts.
= Upgrade procedure =
* Database code under pgq and londiste schemas need to be upgraded.
That can be done on running databases with following command:
$ skytools_upgrade.py "connstr"
Or by applying 'londiste.upgrade.sql' and 'pgq.upgrade.sql' by hand.
The changes were only in functions, no table sctructure changed.
2008-04-05 - SkyTools 2.1.6 - "Quick Bugfix Release"
Now we have upgrade script, see 'man skytools_upgrade'
for info how to upgrade database code.
= Fixes =
* Include upgrade sql scripts in .tgz
* Fix 'londiste provider seqs'
* Fix 'londiste provider fkeys' in no tables added.
* Fix "londiste copy" pidfile timing race
* Fix Solaris build - avoid grep -q / define HAVE_UNSETENV
* Fix "make debXX" when several Postgres version are installed.
* New-style AC_OUTPUT usage.
* Disable manpage creation by default, --with-asciidoc to enable.
They are still included in .tgz so users should have less problems now.
* Restore iter-on-values behaviour for rows from curs.fetch*. The attempt
to make them iter-on-keys seems now misguided, as iter-on-values is already
used in existing code, and iter-on-keys returns keys in random order.
* londiste subscriber add: Dont drop triggers on table if --expect-sync is used.
* londiste copy: drop triggers and fkeys in case "replay" or "subscriber add" was skipped
* walmgr restore: better detection if old postmaster is running (Charles Duffy)
* walmgr xrestore: detect the death of parent process
* walmgr restore: create pg_tblspc - its required for 8.3 (Zoltán Böszörményi)
* walmgr restore: copy old config files over if exist (Zoltán Böszörményi)
= Features =
* Table name globbing for Londiste commands (Erik Jones)
* New manpages for scripts (Asko Oja & me)
* Upgrade script with manpage: scripts/skytools_upgrade.py
* Add .version() function to pgq_ext & londiste schemas.
* pgqadm: allow parameters without queue_ prefix in 'config' command.
* skytools Python module:
- intern() keys in db_urldecode() to decrease memory usage
- udp-logger: more fields: hostaddr, service_name
- udp-logger: dont cache udp socket, seem to hang in some cases
- DBScript.get_database() allows explicit connect string
- DBScript allows disabling config file loading
- magic_insert on dicts allows missing columns, uses NULL
- new parsing functions:
- parse_pgarray(), parse_logtriga_sql(), parse_tabbed_table(),
- parse_statements() - this one is used to split SQL install
files to separate statements.
- exists_function() checks both 'public' and 'pg_catalog'
for unqualified functions names.
- skytools.quoting: add C implementation for quote_literal, quote_copy,
quote_bytea_raw, unescape, db_urlencode, db_urldecode. This gives
20x speedup for db_urlencode and 50x for db_urldecode on some
real-life data.
- unquote_ident(), unquote_literal()
2007-11-19 - SkyTools 2.1.5 - "Enterprise-Grade Duct Tape"
= Big changes =
* Lot of new docs [Dimitri Fontaine, Asko Oja, Marko Kreen]
* Support for fkey and trigger handling in Londiste. [Erik Jones]
* Rewrite pgq.insert_event() and log triggers in C, thus SkyTools does
not depend on PL/Python anymore.
= Small changes =
* pgq+txid: convert to new API appearing in 8.3 /contrib/txid/
* Support psycopg2, preferring it to psycopg1.
* Improved bulk_loader, using temp tables exclusively.
* skytools.config: API change to allow usage without config file.
* skytools module: quote_ident(), quote_fqident()
* install .sql files under share/skytools in addition to contrib/
* pgqadm: also vacuums londiste and pgq_ext tables, if they exist
* londiste: provider add/remove --all [Hans-Juergen Schoenig]
* backend modules support 8.3
* pgq: switch pgq_lazy_fetch=NROWS for pgq.Consumer, which makes it use
cursor for event fetching, thus allowing larger batches
* txid: use bsearch() for larger snapshots
= Fixes =
* londiste fkeys: look also at dependers not only dependencies.
* pgq.consumer: make queue seeking in case of failover more strict.
* scriptmgr: dont die on user error.
* pgq: there was still fallout from reorg - 2 missing indexes.
* Due to historical reasons SerialConsumer (and thus Londiste)
accessed completed tick table directly, not via functions.
Make it use functions again.
* londiste: set client_encoding on subscriber same as on provider
* londiste: remove tbl should work also if table is already dropped [Dimitri Fontaine]
* couple walmgr fixed [Martin Pihlak]
= Upgrade procedure for database code =
* PgQ (used on Londiste provider side), table structure, plpgsql functions:
$ psql dbname -f upgrade/final/v2.1.5.pgq_core.sql
* PgQ new insert_event(), written in C:
$ psql dbname -f sql/pgq/lowlevel/pgq_lowlevel.sql
* PgQ new triggers (sqltriga, logtriga, logutriga), written in C:
$ psql dbname -f sql/pgq/triggers/pgq_triggers.sql
* Londiste (both provider and subscriber side)
$ psql dbname -f upgrade/final/v2.1.5.londiste.sql
* pgq_ext:
$ psql dbname -f upgrade/final/v2.1.5.pgq_ext.sql
2007-04-16 - SkyTools 2.1.4 - "Sweets from last Christmas"
= Fixes =
* logtriga.c was supposed to survive mismatched column string,
but the logic were buggy. Thanks go to Dmitriy V'jukov for
good analysis.
* Couple of scripts were not converted to new API. Fix it.
* Quiet a warning in textbuf.c
* Make configure and Makefiles survive on BSD's where 'make'
is not GNU make. Thanks to Hans-Juergen Schoening.
= Features =
* Published WalMgr was an old version. Sync with internal code,
where Martin has done lot of enhancements.
* Small incompat change in PGQ: add fields to pgq.get_consumer_info()
return type. Example upgrade procedure:
DROP TYPE pgq.ret_consumer_info cascade;
\i structure/types.sql
\i functions/pgq.get_consumer_info.sql
It will show some errors but thats ok. Its annoying but needed
for the tick_id cleanup in SerialConsumer/Londiste.
2007-04-10 - SkyTools 2.1.3 - "Brown Paper Bag"
Still managed to sneak in a last-minute typo.
* Fix copy-paste error in table_copy.py
* Remember to bump version in pgq.version()
2007-04-09 - SkyTools 2.1.2 - "Help screen works"
Most fallout from reorg is hopefully cleaned now.
* Dumb bug in ticker wich made it almost non-working,
except it managed to complete the regression test...
* Move --skip-truncate switch from 'copy' to 'londiste add'.
* 'londiste install' also installs plpgsql+plpythonu.
* SQL installer logs full path to file it found.
* Change pgq.get_queue_info() FOR loop variable to record
instead text that was reported to fail, although it did work here.
* Remember where the SQL files were installed.
2007-04-06 - SkyTools 2.1.1 - "Needs more thrust to fly"
SkyTools got big reorg before release, but with the hoopla
with the 3 projects at once, it did not get much testing...
There are some untested areas still, but at least pgq/londiste
are in better shape now.
* pgqadm: finish conversion...
* londiste.Syncer:
- Convert to new API
- Avoid ticking, parallel ticks are dangerous
- Bad arg in repair
* pgq:
- too aggressive check in register_consumer
- Algo desc for batch_event_sql
* Add some knobs to make regtests for londiste pass
more predictibly.
2007-03-13 - SkyTools 2.1 - "Radioactive Candy"
* Final public release.
|
