1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523
|
############################
testrepository release notes
############################
NEXT (In development)
+++++++++++++++++++++
0.0.20
++++++
IMPROVEMENTS
------------
* Tests will be reliably tagged with worker-%d. The previous tagging logic
had an implicit race - the tag id was looked up via a closure which gets
the state of the pos variable at the position the overall loop has advanced
too, not the position when the closure was created.
(Robert Collins, #1316858)
0.0.19
++++++
CHANGES
-------
* Passing --subunit to all testr commands will now consistently output subunit
v2. Previously it would output v1 for stored streams and v2 for live
streams. (Robert Collins)
* ``run`` was outputting bad MIME types - test/plain, not text/plain.
(Robert Collins)
* Test filtering was failing under python3 and would only apply the
filters to the first test listed by discover. (Clark Boylan, #1317607)
* Tests that are enumerated but not executed will no longer reset the test
timing data. Enumeration was incorrectly recording a 0 timestamp for
enumerated tests. This leads to poor scheduling after an interrupted test
run. (Robert Collins, #1322763)
* Version 0.0.18 of subunit is now a hard dependency - the v2 protocol solves
key issues in concurrency and stream handling. Users that cannot use subunit
v2 can run an older testrepository, or contact upstream to work through
whatever issue is blocking them. (Robert Collins)
* When list-tests encounters an error, a much clearer response will
now be shown. (Robert Collins, #1271133)
INTERNALS
---------
* The ``get_subunit_stream`` methods now return subunit v2 streams rather
than v1 streams, preparing the way for storage of native v2 streams in
the repository. (Robert Collins)
* ``UI.output_stream`` is now tested for handling of non-utf8 bytestreams.
(Robert Collins)
0.0.18
++++++
CHANGES
-------
* ``run`` now accepts ``--isolated`` as a parameter, which will cause each
selected test to be run independently. This can be useful to both workaround
isolation bugs and detect tests that can not be run independently.
(Robert Collins)
INTERNALS
---------
* ``capture_ids`` in test_run now returns a list of captures, permitting tests
that need to test multiple runs to do so. (Robert Collins)
0.0.17
++++++
CHANGES
-------
* Restore the ability to import testrepository.repository.memory on Python 2.6.
(Robert Collins)
0.0.16
++++++
CHANGES
-------
* A new testr.conf option ``group_regex`` can be used for grouping
tests so that they get run in the same backend runner. (Matthew Treinish)
* Fix Python 3.* support for entrypoints; the initial code was Python3
incompatible. (Robert Collins, Clark Boylan, #1187192)
* Switch to using multiprocessing to determine CPU counts.
(Chris Jones, #1092276)
* The cli UI now has primitive differentiation between multiple stream types.
This is not yet exposed to the end user, but is sufficient to enable the
load command to take interactive input without it reading from the raw
subunit stream on stdin. (Robert Collins)
* The scheduler can now groups tests together permitting co-dependent tests to
always be scheduled onto the same backend. Note that this does not force
co-dependent tests to be executed, so partial test runs (e.g. --failing)
may still fail. (Matthew Treinish, Robert Collins)
* When test listing fails, testr will now report an error rather than
incorrectly trying to run zero tests. A test listing failure is detected by
the returncode of the test listing process. (Robert Collins, #1185231)
0.0.15
++++++
CHANGES
-------
* Expects subunit v2 if the local library has v2 support in the subunit
library. This should be seamless if the system under test shares the
Python libraries. If it doesn't, either arrange to use ``subunit-2to1``
or upgrade the subunit libraries for the system under test.
(Robert Collins)
* ``--full-results`` is now a no-op, use ``--subunit`` to get unfiltered
output. (Robert Collins)
0.0.14
++++++
IMPROVEMENTS
------------
* First cut at full Python 3 support. The 'works for me' release.
(Robert Collins)
0.0.13
++++++
IMPROVEMENTS
------------
* ``setup.py testr`` was not indicating test failures via it's return code.
(Monty Taylor)
0.0.12
++++++
IMPROVEMENTS
------------
* There is now a setuptools extension provided by ``testrespository`` making it
easy to invoke testr from setup.py driven workflows.
(Monty Taylor, Robert Collins)
INTERNALS
---------
* BSD license file incorrectly claimed copyright by subunit contributors.
(Monty Taylor)
* .testr.conf is now shipped in the source distribution to aid folk wanting to
validate that testrepository works correctly on their machine.
(Robert Collins)
0.0.11
++++++
IMPROVEMENTS
------------
* Fix another incompatability with Mac OS X - gdbm dbm modules don't support
get. (Robert Collins, #1094330)
0.0.10
++++++
IMPROVEMENTS
------------
* It's now possible to configure ``test_run_concurrency`` in ``.testr.conf``
to have concurrency defined by a callout. (Robert Collins)
* Testr supports running tests in arbitrary environments. See ``Remote or
isolated test environments`` in MANUAL.txt / ``testr help run``
(Robert Collins)
INTERNALS
---------
* TestCommand is now a fixture. This is used to ensure cached test instances
are disposed of - if using the object to run or list tests, you will need
to adjust your calls. (Robert Collins)
* ``TestCommand`` now offers, and ``TestListingFixture`` consumes a small
protocol for obtaining and releasing test execution instances.
(Robert Collins)
0.0.9
+++++
IMPROVEMENTS
------------
* On OSX the ``anydbm`` module by default returns an implementation that
doesn't support update(). Workaround that by falling back to a loop.
(Robert Collins, #1091500)
* ``testr --analyze-improvements`` now honours test regex filters and only
analyzes matching tests. (Robert Collins)
0.0.8
+++++
CHANGES
-------
* As a side effect of fixing bug #597060 additional arguments passed to testr
run or testr list are only passed to the underlying test runner if they are
preceeded by '--'. (Robert Collins, #597060)
* ``testr run --failing`` will no longer run any tests at all if there are
no failing tests. (Robert Collins, #904400)
IMPROVEMENTS
------------
* ``AbstractArgument`` now handles the case where additional arguments are
present that the argument type cannot parse, but enough have been parsed for
it to be valid. This allows optional arguments to be in the middle of a
grammar. (Robert Collins)
* ``cli.UI`` now passed '--' down to the argument layer for handling rather
than implicitly stripping it. (Robert Collins)
* ``DoubledashArgument`` added to allow fine grained control over the impact of
-- in command lines. (Robert Collins)
* New argument type ``ExistingPathArgument`` for use when commands want to take
the name of a file. (Robert Collins)
* ``testr`` will now show the version. (Robert Collins)
* ``testr last`` when just one test run has been run works again.
(Robert Collins)
* ``testr help command`` now shows the docstring for commands (Robert Collins)
* ``testr --help command`` or ``testr command --help`` now shows the options
for the command. (Robert Collins)
* ``testr run --analyze-isolation`` will search the current failing tests for
spurious failures caused by interactions with other tests.
(Robert Collins, #684069)
* ``testr run --until-failure`` will repeat a test run until interrupted by
ctrl-C or until a failure occurs. (Robert Collins, #680995)
* ``Repository.get_test_run`` now raises KeyError if asked for a missing or
nonexistant test run. (Robert Collins)
* Sphinx has been added to tie the documentation toghether (And it is available
on testrepository.readthedocs.org). (Robert Collins)
* ``StringArgument`` now rejects '--' - it should be handled by the use of a
``DoubledashArgument`` where one is expected. This is a bit awkward and does
not permit passing '--' down to a child process, so further work may be
needed - file a bug if this affects you. (Robert Collins)
* ``test failing --subunit`` now exits 0 unless there was a problem generating
the stream. This is consistent with the general processing model of subunit
generators. (Robert Collins)
* ``testr last`` now supports ``--subunit`` and when passed will output
the stored subunit stream. Note that the exit code is always 0 when this
is done (unless an exception occurs reading the stream) - subunit consumers
should parse the subunit to determine success/failure. (Robert Collins)
* ``testr load`` now supports passing filenames to subunit streams to load.
(Robert Collins, #620386)
* ``testr run`` will now fail a test run if the test process exits non-zero.
As a side effect of this change, if the test program closes its stdout but
does not exit, ``testr run`` will hang (waiting for the test program to
exit). (Robert Collins)
* ``testr run --load-list FILENAME`` will limit the tests run to the test ids
supplied in the list file FILENAME. This is useful for manually specifying
the tests to run, or running testr subordinate to testr (e.g. on remote
machines). (Robert Collins, partial fix for #597060)
* ``testr run foo`` now applies foo as a regex filter against the tests
found by doing a listing of the test runners tests. Likewise
``testr list-tests foo`` will apply foo as a filter against the found tests.
This makes it easy to limit the tests that will be requested for running by
the backend test process - simply pass one or more regex filters into testr
run. (Robert Collins, #597060)
* Test tags are now shown in failures. Of particular interest for folk debgging
cross-test interactions will be the worker-N tags which indicate which
backend test process executed a given test. (Robert Collins)
0.0.7
+++++
CHANGES
-------
* testrepository is now distributed via distribute rather than distutils,
allowing installation via pip into virtualenv environments. (Robert Collins)
IMPROVEMENTS
------------
* stream loading will now synthesise datestamps before demultiplexing rather
than on insertion into the repository. This fixes erroneously short times
being recorded on non timestamped streams. Additionally, moving the automatic
addition of timestamp material in front of the demuxer has removed the skew
that caused test times to be reported as longer than the stream could
indicate (by the amount of time the test runner took to start outputting
subunit). This time may be something we want to track later, but the prior
mechanism was inconsistent between the current run and reporting on prior
runs, which lead to a very confusing UI. Now it is consistent, but totally
ignores that overhead.
(Robert Collins, #1048126, #980950)
* ``testr run`` now accepts a --concurrency option, allowing command line
override of the number of workers spawned. This allows conccurency on
operating systems where autodetection is not yet implemented, or just
debugging problems with concurrent test suites. (Robert Collins, #957145)
* ''test_id_list_default'' would prevent ''test_list_option'' being used in
previous releases. For Python environments where the context to load tests
from is always needed this was not an issue (and thus not uncovered). However
given a test runner which wants either a global context or a list of specific
tests with no global context, there was no way to achieve that with this bug.
(Robert Collins, #1027042)
0.0.6
+++++
CHANGES
-------
* Now relies on subunit 0.0.8 or better and testtools 0.9.15 or better.
IMPROVEMENTS
------------
* Much better handling of unicode input from subunit streams. Specifically,
we won't crash when we can't figure something out. (Francesco Banconi,
Martin Packman, #955006)
* Parallel tests now record their worker thread number as tags in tests.
This makes identifying test ordering problems much easier.
(Benji York, #974622)
* Python2.7 changed the interface for DBM, this has been worked around.
(Robert Collins, #775214, #961103)
* Subunit 0.0.7 Changes its TestResultFilter implementation, requiring the
subclass in testrepository.filter to be come more robust.
(Robert Collins)
* A horrible thinko in the testrepository test suite came to light and has been
fixed. How the tests ever ran is a mystery. (Robert Collins, #881497)
* ''failing'', ''run'' and ''load'' now both take a ''--subunit'' option, which
displays output in raw subunit format. If ''--full-results'' is passed too,
then all subunit information is displayed. (Brad Crittenden, #949950)
* Setting ''filter_tags'' in ''.testr.conf'' will cause tests tagged with those
tags to be hidden unless the fail/error. This requires Subunit 0.0.8. If
an older version of subunit is configured, testr will return an error.
(Robert Collins, #914166)
* ``testr`` will drop into PDB from its command line UI if the environment
variable TESTR_PDB is set. (Robert Collins)
* Test partitioning now handles a corner case where multiple tests have a
reported duration of 0. Previously they could all accumulate into one
partition, now they split across partitions (the length of a partition is
used as a tie breaker if two partitions have the same duration).
(Robert Collins, #914359)
* The test 'test_outputs_results_to_stdout' was sensitive to changes in
testtools and has been made more generic. (Robert Collins)
0.0.5
+++++
CHANGES
-------
* The testrepository test suite depends on testtools 0.9.8. (Robert Collins)
* If interrupted while updating the ``failing`` list, temp files are now
cleaned up - previously a carefully timed interrupt would leave the
temporary failing file in place. (Robert Collins, #531665)
* Local implementation of MatchesException has been removed in favour of the
testtools implementation. All ``self.assertRaises`` have been migrated to
this new testing interface.
* ``setup.py`` will read the version number from PKG-INFO when it is running
without a bzr tree : this makes it easier to snapshot without doing a
release. (Jonathan Lange)
* Testrepository should be more compatible with win32 environments.
(Martin [gz])
* ``testr init-repo`` now has a ``--force-init`` option which when provided
will cause a repository to be created just-in-time. (Jonathan Lange)
* ``testr load`` and ``testr run`` now have a flag ``--partial``. When set
this will cause existing failures to be preserved. When not set, doing a
load will reset existing failures. The ``testr run`` flag ``--failing``
implicitly sets ``--partial`` (so that an interrupted incremental test run
does not incorrectly discard a failure record). The ``--partial`` flag exists
so that deleted or renamed tests do not persist forever in the database.
(Robert Collins)
* ``testr load`` now loads all input streams in parallel. This has no impact
on the CLI as yet, but permits API users to load from parallel processes.
(Robert Collins)
* ``testr list-tests`` is a new command that will list the tests for a project
when ``.testr.conf`` has been configured with a ``test_list_option``.
(Robert Collins)
* ``test run --parallel`` will parallelise by running each test command once
per CPU in the machine (detection for this only implemented on Linux so far).
An internally parallelising command will not benefit from this, but for
many projects it will be a win either from simplicity or because getting
their test runner to parallise is nontrivial. The observed duration of tests
is used to inform the partitioning algorithm - so each test runner should
complete at approximately the same time, minimising total runtime.
(Robert Collins)
* ``testr run`` no longer attempts to expand unknown variables. This permits
the use of environmen variables to control the test run. For instance,
${PYTHON:-python} in the test_command setting will run the command with
$PYTHON or python if $PYTHON is not set. (Robert Collins, #595295)
* ``testr run`` now resets the SIGPIPE handler to default - which is what
most Unix processes expect. (Robert Collins)
* ``testr run`` now uses a unique file name rather than hard coding
failing.list - while not as clear, this permits concurrent testr invocations,
or parallel testing from within testr, to execute safely. (Robert Collins)
* ``testr run`` uses an in-process load rather than reinvoking testr. This
should be faster on Windows and avoids the issue with running the wrong
testr when PYTHONPATH but not PATH is set. (Robert Collins, #613129)
* ``testr run`` will now pass -d to the ``testr load`` invocation, so that
running ``testr run -d /some/path`` will work correctly.
(Robert Collins, #529698)
* ``testr run`` will now pass ``-q`` down to ``testr load``.
(Robert Collins, #529701)
* The ``testrepository.repository.Repository`` interface now tracks test times
for use in estimating test run duration and parallel test partitioning.
(Robert Collins)
* There are the beginnings of a samba buildfarm backend for testrepository,
though it is not hooked into the UI yet, so is only useful for API users.
(Jelmer Vernooij)
* Updates to next-stream are done via a temporary file to reduce the chance
of an empty next-stream being written to disk. (Robert Collins, #531664)
* Variable expansion no longer does python \ escape expansion.
(Robert Collins, #694800)
* When next-stream is damaged testr will report that it is corrupt rather than
reporting an invalid literal. (Robert Collins, #531663)
0.0.4
+++++
IMPROVEMENTS
------------
* ``failing`` now supports ``--list`` to list the failing tests.
(Jonathan Lange)
* Repository not found errors are now clearer. (Jonathan Lange, #530010)
* The summary of a test run is now formatted as foo=NN rather than foo: NN,
which some folk find easier to read.
* The file implementation of Repository.open now performs ~ expansion.
(Jonathan Lange, #529665)
* Test failures and errors are now shown as we get them in 'load',
'failing' and 'last'. (Jonathan Lange, #613152)
0.0.3
+++++
IMPROVEMENTS
------------
* ``failing`` now correctly calls ``repository.get_failing`` and will this
track all seen failures rather than just the latest observed failures.
* New argument type ``StringArgument`` for use when a supplied argument is just
a string, rather than a typed argument.
* New subcommand 'failing' added.
* New subcommand ``run`` added which reads a .testr.conf file to figure out how
to run tests with subunit output. It then runs them and pipes into testr
load. This allows simpler integration and permits a programming interface so
that tools like Tribunal/Eclipe etc can refresh tests in a testrepository.
``run`` also passes arguments and options down to the child process. ``run``
can also supply test ids on the command, for test runners that want that.
* The command 'last' will no longer error on a new repository.
|