1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286
|
Current
Fixed: GITHUB-1017 Reporter.log is ignored in skipped test listener (Scott Kirkpatrick)
New: Minimal code changes to allow TestNG to work for OpenJDK tests, which should be run with only the java.base module present.
Fixed: GITHUB-1047 IClassListener didn't work (Julien Herr)
Fixed: GITHUB-1049 IClassListener was called many time (Julien Herr)
Fixed: GITHUB-1045 TestNG swallows exceptions silently if @ClassRule is used (Gili Tzabari & Julien Herr)
Fixed: GITHUB-506 TestNG cannot find JUnit method names from Spock (@blackduck-joe & Julien Herr)
Fixed: GITHUB-1009 Iterator<Object[]> DataProvider: indices not working (Mark Fulton & Julien Herr)
Fixed: GITHUB-1030 Parameterized test class crashes when data provider returns empty array (Christoffer Sawicki & Julien Herr)
New: TestNG displays a warning when tests are using an empty data provider
6.9.11:
2016/03/26
Fixed: GITHUB-923 Refactored data provider's parameter values passing to a varargs or non-varargs method with @NoInjection handling (Nitin Verma)
New: GITHUB-933: Deprecate XmlTest#getTestParameters (Julien Herr)
Fixed: GITHUB-911: TestListener#onTestStart should be invoked if a suite configuration method fails (Harmin Parra Rueda & Julien Herr)
Fixed: GITHUB-793: Test suite with tests using dependency and priority has wrong behavior (Martin Hereu & Julien Herr)
Fixed: GITHUB-922: ITestResult doesn't contain name if a class has @Test (@dr29bart & Julien Herr)
Fixed: GITHUB-419: parallel mode was ignored with command line (@khospodarysko & Julien Herr)
New: GITHUB-932: Deprecate true/false parallel values. none is the new default value. (Julien Herr)
Fixed: GITHUB-960: testng-failed.xml gets generated even when there are no failures. (Krishnan Mahadevan)
Fixed: GITHUB-895: Changing status of test by setStatus of ITestResult (Raj Srivastava & Julien Herr)
Fixed: GITHUB-969: testng-failed.xml does not carry over the parameters of methods from origin suite xml(Ning Zhang)
6.9.10:
2015/12/15
Fixed: GITHUB-841: testName from @Test is now used and available from ITestResult#getName() and ITestResult#getTestName() (Julien Herr)
New: GITHUB-776: Add BeforeClass/AfterClass like on ITestListener (@vguna & Julien Herr)
Fixed: GITHUB-872: Enable end-users of TestNG to alter XmlSuite and XmlTest (Krishnan Mahadevan)
New: GITHUB-900: Support @Listeners in annotation transformer (Julien Herr)
New: GITHUB-898: Activate XML validation when possible (Julien Herr)
Fixed: GITHUB-889: XmlSuite in nested directories results in FIleNotFoundException (Virender Singh)
Fixed: GITHUB-811: Timeout is not working with parallel=tests (@michael-yxf & Julien Herr)
Fixed: GITHUB-839: Missing encoding meta data for report file (@banbq & Julien Herr)
Fixed: GITHUB-876: NullPointerException creating tests with parameters by a factory (@vixgeo & Julien Herr)
New: GITHUB-886: Add some checks on factory methods (Julien Herr)
New: GITHUB-874 / GITHUB-875 / GITHUB-882 / GITHUB-850 : Some code cleanup (Testo Nakada)
Fixed: GITHUB-866 / GITHUB-869 : Some attributes were not cloned when XmlSuite#clone was used (Virender Singh)
Fixed: GITHUB-842: Add TestResult#getTestName() support for @Test(testName) (Julien Herr)
Fixed: GITHUB-908: Fix Double.NaN assertion (Julien Herr)
6.9.9:
2015/10/27
Fixed: GITHUB-829: Allowing suites to have duplicate names. You can now configure the same suite-file to run multiple times. (Eduardo Born)
Fixed: GITHUB-834: nested suites not supported by 'testnames' (Tibor Digana & Julien Herr)
6.9.8:
2015/10/12
Replace 6.9.7 that was build with Java8 by error.
6.9.7:
No official release
Fixed: GITHUB-798: Set suitethreadpoolsize for Maven Surefire (Jan Dundáček)
Fixed: GITHUB-171: ISuiteListener methods called multiple times if multiple test elements (Daniel Qian & Julien Herr)
Fixed: GITHUB-169: IInvokedMethodListener methods executed several times before/after each test method (Mario Duarte & Julien Herr)
Fixed: GITHUB-154: MethodInterceptor will be called twice (Tim wu & Julien Herr)
6.9.6:
2015/07/15
New: GITHUB-717: Add assertThrows and expectThrows (Ryan Schmitt)
Fixed: GITHUB-755: Fixed reporting of retried tests (Ryan Schmitt)
Fixed: GITHUB-773: Test should not be skipped when the exception is expected (@CandyLiuM & Julien Herr)
6.9.5:
2015/07/12
Fixed: The ServiceLoaderTest on Windows (Mathieu Sebire)
Fixed: GITHUB-691: Fix classloading issue when using TestNG 6.9.4 and JMockit. (Mathieu Sebire)
Fixed: GITHUB-686: IAnnotationTransformer.transform is called for methods with testClass populated. (Łukasz Rekucki & Julien Herr)
Fixed: GITHUB-420: Before/AfterSuite methods may not run, when use inheritance, and enabled=false (Jakub Tokaj & Julien Herr)
Fixed: GITHUB-697: Make addFailedInvocationNumber thread-safe (Ryan Schmitt)
Fixed: GITHUB-698: Fix exit code reporting when IRetryAnalyzer is used (Ryan Schmitt)
Fixed: GITHUB-465: assertEquals(Collection, Collection) prints "null" when collections are different sizes (Michael Diamond)
New: GITHUB-710: AppVeyor is used for continuous integration on Windows (Julien Herr)
Fixed: GITHUB-599: IHookable ignored when a timeout is set (@ryanlevell & Julien Herr)
New: GITHUB-723: Allow users to add their own suite parser (Julien Herr)
Fixed: Allow '-testnames' option to work with '-xmlpathinjar' (@earthling)
Fixed: GITHUB-739: TestNG skips all test classes from suite when a @BeforeClass fails (Priyanshu Shekhar & Julien Herr)
Fixed: GITHUB-471: If @beforeMethod or @afterMethod fails then all children of the same base class will be skipped (Anton Panferov & Julien Herr)
Fixed: GITHUB-595: testng hang at switching test cases when running test cases with high thread count (vit0rg)
Eclipse:
Fixed: The 57% freeze bug (Patrick Hensley and @denyska)
6.9.4:
2015/05/09
Added: GITHUB-631: Avoid the static limitation of external DataProvider. (Julien Herr)
Added: GITHUB-631: Allow to use Guice injection in DataProvider. (Julien Herr)
Added: Drop support of Java6 and previous.
Added: GITHUB-617: Allow injection of org.testng.ITestContext into the guice parent module. (Julien Herr)
Fixed: GITHUB-606: RetryAnalyzer loops endlessly. (Krishnan Mahadevan)
Fixed: GITHUB-618: Start TestNG from jar cause recursive run of tests from packages in Suite XML without ".*" on the end (Stas Gromov)
Fixed: GITHUB-639: Typo on preserveOrder (tabei-k & Julien Herr)
Fixed: GITHUB-632: Typo in doc (Pétur Ingi Egilsson & Julien Herr)
Fixed: GITHUB-629: InvokedMethod doesn't recognize configuration method (Jan Mewes & Julien Herr)
Fixed: GITHUB-615: XmlSuite, XmlTest: Time-out tag not preserved (jphollingworth & Julien Herr)
New: GITHUB-638: Travis CI is used for continuous integration (Julien Herr)
New: GITHUB-647: SonarQube is used to follow technical debt (Julien Herr)
Added: GITHUB-616: org.testng.internal.Version will be always up-to-date (Julien Herr)
Fixed: GITHUB-634: Review of the collections package (Julien Herr)
Fixed: GITHUB-624: Fixed failure/error inversion in JUnitReportReporter (Jerome Jacob)
Fixed: GITHUB-545: TestNG running JUnit tests but not reporting all results for parameterized tests (Jonathan Leitschuh & jdillet)
Fixed: GITHUB-610: CustomizedSuites must be saved using utf-8 encoding (Juha Heljoranta)
Fixed: GITHUB-602: NoClassDefFoundError in TestNGClassFinder.<init> (aanno)
Fixed: GITHUB-529: Close InputStream and OuputStream after use (Andrew Gaul)
Fixed: GITHUB-532: Create the parent directory if it's missing (Ion Savin)
Fixed: GITHUB-541: Some OSGi manifest fixes (Evgeny Zhuravlev)
Fixed: GITHUB-657: Fix OSGI Import-Package to make jUnit4 dependency optional (Xavier Fournet)
Fixed: GITHUB-523: externally synchronize our use of the static SimpleDateFormat (mcosby)
Fixed: GITHUB-477: Typo in DTD attribute comment (Kamil Szymański)
Fixed: GITHUB-353: Typo in documentation (Jan Święcki)
Fixed: GITHUB-656: Upgrade to JCommander 1.48 (Ryan Schmitt)
Fixed: GITHUB-582: TestNG tests don't pass reliably on JDK8 (Ryan Schmitt)
New: GITHUB-645: TestNG project on Google Code redirect to GitHub
Fixed: GITHUB-310: Upgrade Guice (kronar & Julien Herr)
Fixed: GITHUB-87: @BeforeSuite/@BeforeTest methods happens to be disabled by mistake (romlom & Julien Herr)
Fixed: GITHUB-425: Wrong invocation order with lastTimeOnly (Rafael Winterhalter & Julien Herr)
Fixed: GITHUB-417: Expected Exceptions Message fails to match multi-line messages (Michael Diamond)
New: GITHUB-663: Add Guice Stage configuration for a suite (Clément Guillaume)
6.8.21:
2015/02/02
6.8.15:
2015/01/14
Fixed: OutOfMemoryException while generating reports.
Fixed: GITHUB-566: Build does not fail when successPercentage for @Test is not met
Fixed: XmlTest#setGroupInstances was not being shown in toXml().
Fixed: GITHUB-376: Some results can be lost (Konstantin Savin).
Fixed: Handle relative paths of Suite XML files properly (Nalin Makar)
6.8.5:
2013/05/13
Fixed: the OutOfMemoryException in reports
Fixed: Surefire + listeners "Can't load class" problem
===========================================================================
6.8.1
2013/03/30
Added: Descriptions in the HTML reports
Added: Various improvements to EmailableReporter (Abraham Lin)
Added: Allow injection of java.lang.reflect.Constructor and org.testng.ITestNGMethod into DataProvide (Vladislav Rassokhin)
Fixed: Assertions in the Assertions class were not failing properly.
Fixed: GITHUB-337: ConfigurationMethod#m_instance set to Boolean.FALSE due to incorrect constructor call in clone() + auto-boxing (davidely)
Fixed: Fix NPE for dependency methods/groups (Krishnan Mahadevan)
Fixed: preserve-order bug (found by VladSarrokhin).
Fixed: GITHUB-300: OutOfMemoryException from reporters when there are a lot of tests
Fixed: GITHUB-137: Main parameters with a default value should be overridden if a main parameter is specified
Fixed: GITHUB-107: Allow enum values without converting them to uppercase.
Fixed: @Guice with no modules specified is now supported
Fixed: Reporter.log() invoked from listeners were being discarded
Eclipse:
Added: Predefined listeners (Tim Wu)
Fixed: Compare dialog
===========================================================================
6.7
2012/07/15
Added: Big performance improvement when generating the reports (Frank Pavageau)
Added: <dependencies> allows you to specify group dependencies in testng.xml
Added: Blow up early if trying to include/exclude an unknown method
Added: <parameters> can now be specified under <include> (Storm Qi)
Added: GITHUB-243: Add Reporter Output per Test in XMLReporter (dunse)
Fixed: Better HTML escaping of the stack traces
Fixed: The failed assertions now use [] as delimiters instead of <> (better for the HTML reports)
Fixed: GITHUB-237: Wrong time format in XML reporter
Fixed: Threads were started sequentially instead of being interleaved
Fixed: dataProvider(parallel = true) was not killing its threads properly
Fixed: XmlSuite#toXml wasn't outputting the <groups> tag correctly
Fixed: testng-failed.xml was not carrying over the parameters from the original testng.xml
Fixed: BeforeClass failing in parent failed to skip methods in sub classes
Fixed: Better error message if <suite name=""> is missing
Fixed: GITHUB-221: Honor excludeGroups on testng tests when run in mixed mode (criccio)
Fixed: dependsOnGroups = {regexp} wasn't working properly (Alistair Ward)
Fixed: GITHUB-205: white-space was spelled whitespace in testng.css (carlin-scott)
Eclipse:
Fixed: Environment is not transferred when rerunning failed tests.
Fixed: Rerunning failed tests will preserve the environment of the original launch
===========================================================================
6.5.1
04/10/2012
Added: <suite allow-return-values="true"> (and in <test> as well)
Added: data-provider attribute to testng-results.xml
Added: Reporter display the results in the same order as test methods (Libor Zoubek)
Added: Support for running JUnit 4 tests (Lukas Jungmann)
Added: Ability to auto-detect JUnit tests ('-mixed' mode) (Lukas Jungmann)
Added: Support for ResourceCollections in an Ant tasks (requires Ant >= 1.7.0) (Lukas Jungmann)
Fixed: GITHUB-198: JUnitReportsReporter use commas in certain locales, which JUnitReports doesn't like
Fixed: GITHUB-173: Dependent methods executed out-of-order if method names match across classes (jjedMoriAnktah)
Fixed: ThreadLocal<ITestResult> leak (aslakknutsen)
Fixed: In the HTML reports, only show the first 100 characters of the parameters
Fixed: SkippedException are considered as real exception with @Test(expectedExceptions)
Eclipse:
Fixed: Java constants are properly resolved if they are used as group names (susanin)
Fixed: @Test(groups = Foo.CONSTANT) (susanin)
Fixed: Failed tests with allow-return-values="true" were not rerun
Added: <suite allow-return-values="true"> (and in <test> as well)
===========================================================================
6.4:
02/15/2012
Added: @DataProvider(indices) to return specific indices of a data provider
Added: New HTML reports
Added: configfailurepolicy=continue with DataProviders (toddq)
Added: ITestResult#getTestContext (bpedman)
Fixed: invocationCount > 1 + timeOut wasn't timing out properly
Fixed: When running TestNG programmatically, child xml suites are not run (when added using setSuiteFIles()) (Gaurav Gupta)
Fixed: GITHUB-145: Excessive test method execution (githubCast)
Fixed: GITHUB-149: reversed arguments in failAssertEqualsNoOrder().
Fixed: EmailableReporter: methods are now *really* sorted chronologically.
Eclipse:
Added: You can now add the testng.jar sources as a library (Nick Tan)
Added: Upgraded the plug-in to 3.4+ (Nick Tan)
Added: dependsOnGroups now fully supported
Fixed: @Parameters now works with both ("foo") and ({"foo"}) (davekerber)
===========================================================================
6.3.1
10/22/2011
Added: New system property: dataproviderthreadcount (Bill Ross)
Fixed: Configuration methods were reported incorrectly in listeners.
Fixed: Was creating too many listeners (Jacek Pulut)
Fixed: IAnnotationTransformer2 beforeTest/afterTest booleans were not being set
Fixed: GITHUB-92: @BeforeTest method in a super class will be called multiple time when alwayRun = true (Bubuntux)
Fixed: GITHUB-111: @AfterClass on base classes run once too many (lrivera)
Fixed: GITHUB-107: Displaying 0 tests run if a listener modifies the parameters of the suite
===========================================================================
6.3
10/17/2011
Added: "description" attribute on <include>, made available on ITestNGMethod#getDescription
Added: RemoteTestNG waits infinitely for a connection (Aleksey Kabanov)
Fixed: A method that's both a test and a factory would not invoke its data provider
Fixed: @AfterClass was not called if one of the methods was not enabled (Aleksey Kabanov)
Fixed: Groovy access bug
Fixed: The XML parser doesn't recognize parallel="instances"
Fixed: NPE when using inner classes
Fixed: GITHUB-90: @AfterClass not being run when the class contains included and not included methods
Fixed: @AfterClass not being run in some subclassing situations
Eclipse:
Fixed: Verbose levels specified in suites not respected
Fixed: Variable substitution in VM arguments is not working properly (svenhoff)
===========================================================================
6.2
08/18/2011
Added: xmlpathinjar to the TestNG ant task
Added: TestNG can now invoke package protected constructors
Added: Injectors created by the @Guice annotation are now shared at the <test> level
Added: IConfigurationListener is now a public listener, along with a new one: IConfigurationListener2
Added: When a method fails, only dependents of the same instance will be skipped
Added: parallel=instances for factory instance parallel runs
Added: @Factory(enabled)
Fixed: JUnitReports reports now report the cumulated time @{Before,After}Method+@Test for each test method
Fixed: JUnitReports reports have the name of the <test> instead of that of the first class
Fixed: Using preserve-order with a factory that creates instances of a different class causes NPE
Fixed: GITHUB-74: Bad ordering of test methods when using a @Factory constructor with dataProvider
Fixed: Changing the test result from success to failure in a listener would still count the test as a success
Fixed: ServiceLoader wasn't resolving correctly if no service loader classloader was specified
Fixed: Better ordering with mixed priorities and dependencies
Fixed: Improved detection of graph cycles in parallel runs
Fixed: @BeforeTest was invoked multiple times if a factory is used
Fixed: GITHUB-57: Allow usage of package protected constructor of test classes
Fixed: Injecting both Object[] and Method in @BeforeMethod didn't always work
Fixed: testng-results.xml now lists the results chronologically
Fixed: @Listeners specified on a base class will only be run once per listener class (dbriones)
Fixed: -groups and -excludegroups were no longer overriding testng.xml
Eclipse:
Added: Each data provider method now has a separate node entry in the TestNG view
Fixed: Nodes in error would sometimes remain green
Fixed: The TestNG context menu no longer appears where it shouldn't
===========================================================================
6.1.1
7/5/2011
Fixed: https://github.com/cbeust/testng/issues/56 testng-results.xml was reporting the instance name instead of the method name
Fixed: NPE when using preserve-order and factories.
Fixed: Depending on a skipped method would not cause a method to be skipped
===========================================================================
6.1
6/30/2011
Possible backward incompatible changes:
- Don't mutate the value returned by XmlTest#getIncludedGroups and XmlTest#getExcludedGroups.
Instead, use addIncludedGroup/addExcludedGroup.
- Failing methods that have dependees will only cause skips in the same instance. Different
test instances will not be affected
Added: Support for ServiceLoader for ITestNGListener
Added: @Factory(dataProvider / dataProviderClass) on constructors
Added: assertNotEquals() to Assert
Added: assertArrayEquals() to AssertJUnit
Added: Nested classes are now automatically added for consideration for inclusion
Added: <suite preserve-order="true"> will cause this attribute to be propagated to all <test> tags
Added: <groups> can now be specified under a <suite>
Added: Tycho compatibility (Aleksander Pohl)
Added: New <test> and <suite> flag: group-by-instances
Added: -xmlpathinjar to specify the path of testng.xml inside a test jar file
Added: ISuite#getAllMethods, to retrieve all the methods at the start of a suite.
Added: Output ITestResult attributes in xml report (nguillaumin)
Fixed: Thread safety problem in MethodInvocationHelper (Baron Roberts)
Fixed: Group dependencies were not being skipped properly.
Fixed: Dependency failures only impact the same instance
Fixed: Static classes could cause a StackOverFlowError
Fixed: IConfigurationListener was not extending ITestNGListener
Fixed: IConfigurationListener#onConfigurationFailure was never called
Fixed: TESTNG-476: <test> tags are now run in the order found in testng.xml
Fixed: Now showing failed/skipped error messages on the console for verbose >= 2
Fixed: ITestResult#getEndMillis() return 0
Fixed: TESTNG-410: Clearer error message
Fixed: TESTNG-475: @DataProvider doesn't support varargs
Fixed: Performance problems in EmailableReporter
Fixed: TESTNG-472: Better output for assertNull()
Fixed: ConcurrentModificationException when using parallel data providers.
Fixed: TESTNG-282: Problem when including+excluding packages (addicted)
Fixed: TESTNG-471: assertEquals(Map, Map) fails if a map is a subset of the other
Fixed: JUnitReporter generates an <error> tag for successful expectedExceptions tests
Fixed: ISSUE-47: Don't allow two <test>s with same name within same suite (Nalin Makar)
Fixed: If a listener implements both ISuiteListener and IInvokedMethodListener, only one of them gets invoked
Eclipse:
Added: New quick fix "Add static import org.testng.AssertJUnit.assertXXX"
Added: New workspace wide setting: excluded stack traces, to provide shorter stack traces in the view
Added: New "Clear results" icon in the tool bar
Added: When the search filter is modified, don't update the tree live if it is too big
Added: Two new @Test refactorings (pull to class level, push to method level)
Added: JUnit conversion: @Ignore
Added: JUnit conversion: assertArrayEquals()
Added: JUnit conversion: @RunWith(Parameterized.class)
Added: Support for Hamcrest failed assertions in the compare dialog
Added: JUnit conversion: suite() methods can now either be removed, commented out or left untouched
Fixed: JUnit conversion: super.setUp()/tearDown() were being removed when extending a class other than TestCase
Fixed: "Run as" menu not appearing for methods that take a generic parameter.
Fixed: The tree was incorrect if the same class is used in different <test> tags
Fixed: When creating a new Run/Debug configuration, "Launch.label" was displayed
Fixed: TESTNG-459: TestNG menu should not always be present in context menu (Mykola Nikishov)
Fixed: Performance problems in the plug-in
Fixed: Workspace-wide XML template files are not being honored.
Fixed: @BeforeClass/@AfterClass from JUnit4 are not being properly converted
Fixed: Conversions generate @Test() instead of @Test
===========================================================================
6.0
2011/03/16
Added: @Guice(moduleFactory) and IModuleFactory
Added: @Guice(module)
Added: timeOut for configuration methods
Added: -randomizesuites (Nalin Makar)
Added: IConfigurable
Fixed: @Test(priority) was not being honored in parallel mode
Fixed: @Test(timeOut) was causing threadPoolSize to be ignored
Fixed: TESTNG-468: Listeners defined in suite XML file are ignored (Michael Benz)
Fixed: TESTNG-465: Guice modules are bound individually to an injector meaning that multiple modules can't be effectively used (Danny Thomas)
Fixed: Method selectors from suites were not properly initialized (toddq)
Fixed: Throw an error when two data providers have the same name
Fixed: Better handling of classes that don't have any TestNG annotations
Fixed: XmlTest#toXml wasn't displaying the thread-count attribute
Fixed: TESTNG-438: Regression in 5.14.1: JUnit Test Execution no longer working
Fixed: TESTNG-436: Deep Map comparison for assertEquals() (Nikolay Metchev)
Fixed: Skipped tests were not always counted.
Fixed: test listeners that throw were not reporting correctly (ansgarkonermann)
Fixed: <suite junit="true"> wasn't working.
Fixed: In parallel "methods" mode, method interceptors that remove methods would cause a lock up
Fixed: EmailableReporter now sorts methods chronologically
Fixed: TESTNG-411: Throw exception on mismatch of parameter values (via DP and/or Inject) and test parameters
Fixed: IDEA-59073: exceptions that don't match don't have stack trace printed in console (Anna Kozlova)
Fixed: IDEA's plug-in was not honoring ITest (fixed in TestResultMessage)
Fixed: Methods depending on a group they belong were skipped instead of throwing a cycle exception
Fixed: TESTNG-401: ClassCastException when using a listener from Maven
Fixed: TESTNG-186: Rename IWorkerApadter to IWorkerAdapter (Tomas Pollak)
Fixed: TESTNG-415: Assert.assertEquals() for sets and maps fails with 'null' as arguments
Fixed: typo -testRunFactory
Fixed: NPE while printing results for an empty suite (Nalin Makar)
Fixed: Invoke IInvokedMethodListener.afterInvocation after fixing results for tests expecting exceptions (Nalin Makar)
Fixed: TESTNG-441: NPE in SuiteHTMLReporter#generateMethodsChronologically caused by a race condition (Slawomir Ginter)
Eclipse:
Added: Convert to YAML
Added: New global preference: JVM args
Added: Eclipse can now monitor a test-output/ directory and update the view when a new result is created
Added: Right clicking on a class/package/project now offers a menu "TestNG/Convert to TestNG"
Added: Excluded methods are now listed in the Summary tab
Added: "Description" column in the excluded methods table
Added: Dialog box when the plug-in can't contact RemoteTestNG
Added: Double clicking on an excluded method in the Summary tab will take you to its definition
Added: If you select a package before invoking the "New TestNG class" wizard, the source and package text boxes will be auto-filled
Added: When an item is selected in a tab, the same item will be selected when switching tabs
Added: A new "Summary" tab that allows the user to see a summary of the tests, sort them by time, name, etc...
Added: It's now possible "Run/Debug As" with a right click from pretty much any element that makes sense in the tree.
Added: JUnit conversion: correctly replaces assertNull and assertNotNull
Added: JUnit conversion: removes super.setUp() and super.tearDown()
Added: JUnit conversion: removes @Override
Added: JUnit conversion: replaces @Test(timeout) with @Test(timeOut) (5.14.2.4)
Added: JUnit conversion: replaces @Test(expected) with @Test(expectedExceptions) (5.14.2.4)
Added: JUnit conversion: replaces fail() with AssertJUnit.fail() (5.14.2.2)
Added: JUnit conversion: replaces Assert with AssertJUnit (5.14.2.1)
Added: The progress bar is now orange if the suite contained skipped tests and no failures
Added: Skipped test and suite icons are now orange (previously: blue)
Added: New method shortcuts: "Alt+Shift+X N", "Alt+Shift+D N" (Sven Johansson)
Added: "Create TestNG class" context menu
Added: When generating a new class, handle overridden methods by generating mangled test method names
Fixed: Green nodes could override red parent nodes back to green
Fixed: Was trying to load the classes found in the XML template file
Fixed: Stack traces of skipped tests were not showing in the Exception view
Fixed: XML files should be run in place and not copied.
Fixed: NPE when you select a passed test and click on the Compare Result icon (Mohamed Mansour)
Fixed: When the run is over, the plug-in will no longer force the focus back to the Console view
Fixed: The counter in the progress bar sometimes went over the total number of test methods (5.14.2.9)
Fixed: org.eclipse.ui.internal.ErrorViewPart cannot be cast to org.testng.eclipse.ui.TestRunnerViewPart (5.14.2.9)
Fixed: Workspace preferences now offer the "XML template" option as well as the project specific preferences (Asiel Brumfield)
Fixed: TESTNG-418: Only last suite-file in testng.xml run by Eclipse plugin
Documentation:
Added: Section on Selenium (Felipe Knorr Kuhn)
Added: Link to an article on TestNG, Mockito and Emma in the Misc section
===========================================================================
5.14.7
2011/01/27
Release for IDEA
===========================================================================
5.14.1
2010/10/2
Fixed: TESTNG-401: ClassCastException when using a listener from Maven
===========================================================================
5.14
2010/08/28
Added: test suites can now be run in parallel with -suitethreadpoolsize
Fixed: @Listeners now aggregate through base classes
Fixed: ISuite was no longer serializable
Fixed: Injection was sometimes not working properly when used with @Parameters
Fixed: TESTNG-400: afterMethod was called after onTestFailure()
Fixed: "excludedgroups" was not working on the ant task because of a typo
Fixed: ant task error if <classfileset> is used with no classes (welex91)
Fixed: TESTNG-404: threaded tests fail due to use of non-threadsafe collections (Marcus Better)
Fixed: preserve-order was not preserving class order with dependent methods
Fixed: RetryAnalyzer wasn't working properly with factories
Fixed: The ant task was no longer supporting ',' for testclass
Eclipse:
Fixed: The plug-in wasn't running Groovy tests correctly (Andrew Eisenberg)
Fixed: TESTNG-402 [Eclipse Plug-In] NPE occurred when I run twice a custom "Run configuration" on a group
===========================================================================
5.13.1
2010/08/05
Added: -methods
Added: -configfailurepolicy (Todd Quessenberry)
Added: -methodselectors (Todd Quessenberry)
Added: @NoInjection
Added: <test preserve-order="true">
Added: -testnames (command line) and testnames (ant)
Added: New ant task tag: propertyset (Todd Wells)
Added: ITestNGListenerFactory
Added: Passing command line properties via the ant task and doc update (Todd Wells)
Added: Hierarchical XmlSuites (Nalin Makar)
Added: Reporter#clear()
Fixed: NullPointerException when a suite produces no results (Cefn Hoile)
Fixed: Identical configuration methods were not always invoked in the correct order in superclasses (Nalin Makar)
Fixed: @DataProvider(parallel = true) was passing incorrect parameters with injection
Fixed: Replaced @Test(sequential) with @Test(singleThreaded)
Fixed: If inherited configuration methods had defined deps, they could be invoked in incorrect order (Nalin Makar)
Fixed: Initialize all Suite/Test runners at beginning to catch configuration issues right at start (Nalin Makar)
Fixed: Issue7: Issue86 Incorrect dates reported for configuration methods
Fixed: Issue24: OOM errors in SuiteHTMLReporter (Nalin Makar)
Fixed: Time outs specified in XML were not honored for <suite parallel="tests">
Fixed: <suite> and <test> time outs were hardcoded, they now honor their time-out attribute
Fixed: TestNG was hanging if no test methods were found
Fixed: onTestSuccess() was called after @AfterMethod instead of after the test method (test: test.listener.ListenerTest)
Fixed: XML test results contained skipfailedinvocationCounts instead of skipfailedinvocationcounts
Fixed: Issue4 assertEquals for primitive arrays, Issue34 assertNull javadoc updated
Fixed: Issue78 NPE with non-public class. Now throws TestNG exception
Fixed: NPE with @Optional null parameters (Yves Dessertine)
Fixed: TESTNG-387 TestNG not rerunning test method with the right data set from Data Provider (Francois Reynaud)
Fixed: Show correct number of pass/failed numbers for tests using @DataProvider
Fixed: Return correct method status and exception (if any) in InvokedMethodListener.afterInvocation()
Fixed: Trivial fixes: TESTNG-241 (log message at Info), Issue2 (throw SAXException and not NPE for invalid testng xml)
Fixed: Configuration methods couldn't depend on an abstract method (Nalin Makar)
Fixed: TestNG#setTestClasses was not resetting m_suites
Fixed: Exceptions thrown by IInvokedMethodListeners were not caught (Nalin Makar)
Fixed: @Listeners now works on base classes as well
Fixed: Test priorities were not working properly in non-parallel mode
Fixed: @Listeners wasn't working properly with ITestListener
Eclipse
Fixed: TESTNG-395 New wizard was creating classes called "NewTest"
Fixed: TESTNG-397 Class level @Test was preventing groups from showing up in the launch configuration
Doc
Updated Maven documentation (Brett Porter)
===========================================================================
5.12.1
2010/03/29
Maven update
===========================================================================
5.12
Removed: Javadoc annotation support
Added: @Listeners
Added: IAttributes#getAttributeNames and IAttributes#removeAttribute
Added: testng-results.xml now includes test duration in the <suite> tag (Cosmin Marginean)
Added: Injection now works for data providers
Added: TestNG#setObjectFactory(IObjectFactory)
Added: Priorities: @Test(priority = -1)
Added: New attribute invocation-numbers in <include>
Added: testng-failed.xml only contains the data provider invocations that failed
Added: IInvokedMethodListener2 to have access to ITestContext in listeners (Karthik Krishnan)
Fixed: @Before methods run from factories were not properly interleaved
Fixed: The TextReporter reports skipped tests as PASSED (Ankur Agrawal)
Eclipse:
Added: New file wizard: can now create a class with annotations, including @DataProvider
Added: You can now select multiple XML suites to be run in the launch dialog
Fixed: @Test(groups = <constant>) was taking name of the constant instead of its value.
Fixed: http://jira.codehaus.org/browse/GRECLIPSE-476 NPE with Groovy Tests (Andrew Eisenberg)
Fixed: The custom XML file is now created in the temp directory instead of inside the project
Fixed: In the launch dialog, now display an error if trying to pick groups when no project is selected
Fixed: Was not setting the parallel attribute correctly on the temporary XML file
===========================================================================
5.11
2009/12/08
Added: Dependent methods can now run in their own thread
Added: dataProviderThreadCount can be set from the command line and from ant (Adrian Grealish)
Added: ITestAnnotation#setDataProvider
Added: Assert#assertEquals() methods for Sets and Maps
Fixed: The text reporter was no longer reporting stack traces for verbose >= 2
Fixed: dataProviderClass was not respecting inheritance (like most attributes still)
Fixed: @BeforeSuite/@AfterSuite would run multiple times when used in a @Factory
Fixed: packages=".*" wasn't working properly (sandopolus)
Fixed: TestResult#getName now returns the description instead of the method
Fixed: @DataProvider and dependent methods were not skipping correctly (Francois Reynaud)
Fixed: TESTNG-347 suite with parallel="tests" and test with parallel="classes" doesn't work correctly (Rob Allen)
Fixed: TESTNG-67: @Configuration/@Factory methods in base class being ignored
Fixed: Inner test classes were not excluded properly (Carsten Gubernator)
Fixed: threadPoolSize without invocationCount was causing reporters not to be invoked
Fixed: A @Factory throwing an exception did not cause any error
Fixed: <classfilesetref> was not working properly in the ant task (Ed Randall)
Fixed: @BeforeClass methods were not running in parallel (Aidan Short)
Fixed: Test class with @ObjectFactory doesn't get instantiated via the factory
Fixed: Allow IObjectFactory to load from non-standard classloader (for PowerMock support)
Eclipse 5.11.0.19:
Added: New "parallel" preference setting (Windows / Preferences / TestNG)
Fixed: IIinvokedMethodListeners were not invoked
===========================================================================
5.10
Added: The output in the testng-results.xml is now sorted by the starting timestamp (Daniel Rudman)
Added: Better display of the test name and method description in the default and Emailable report
Added: If both -testjar and an XML file are provided on the command line, the latter will be used
Added: @Before and @After methods can be injected with the current XmlTest
Added: Methods that time out now display the stack trace showing where the time out occurred
Added: ITestResult#getAttribute and ITestResult#setAttribute
Added: @After methods can now be injected with an ITestResult
Added: @BeforeMethod and @AfterMethod methods can now be injected an ITestResult
Added: ISuite#getAttribute and ISuite#setAttribute to share data within a suite
Added: @Test(expectedExceptionsMessageRegExp = ".*foo.*")
Added: @DataProvider(parallel=true)
Fixed: @Test(dataProvider) was not working at the class level
Fixed: Display a better error message if the wrong exception is thrown with an expectedExceptions
Fixed: Classes created by factories were not run in the order they were created
Fixed: Dependent methods are now run closer to methods within their class
Fixed: xmlFileSet in ant was not working correctly (Sean Shou)
Fixed: Various oversights in the DTD (Will McQueen)
Fixed: XMLUtils was not escaping XML attribute values
Fixed: TESTNG-317: Sequence order mis-calculation: testing using suite in sequence for classes and same method names creates non-sequential order
Fixed: Test names (classes that implement org.testng.ITest) now appear more prominently in the HTML reports
Fixed: expectedExceptions=RuntimeException.class was not failing when no exception was throw
Fixed: TESTNG-291: Exceptions thrown by Iterable DataProviders are not caught, no failed test reported (Roberto Tyley)
Fixed: TESTNG-301: Need to include parameters in testNG report for test created by @Factory
Fixed: testng-failed.xml now includes skipped tests
Fixed: TestNG couldn't find Groovy files (Haw-Bin)
Eclipse
Fixed: TESTNG-313: Provide extension point to contribute test and report listeners (Erik Putrycz)
Fixed: Quick fixes no longer introduce deprecated annotations (Greg Turnquist)
===========================================================================
5.9
2009/04/09
Added: New ant task boolean flag: delegateCommandSystemProperties (Justin)
Added: skipfailedinvocations under <suite> in testng-1.0.dtd (Gael Marziou / Stevo Slavic)
Added: -testrunfactory on the command line and in the ant task (Vitalyi Pamajonkov)
Added: TESTNG-298: parallel="classes", which allows entire classes to be run in the same thread
Added: @BeforeMethod can now declare Object[] as a parameter, which will be filled by the parameters of the test method
Added: IAnnotationTransformer2
Added: @Test(invocationTimeOut), which lets you set a time out for the total time taken by invocationCount
Added: IInvokedMethodListener
Added: -testjar supports jar file with no testng.xml file
Fixed: IInvokedMethodListener wasn't properly recognized from the command line (Leonardo Rafaeli)
Fixed: TESTNG-309 Illegal default value for attribute in DTD file
Fixed: TESTNG-192: JUnit XML output includes wrong tests (Aleksandar Borojevic)
Fixed: Set a generated suite to default to non-parallel (Mark Derricutt)
Fixed: -testJar command line parsing bug
Fixed: testng-failed.xml didn't include the listeners
Fixed: annotation transformers were not run when specified in testng.xml
Fixed: TESTNG-192: JUnit XML output includes wrong tests (Borojevic)
Fixed: @Parameters was not working correctly on @BeforeMethods with @DataProvider used on @Test methods
Fixed: testng-failed.xml was sometimes incorrectly generated (Borojevic)
Fixed: TestNG-228: Assert.assertEqualsNoOrder
Fixed: TestNG-229: Assert.assertEquals does not behave properly when arguments are sets
Fixed: TESTNG-36: assertEquals(Collection actual, Collection expected, String message) may have bug
Fixed: TESTNG-296: Malformed jar URLs breaking -testJar
Fixed: TESTNG-297: TestNG seemingly never stops running while building failed test suite (Gregg Yost)
Fixed: TESTNG-285: @Test(sequential=true) works incorrectly for classes with inheritance
Fixed: TESTNG-254: XMLSuite toXML() ignores listeners
Fixed: TESTNG-276: Thread safety problem in Reporter class
Fixed: TESTNG-277: Make Reporter.getCurrentTestResult() public
Fixed: Potential NPE in XmlTest#getVerbose (Ryan Morgan)
Fixed: EmailableReporter only displayed the first group for each test method
Fixed: time-outs were not working in <test> and <suite>
Fixed: @BeforeTest failing in a base class would not cause subsequent test methods to be skipped
Fixed: TESTNG-195: @AfterMethod has no way of knowing if the current test failed
Fixed: TESTNG-249: Overridden test methods were shadowing each other if specified with <include>
Fixed: DataProviders from @Factory-created tests were all invoked from the same instance
Fixed: enabled was not working on configuration methods
Fixed: IIinvokedMethodListener was not correctly added in TestNG
Fixed: NPE in XmlSuite#toXml
Fixed: TESTNG-231: NullPointerException thrown converting a suite to XML (Mark)
Doc:
Added: 5.20: IInvokedMethodListener
Added: -testjar
===========================================================================
5.8
Fixed: TestNG-220: Ignore class definition/loader issues when scanning classpath for implicit classes
Fixed: TestNG-224: Fix for relative suite filenames in XML file
Added: TestNG-213: @Optional on a method parameter to allow optional @Parameters
Fixed: TestNG-214: SkipException and TimeBombSkipException should accept nested exceptions
Fixed: TestNG-211: new Parser(inputStream) doesn't work
Added: Methods that form a cycle are now shown when the cycle is detected
Added: Support for <listeners> in testng.xml
Added: IMethodInterceptor
Added: @TestInstance on a data provider method parameter
Fixed: @AfterMethod(lastTimeOnly) didn't work properly with data providers
Added: antlib.xml to allow autodiscovery of Ant task definition
Fixed: name attribute on <test> is required
Doc:
Added: Method Interceptor
Added: @Optional
Added: Doc for IMethodInterceptor (5.16) and TestNG listeners (5.18)
Added: 5.19: Dependency injection
===========================================================================
5.7
Added: @BeforeMethod(firstTimeOnly) and @AfterMethod(lastTimeOnly)
Added: @BeforeMethods can now take a Method and ITestContext parameters (like @DataProvider)
Fixed: logging about abstract classes moved to level 5
Added: if @Parameter is missing from testng.xml then it is read from the System properties
Fixed: Don't run a @DataProvider method as a test when a class-level @Test is present
Added: Attribute @Test#skipFailedInvocations
Fixed: TESTNG-169 Error message: <method> is depending on nonexistent method null ("null" is uninformative)
Fixed: -listener takes comma-separated classes
Added: RetryAnalyzer (experimental) (Jeremie)
===========================================================================
5.6
2007/06/14
Added: SkipException/TimeBombedSkipException for manual skipping
Added: <tests> can now be disabled at xml level using <test enabled="false">
Added: Suite files that only contain other suites do not get reported
Fixed: @BeforeClass methods would incorrectly report cyclic graphs
Added: get/setAttribute to ITestContext
Added: plugging in factory objects to handle the actual instantiation of tests
Added: dataProvider to @Factory
Added: ISuite now gives access to the current XmlSuite
Fixed: TESTNG-139 dependsOnMethods gets confused when dependency is "protected"
Fixed: TESTNG-141 junit attribute set to false in testng-failed.xml when it should be true
Fixed: TESTNG-142 Exceptions in DataProvider are not reported as failed test
Added: Improved behavior for @Before/@AfterClass when using @Factory
(http://forums.opensymphony.com/thread.jspa?threadID=6594&messageID=122294#122294)
Added: Support for concurrent execution for invocationCount=1 threadPoolSize>1 and @DataProvider
(http://forums.opensymphony.com/thread.jspa?threadID=64738&tstart=0)
Added: New TestNG specific XML report, generated by default in 'xml' subdirectory of test-output
Added: support in strprotocol for passing the ITest.getTestName() information
Fixed: TESTNG-152 If DataProvider is not found, the exception message should tell exactly what happened
Eclipse plug-in
Fixed: Bug that made group launch configurations unusable
Fixed: The plugin doesn't create the correct launch configuration for @Factory
Fixed: Method based launchers cannot be editted
Fixed: Plugin hangs while executing test with dataprovider that sends \n, \r messages
Added: display ITest.getTestName()
IDEA plug-in
Fixed: IDEA 7.0 compatibility
Fixed: occasional 'illegal arguments exception'
Fixed: TESTNG-151 Final passing test result is not properly hidden
Added: Auto-completion for dependsOnMethods
Added: Highlighting of invalid groups/methods in dependsOn*
===========================================================================
5.5
2007/01/25
Fixed: @BeforeGroup methods were run twice when in a base class
Fixed: @BeforeGroup methods were run twice with a @Test at class level
Fixed: parallel="tests" didn't work as advertised
Added: Support for thread-count at test level
Added: Method selectors receive a Context and can stop the chain with setStopped()
Fixed: XmlMethodSelector was always run first regardless of its priority
Added: @BeforeGroups/@AfterGroups can live in classes without @Test methods
Added: DataProvider can now take an ITestContext parameter
Fixed: Wasn't parsing <selector-class-name> correctly
Fixed: Annotation Transformers now work on class-level annotations
Fixed: Some class-level @Test attributes were not always honored
Added: Clean separation between @Test invocation events and @Configuration invocation events
(see also TESTNG-111)
Added: Test instances created by @Factory now run in multiple threads in parallel mode
Fixed: @Before/@AfterGroups invocation order
Fixed: TESTNG-27: Parameters are not used on <test> level anymore
Fixed: TESTNG-107 don't create an output directory if "outputDirectory" is null
Fixed: TESTNG-127 UseDefaultListeners in Ant Task does not work
Fixed: TESTNG-119 Running TestNG runner with invalid '-sourcedir' on JDK14 JavaDoc annotated test classes won't fail.
Fixed: TESTNG-113 Dependent methods within the same static inner class are not found
Fixed: TESTNG-125 TestNG failed for test classes under *.java*.* pakages
Eclipse plug-in
Fixed: issue with launch configuration
Fixed: TESTNG-124: setting location of testng reports output
===========================================================================
5.4
Fixed: Ant task issue with paths containing spaces
Added: for @BeforeGroups and @AfterGroups specifying the groups() attribute will auto-include the method
into those groups by default (previously you had to also provide the value() attribute).
Added: the load @Tests (invocationCount + threadPoolSize) are triggered simultaneous
Fixed: reports are correctly displaying the thread info
Added: @DataProvider name defaults to method name
Added: support for remote protocol to pass parameter information
Fixed: TextReporter logs information about the parameters of the test methods
Fixed: concurrency issue in JUnitXMLReporter
Fixed: output of JUnitXMLReporter must be CDATA
Fixed: XML unsupported annotations/parallel attribute values are reported
Eclipse plug-in
Fixed: groups with multi-attribute javadoc annotations
Fixed: consistent behavior for dependsOnMethods
Fixed: consistent behavior for tests with dependsOnGroups (a warning is emitted)
Fixed: consistent merge of configuration arguments when an existing launch configuration exists
===========================================================================
5.3
2006/10/30
Fixed: use a single instance of bsh.Interpreter
Added: @Before/@AfterMethod can declare a java.lang.reflect.Method parameter to be informed about the @Test method
Fixed: super classes must not be listed in testng-failures.xml
Fixed: parallel attribute must not appear if empty or null in testng-failures.xml
Fixed: parsing for javadoc annotations is done on request only
Added: improved multiple suite summary page report
Added: -target option deprecated in favor of -annotations javadoc|jdk
Fixed: filesets in the ant task didn't work if the paths have spaces in them
Fixed: Before/After Suite were behaving wrong in parallel execution
Added: A generic/extensible RemoteTestNG was added to the core
Fixed: Before/AfterGroup-s were behaving wrong when using invocationCount, dataProvider and threadPoolSize
Fixed: improved support for running different annotation type tests in the same suite
Fixed: testng-failed.xml was generated even if there were no failures/skipps
Fixed: -usedefaultlisteners was wrongly passed to JVM instead of TestNG options
Added: Attribute dataProviderClass for @Test and @testng.test
Fixed: Forgot to account for cases where both invocationCount and DataProviders are present
Fixed: AfterGroups were invoked out of order with invocationCount and DataProviders
Fixed: Reporter.getOutput() returned an empty array if a timeOut was specified
Added: testng.xml now supports <suite-files>
Added: ant task can receive several listeners
Fixed: TESTNG-109 Skipped tests with expected exceptions are reported as failures
Added: ant task can now select the parallel mode for running tests
Fixed: ant task correctly deals with empty groups and excludedgroups parameters
Added: ant task can override default suite and test names
Added: comand line support for setting parallel mode, suite and test names
Eclipse plug-in
Added: Support for configuring per project usedefaultlisteners
Added: Contextual drop-down menu on failures tab of the TestNG view to enable running/debugging method failure only
Added: Suppport for configuring per project TestNG jar usage (project provided one or plugin provided one)
===========================================================================
5.2
Added: "-usedefaultlisteners true/false" to command line and ant
Added: EmailableReporter (from Paul Mendelson)
Added: parallel can now be "methods" or "tests". Boolean version deprecated
Added: TestNGAntTask now uses the @ syntax to invoke TestNG
Added: Command line understands @ syntax
Added: JUnitConverter uses the new syntax
Added: -groups to JUnitConverter
Fixed: Throw proper exception when a DataProvider declares parameters
Added: completely revamped JUnit support (should run all kind of JUnit tests)
Fixed: TESTNG-40 (Bug in testng-failed.xml generation)
Fixed: TESTNG-106 (Failed "@BeforeSuite" method just skipps the last test in xml-file)
Fixed: Success on 0 tests (http://forums.opensymphony.com/thread.jspa?threadID=41213)
Eclipse plug-in
Added: TESTNG-105 Automaticaly define TESTNG_HOME classpath variable
===========================================================================
5.1
2006/08/18
Added: @Test(sequential = true)
Fixed: TESTNG-102 (Incorrect ordering of @BeforeMethod calls when a dependency is specified)
Fixed: TESTNG-101 (HTML output contains nested <P> tags and a missing <tr> tag)
Added: support for specifying test-only classpath (http://forums.opensymphony.com/thread.jspa?messageID=78048&tstart=0)
Fixed: TESTNG-93 (method selectors filtering @BeforeMethod)
Fixed: TESTNG-81 (Assert.assertFalse() displays wrong expected, actual value)
Fixed: TESTNG-59 (multiple method selectors usage results in no tests run)
Fixed: TESTNG-56 (invocation of @Before/AfterClass methods in parallel/sequential scenarios)
Fixed: TESTNG-40 (failures suite does not contain @Before/After Suite/Test methods)
Fixed: TESTNG-37 (allow passing null parameter value from testng.xml)
Fixed: TESTNG-7 (display classname when hovering method)
Eclipse plug-in
Added: run contextual test classes with parameters from suite definition files
Added: TESTNG-100 (Show HTML reports after running tests)
Added: TESTNG-97 (Double click top stack to raise comparison)
Added: TESTNG-84 (plug-in UI for suite option should support absolute path)
Added: TESTNG-20 (copy stack trace)
Fixed: TESTNG-72 (display groups with non-array values)
Fixed: TESTNG-64 (Eclipse plug-in applies added groups to all launch configurations)
Fixed: TESTNG-28 (Cannot select groups from dependent eclipse projects)
Fixed: TESTNG-25 (do not display fully qualified method name when running contextual test class)
Improved behavior:
TESTNG-98 (temporary files have guaranteed fixed names)
TESTNG-95 (Assertion failed comparison trims trailing ">")
TESTNG-70 (TestNG prevents eclipse from opening an older CVS version of a java class)
display of test hierarchy information (TESTNG-29)
===========================================================================
5.0.1
Eclipse plug-in
Added: Output directory for the tests
Added: Can now specify listener classes
===========================================================================
5.0.1
Fixed: reports generated by SuiteHTMLReporter do not work with JDK1.4
===========================================================================
5.0
2009/04/01
Added: Ant task: support for JVM, workingDir, timeout
Added: Stack traces can be interactively shown in the HTML reports
Added: Link to testng.xml in the reports
Added: New structure for reports, suites go in their individual directory
Added: @Test(suiteName) and @Test(testName)
Added: The stack traces in reports do not include TestNG frames (system property testng.exception)
(see: http://groups.google.com/group/testng-dev/browse_thread/thread/9f4d46ade10b0fda)
Fixed: Exit with error when no methods are run
(see: http://groups.google.com/group/testng-dev/browse_thread/thread/3c26e8a5658f22ac)
Added: List of methods in alphabetical order
Fixed: Class-scoped annotations were not recognized when inherited
Added: Deprecated @Configuration and introduced @BeforeSuite/Test/Class/TestMethod
Added: Deprecated @ExpectedExceptions and moved it into @Test
Added: expectedExceptions to @Test, deprecated @ExpectedExceptions
Added: New annotations: @BeforeSuite, @BeforeTest, etc...
Fixed: Was returning an exit code of 0 if a cyclic graph was detected
Added: Interface org.testng.ITest so that tests can declare a name
Fixed: The Text reporter was reporting the square of the actual number of methods
Fixed: Bug reported by Eran about dependencies with an afterClass method
Added: IHookCallBack now receives the ITestResult in its run() method
Added: Name of suite for command line can be set with -Dtestng.suite.name=xxx
Fixed: TestNGAntTask was hardcoding m_haltOnFSP to true
Fixed: Passing a null parameter caused an NPE in the reports
Added: "listener" to the ant task (and documentation)
Added: if patch-testng-sourcedir.properties is found in the classpath
with a property "sourcedir" containing a ; separated list of
directories, this list will override -sourcedir.
===========================================================================
4.7
Added: Maven 2 plug-in
Fixed: Message formattings in TestNG assertion utility class
Fixed: @Factory methods were counted as @Test as well
http://jira.opensymphony.com/browse/TESTNG-51
Fixed: All DataProvider parameters were shown in the HTML report
Fixed: Bug in testng-failed.xml generation
Fixed: <packages> bug when using a jar file to load the test classes
Added: alwaysRun for before @Configuration methods
http://jira.opensymphony.com/browse/TESTNG-35
Fixed: groupless @Configurations were not invoked if a method depends on a group
http://jira.opensymphony.com/browse/TESTNG-45
Added: beforeGroups/afterGroups to @Configuration
Eclipse plugin:
Added: last contextual launch is available in Eclipse launcher lists
Fixed: 3.2M5 integration (removed dependency on non-existing class)
Fixed: testng-failures.xml generation
===========================================================================
4.6
2006/27/02
Added: Documentation contains the new reports
Added: TestNG.setUseDefaultListeners(boolean)
Added: Descriptions now appear in TextReporter (verbose>=2) and the HTML reports
Added: description attribute to @Test and @Configuration
Added: combined Reporter output in the reports
Added: methods not run in the reports
Added: org.testng.IReporter
Added: threadPoolSize to @Test
Added: Reports now show relative timings (start at 0)
Added: Reports now show different colors depending on the methods' classes
Added: Reports now show all parameters used to invoke the test method
Added: org.testng.Reporter
Added: DataProviders can accept a Method as first parameter
Fixed: Extraneous implicit inclusion of a method
Eclipse plugin:
Added: Run/Debug as TestNG test from the editor contextual menu
Fixed: TESTNG-24: 'Run as testng test' does not appear of the Test annotation does not have a group
Fixed: TESTNG-18: Eclipse plugin ignores Factory annotation
Fixed: TESTNG-21: Show differences when double clicking assertion exceptions
Added: UI allows setting orientation (even more space)
http://forums.opensymphony.com/thread.jspa?threadID=17225&messageID=33805#33805
===========================================================================
4.5
2007/07/02
Core:
Fixed: Methods were not implicitly included, only groups
Fixed: Bug with failed parent @Configuration don't skip child @Configuration/@Test invocations
Fixed: Bug with overridding @Configuration methods (both parent and child were run)
Fixed: Bug when overriding beforeClass methods in base class (cyclic graph)
Added: Support for JAAS (see org.testng.IHookable)
Fixed: Problem with nested classes inside <package name="foo.*"
Fixed: If a group is not found, mark the method as a skip instead of aborting
Fixed: testng-failed.xml was not respecting dependencies
Fixed: class/include method in testng.xml didn't work on default package
Fixed: DTD only allowed one <define>
Fixed: ArrayIndexOutOfBoundsException for jMock
Added: dependsOnMethods can contain methods from another class
Fixed: JUnitConverter required -restore, not any more (option is now a no-op)
Fixed: JUnit mode wasn't invoking setName() on test classes
Added: Regular expressions for classes in <package>
Added: Distributed TestNG
Fixed: Command line parameters and testng.xml are now cumulative
Fixed: Reports now work for multiple suites
Fixed: Was ignoring abstract classes even if they have non-abstract instances
Fixed: If setUp() failed, methods were not skipped
Fixed: Was not clearly indicating when beforeSuite fails
Added: @Configuration.inheritGroups
Fixed: inconsistency between testng.xml and objects regarding method selectors
Eclipse plug-in:
New look for the progress view.
===========================================================================
4.4
Core:
Fixed: testng-failures.xml was not excluding methods from base classes
Fixed: Bug in suites of suites for JUnit mode
===========================================================================
4.3
Core:
Fixed: testng-failures.xml was not excluding methods from base classes
Fixed: Bug in suites of suites for JUnit mode
Added: Excluded groups on command line and ant task
Fixed: When including a group, implicitly include groups depended upon
Fixed: When depending on several groups, wasn't skipped if one of them failed
Fixed: Failures weren't reported accurately in the JUnitReports report
Fixed: Wasn't throwing an exception if depending on a non-existing group
===========================================================================
4.2
Core:
Fixed: wasn't excluding methods in base classes
Added: alwaysRun for tests (soft dependencies)
Fixed: Class-level enabled=false were not honored
Fixed: Bug with multiple dataproviders on same class
Fixed: Bug with dataprovider defined in the parent class
Fixed: Bug with dataprovider defined in a subclass
Fixed: Bug with dataprovider defined in an abstract class
Fixed: testng-failures generation was excluding the methods even if a failed test depended on it
===========================================================================
4.1
Core:
Added: @DataProviders can return Iterable<Object[]>
Fixed: Superclass test methods were not called in the presence of a class @Test
Added: Reporter class to log messages in the HTML reports
===========================================================================
4.0
2005/11/10
Core:
Fixed: suite methods now invoked only once in a hierarchy
Added: @DataProvider and @testng.data-provider
Fixed: Interleave order now respected for before/afterClass methods
Added: Can now invoke java -jar testng-2.6.jar <...>
Added: Support for BeanShell
Added: Method Selectors (IMethodSelector)
Fixed: In the absence of dependencies, @Configuration methods respect inheritance
Fixed: Bug in multithreaded dependencies on methods
Fixed: dependsOnGroups wasn't working on regular expressions
Fixed: Bug in <package> when directories contain spaces in their names
Fixed: Introduced a JDK5 dependency in the JDK1.4 build (getEnclosingClass())
Fixed: Output directory in ant task was not honored if it didn't exist
Fixed: Problem with timeout according to
http://forums.opensymphony.com/thread.jspa?threadID=6707
Eclipse plug-in:
Fixed: Wasn't handling linked directories correctly
Fixed: Bug in QuickFix implementation
Added: Quick Fix for JUnit conversion (Annotations and JavaDoc)
Fixed: Methods Run as TestNG test
Added: Package level Run as TestNG test
Fixed: Resources from the linked directories are using a wrong path when
passed to command line TestNG
IDEA plug-in:
Added: Support for JDK 1.4 (both projects and IDEA itself)
Fixed: Classes that contained only configuration were ignored
===========================================================================
2.5
2005/08/08
Added: ITestListener.onTestStart(ITestResult)
Added: Support for <packages>
Added: Resource files for easier ant taskdefs
Fixed: @Configuration methods were not invoked with individual test methods
Fixed: Bug with ExpectedExceptions
Fixed: Didn't support nested factory classes
Fixed: NPE if -target is omitted with JDK 1.4
Fixed: @Configuration failures in a class would cause other classes to fail
Added: alwaysRun
Fixed: beforeTestClass/afterTestClass were broken for a pathological case
Added: @Configuration(alwaysRun)
Added: JUnitConverter task
Fixed: < and > characters in reports were not escaped
Eclipse plug-in:
Fixed: Class dialog wasn't showing @Factory classes
IDEA plug-in:
First release!
Documentation:
Added: Brand new look!!!
Added: Section on testng.xml
Fixed: Numbering of sections
===========================================================================
2.4
2005/07/05
Changed: New package: testng.org
Fixed: Bug with @ExpectedException occuring the parallel mode
Fixed: Bug with parameters and beforeTest
Added: IInstanceInfo support
Fixed: methods were not excluded when included by groups
Fixed: testng-failures.xml is now including also the beforeSuite/afterSuite methods
Fixed: generating the testng-failures.xml is now working as expected
Fixed: Factories call all the tests even if some of them fail along the way
Fixed: Better JUnit support (wasn't creating individual instances)
Fixed: dependsOnGroups didn't work across different classes
Added: command line (and Ant) -groups option
Added: @Parameters (and made parameters attribute deprecated)
Added: Parameters for constructors
Fixed: Better interleaving of before/afterTestMethods
Fixed: Ant task
Fixed: TestNGException thrown when TestNG conditions are not fulfilled
Documentation:
- New assert classes
- New ways to launch
- JUnitConverter documentation
- new beforeSuite/afterSuite
===========================================================================
2.3
2005/04/12
Fixed: Spaces are now legal in JavaDoc comments
Added: documentation for @Factory
Fixed: factories were called multiple times
Added: beforeSuite and afterSuite
Fixed: inheritance and scope now working properly for annotations
Fixed: dependsOnMethods wasn't working for 1.4
Added: Better stack traces
Added: Better syntax for included/excluded methods
Fixed: Better verbose support
Fixed: Various fixes for the Eclipse plug-in
Added: Can specify a class name on the command line
Fixed: Default package bug in JUnitConverter
Added: Regression tests for JUnitConverter
Added: -quiet option to JUnitConverter
===========================================================================
2.2
Fixed: Wasn't handling several testng.xml files correctly
Fixed: Renamed -src to -sourcedir
Fixed: Complains if no sourcedir is specified in 1.4
Added: In 1.4, don't require annotations="javadoc"
Fixed: If setUp fails, complain and mark test methods as skips
Fixed: Dependent methods weren't working for 1.4
===========================================================================
2.1
2005/02/12
Added: Parser can accept an InputStream for testng.xml
Fixed: expected-exceptions now fails if test passes
Fixed: reports now use the suite name in HTML
Added: invocationCount and successPercentage
Added: dependsOnMethods
Added: timeOut works in non-parallel mode
===========================================================================
2.0
2004/12/06
Added: port on JDK 1.4
===========================================================================
1.3
Added: new view: classes (still experimental)
Added: timeout on methods
Added: thread-count
Added: TestNG is now multithread, see "parallel" in <suite>
===========================================================================
1.2
Added: JUnitConverter
Fixed: Bug with afterClasses (test: AfterClassCalledAtTheEnd)
===========================================================================
1.1
Added: new links for methods and groups in the HTML report
Added: <methods>
Added: <fileset> to <testng>
===========================================================================
1.0
2004/04/28
http://beust.com/weblog/2004/04/28/
Fixed: Updated to the new DTD
Fixed: Suite table of contents displays failures first
Fixed: Bug in afterTestClass
Added: Validating testng.xml
Added: Scoped parameters
Added: testng.xml
Removed: Property quiet
Changed: Verbose is now an integer
Added: Dependent methods
===========================================================================
0.9
Added: Groups of groups
Added: Groups for Configuration methods
Added: Parameters
===========================================================================
0.2
Fixed: Merged TestMethod and TestClass into Test
Added: HTML report
Added: Regexps for groups
Fixed: Inheritance of methods
Fixed: ExpectedException is now called ExpectedExceptions
|