1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690
|
\input texinfo @c -*-texinfo-*-
@finalout
@settitle Greg testing framework
@setfilename greg.info
@c
@c This file documents the GNUstep-Guile Regression testing framework `Greg'
@c
@c Copyright (C) 1998,1999,2000,2001 Free Software Foundation, Inc.
@c
@c This text may be freely distributed under the terms of the GNU
@c General Public License.
@c
@ifinfo
@format
START-INFO-DIR-ENTRY
* Greg: (greg). Another GNU testing framework.
END-INFO-DIR-ENTRY
@end format
@end ifinfo
@syncodeindex ky cp
@syncodeindex fn cp
@setchapternewpage odd
@settitle Greg testing framework
@titlepage
@title The Greg testing framework
@subtitle for Greg Version @VERSION@
@sp 1
@subtitle February 2001
@author Richard Frith-Macdonald <rfm@@gnu.org>
@page
@vskip 0pt plus 1filll
Copyright @copyright{} 1998,1999,2000,2001 Free Software Foundation, Inc.
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.
@noindent
Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.
Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions.
@end titlepage
@ifinfo
Copyright @copyright{} 1998,1999,2000,2001 Free Software Foundation, Inc.
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.
@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries a copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
@end ignore
Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.
Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions.
@node Top, Copying, ,
@top Greg
Greg is a framework for running test suites on software tools.
Greg was developed for the
@uref{http://www.gnustep.org/,GNUstep} project.
This file describes version @VERSION@ of Greg.
@menu
* Copying:: GNU Library
* News:: New in this release
* Overview:: Greg in brief
* Installation:: Installing Greg
* Invoking Greg:: How to run tests
* Writing tests:: How to write a testsuite
* Index:: Index
@end menu
@end ifinfo
@iftex
@raggedbottom
@end iftex
@node Copying, News, Top, Top
@chapter Copying
See the file @file{COPYING.LIB}.
@node News, Overview, Copying, Top
@chapter New in this release
@cindex News
New in 1.4
@itemize @bullet
@item
Updated to save/restore expect settings at start of a tool and end of a
tool so that they can be readily set up for the duration of a tool test.
@item
The default action on reading an end of file from a child is now to
make the test FAIL rather than be unresolved.
@item
The default action on having a timeout when reading from a child is now to
make the test FAIL rather than be unresolved.
@item
Added hooks for pre and post testcase procedures.
@end itemize
New in 1.3
@itemize @bullet
@item
Add @code{greg-directory} to return the absolute path to the directory
in which the executing test script is located.
@end itemize
New in 1.2
@itemize @bullet
@item
Permit the files in the @code{greg-files} variable to use the @file{.scm}
extension or not.
@item
Added @code{greg-paths} variable to permit direct running of named files
rather than using the DejaGNU-like tools directory structure.
@item
Alter the @file{greg} script to set the @code{greg-paths} variable to
contain any files listed on the command-line.
@item
Added @code{greg-expect-pass} and @code{greg-expect-fail} macros.
@end itemize
New in 1.1
@itemize @bullet
@item
Changed from using GNU Library General Public License to the newer
GNU Lesser General Public License.
@item
Changed from using a compiled module (deprecated in guile) to using a
dynamic library for pty support.
@end itemize
New in 1.0
@itemize @bullet
@item
Nothing really - a few very minor fixes - no bug reports, which is why I
decided to risk bumping the version number to 1.0 - presumably people will
now find hundreds of bugs...
@item
Updated to work with Guile and 1.3.4
@end itemize
New in 0.7
@itemize @bullet
@item
Updated to work with Guile 1.3.2 and 1.3.3
@item
Updated to work on GNU/Linux systems with pts rather than ptys
@item
Improved configuration code
@end itemize
New in 0.6
@itemize @bullet
@item
Converted makefiles to work without GNUstep
@item
Improved configuration code
@item
Improved documentation somewhat
@item
Updated to work with guile version 1.3
@end itemize
New in 0.5
@itemize @bullet
@item
greg-testcase now returns a useful result.
@item
Fixed bug in creating child processes on some systems.
@end itemize
New in 0.4
@itemize @bullet
@item
Allow an empty string as the program name for creating a child process -
in which case, we fork without execing to another program.
Modified Greg to allow @code{(quit)} to exit from the test system.
@end itemize
New in 0.3
@itemize @bullet
@item
Improved installation - now checks for location of guile shell.
@item
Fixed a couple of minor errors in the documentation.
@end itemize
New in 0.2
@itemize @bullet
@item
Ported to Unixware 2.1.3 - should work for any SysV4.2 based unix.
@end itemize
General news
This release of Greg provides a test framework much like that of
DejaGNU, but also provides Guile modules to permit `embedded' testing of
applications that use Guile as a scripting language and libraries that
are directly accessible to Guile.
This release has been tested with Guile-1.3.4 and 1.3.3, it will certainly
not run on versions earlier than 1.3.1
Apologies to Guile/Scheme/Lisp programmers out there - I came to this from
Objective-C programming and taught myself during the four weeks that
I wrote GNUstep-Guile and Greg - so my code is probably really ugly - but it
does seem to work.
The code to run a child process in a pseudo-terminal works for
GNU/Linux and SysV4.2 systems - it probably needs work to make it
more portable -
Please email bugfixes and comments to <bug-gnustep@@gnu.org> or directly to
me <rfm@@gnu.org>
@node Overview, Installation, News, Top
@chapter Greg in brief
@cindex overview
Greg is a framework for testing other programs and libraries.
Its purpose is to provide a single front end for all tests and to
be a small, simple framework for writing tests.
Greg leverages off the Guile language to provide all the power (and more)
of other test frameworks with greater simplicity and ease of use.
The simplicity of the Greg framework makes it easy to write tests for
any program, but it was specifically written for use with GNUstep-Guile
to permit direct testing of the
@uref{http://www.gnustep.org/,GNUstep} libraries without the
necessity to run a separate driver program.
The core functionality of Greg is a Guile module which can be loaded
into any software with an embedded Guile interpreter. Any program
which uses Guile as it's scripting language can therefore use Greg
to test itself directly!
For testing external programs, Greg provides a compiled module that may
be dynamically linked into Guile to permit you to run an application as
a child process on a pseudo-terminal. In conjunction with the standard
Guile `expect' module, this lets you test external programs.
Also provided is @file{greg} - a Guile script to invoke the Greg test
framework in much the same way that @code{runtest} is used in DejaGNU.
All tests have the same output format (enforced by the @code{greg-testcase}
procedure).
Greg's output is designed to be both readable and readily parsed by other
software, so that it can be used as input to customised testing processes.
Greg provides most of the functionality of DejaGNU but is rather simpler.
It omits specific support for cross-platform/remote testing since this is
really rather trivial to add where required and tends to vary from site to
site so much that an attempt at a generic solution is pretty pointless.
What Greg does do, is provide hooks to let you easily introduce site
specific code for handling those sorts of situations.
The current version of Greg can normally be found on GNU ftp sites,
with documentation online at
@uref{http://www.gnu.org/software/greg/gregdoc.html}
or, for the bleeding edge - availably by anonymous cvs as part of the
GNUstep-Guile package in the GNUstep project -
CVSROOT=":pserver:anoncvs@@cvs.net-community.com:/gnustep"@*
export CVSROOT@*
cvs login (password is `anoncvs')@*
cvs -z3 checkout guile@*
@menu
* Running Tests:: How to run a Greg testsuite
* Sample Test:: A trivial example of a testcase
* Design Goals:: Why Greg does what it does
* Posix:: Greg conforms to POSIX 1003.3
@end menu
@node Running Tests
@section How to run a Greg testsuite
@cindex existing tests, running
@cindex running tests
@cindex tests, running
@kindex make check
To run tests from an existing collection, try running
@example
make check
@end example
@cindex @code{check} makefile target
If the @code{check} target exists, it usually saves you some
trouble---for instance, it can set up any auxiliary programs or other
files needed by the tests.
Alternatively, if you are in the top-level source directory of an existing
testsuite (ie. there are subdirectories containing files with a @file{.scm}
extension), you can get the @file{greg} script to test all the tools in the
directory by typing -
@example
greg
@end example
Finally, if you just want to run the tests that are in a specific file
(or files), you can get the @file{greg} script to run them simply by
listing the files on the command line.
@example
greg a-file-to-run another-file-to-run
or, for verbose output -
greg --verbose a-file-to-run another-file-to-run
@end example
If you have a test suite that is intended to be used for `embedded' testing -
You need to start the application to be tested, gain access to it's Guile
command line (or other guile interface) and enter the commands -
@example
(use-modules (ice-9 greg))
(greg-test-all)
@end example
@node Sample Test
@section A trivial example of a testcase
@cindex example
Each Greg test is a Guile script; the tests vary widely in
complexity, depending on the nature of the tool and the feature tested.
@cartouche
@smallexample
;
; GNUstep-guile interface library test
;
; Create an object using the NSString [stringWithCString:] method and
; check that the resulting object is of the correct class.
;
(greg-testcase "The `stringWithCString:' method creates an NSString object" #t
(lambda ()
(define obj ([] "NSString" stringWithCString: "Hello world"))
(gstep-bool ([] obj isKindOfClass: ([] "NSString" class)))
))
@end smallexample
@end cartouche
Though brief, this example is a complete test. It illustrates some of
the main features of Greg test scripts:
@itemize @bullet
@item
The test case does not deal with accessing the library - the Guile interpreter
in use is linked with the library. If you were testing an external application
which needed to be started up as a child process running on a pseudo-terminal,
the process startup would have been performed in a separate @file{begin.grg}
script.
@item
You use the Guile programming language to write the script and, in this case,
the builtin macros and procedures (such as @code{gstep-bool} and @code{[]}
provided by the GNUstep-Guile library.
@item
Test cases use the procedure @code{greg-testcase} to record the test outcome.
You pass this procedure a string (`assertion') describing the testcase,
a boolean specifying the expected outcome, and a `thunk' (parameterless
procedure) that performs the actual test.@*
The `thunk' is normally a `lambda expression' - you can look these up in the
Guile/Scheme documentation - or just follow the example.
@item
As much code as possible is placed inside the `thunk' passed to
@code{greg-testcase} so that, in the event of an error, the @code{greg-testcase}
procedure can trap it and report an unresolved test.@*
The code -
@code{(define obj ([] "NSString" stringWithCString: "Hello world"))} could have
appeared outside the testcase, but that would have been less safe.
@end itemize
Here is the same example in a slightly different form - using the
@code{greg-expect-pass} macro -
@cartouche
@smallexample
;
; GNUstep-guile interface library test
;
; Create an object using the NSString [stringWithCString:] method and
; check that the resulting object is of the correct class.
;
(greg-expect-pass "The `stringWithCString:' method creates an NSString object"
(define obj ([] "NSString" stringWithCString: "Hello world"))
(gstep-bool ([] obj isKindOfClass: ([] "NSString" class)))
)
@end smallexample
@end cartouche
@node Design Goals
@section Why Greg does what it does
@cindex design goals
Greg was written to support regression testing for the
@uref{http://www.gnustep.org/,GNUstep} libraries.
It was inspired by an earlier test framework (by Ovidiu Predescu) that used
DejaGNU along with a `driver' program (to make the calls to the library) and
a suite of TcL scripts to control the driver.
There were three main problems (inherent in the nature of DejaGNU) with that
approach -
@itemize @bullet
@item Writing/Debugging tests was a pain - errors in the TcL code of a testcase
could all too easily interfere with the TcL routines that interfaced to the
driver program.@*
Solution - use another language where errors in testcases are localised.
@item Writing/Debugging tests was a pain - you needed to attach gdb to the
driver program that was running as a child of the DejaGNU process in order
to debug errors in the driver/library under test.@*
Solution - `embedded testing' make the test framework and the code under
test part of the process, so it's easy to run the whole thing under gdb.
@item The driver program and TcL scripts required quite a bit of support
when adding new tests.@*
Solution - `embedded testing' we needed to remove the intermediary layers
and test as directly as possible.
@end itemize
So, something different was required, a test framework in a safer, simpler
language that made it easy to create thin interfaces to libraries, so
simplifying the task of producing testcases.
Of course, the good points of DejaGNU needed to be retained - clear
output, Posix compliance, the ability to test separate programs as well
as libraries.
A couple of additional goals seemed worthwhile -
@itemize @bullet
@item Make the test framework flexible so that applications could easily
use it for self-testing.
@item Give the framework a similar structure to, and a user interface like
that of DejaGNU, so that people with a background in testing could easily
adopt it.
@end itemize
Hopefully, Greg meets all its goals.
@node Posix
@section A POSIX conforming test framework
@cindex POSIX conformance
@cindex standard conformance: POSIX 1003.3
This section is copied almost directly from the DejaGNU documentation
with minor modifications -
Greg is believed to conform to the @sc{posix} standard for test frameworks.
@sc{posix} standard 1003.3 defines what a testing framework needs to
provide, in order to permit the creation of @sc{posix} conformance
test suites. This standard is primarily oriented to running @sc{posix}
conformance tests, but its requirements also support testing of features
not related to @sc{posix} conformance.
The @sc{posix} documentation refers to @dfn{assertions}. An assertion
is a description of behavior. For example, if a standard says ``The sun
shall shine'', a corresponding assertion might be ``The sun is
shining.'' A test based on this assertion would pass or fail depending
on whether it is daytime or nighttime. It is important to note that the
standard being tested is never 1003.3; the standard being tested is some
other standard, for which the assertions were written.
As there is no test suite to test @dfn{testing frameworks} for
@sc{posix} 1003.3 conformance, verifying conformance to this standard is
done by repeatedly reading the standard and experimenting. One of the
main things 1003.3 does specify is the set of allowed output messages,
and their definitions. Four messages are supported for a required
feature of @sc{posix} conforming systems, and a fifth for a conditional
feature. Greg supports the use of all five output messages; in this
sense a test suite that uses exactly these messages can be considered
@sc{posix} conforming. These definitions specify the output of a test
case:
@ftable @code
@cindex success, POSIX definition
@item PASS
A test has succeeded. That is, it demonstrated that the assertion is true.
@cindex UPASS, avoiding for POSIX
@item UPASS
@sc{posix} 1003.3 does not incorporate the notion of unexpected passes,
so for strict @sc{posix}, @code{PASS}, instead of @code{UPASS}, is returned
for test cases which were not expected to pass but did. This means that
@code{PASS} is in some sense more ambiguous than if @code{UPASS} is also
used.
@item FAIL
@cindex failure, POSIX definition
A test @emph{has} produced the bug it was intended to capture. That is,
it has demonstrated that the assertion is false. The @code{FAIL}
message is based on the test case only. Other messages are used to
indicate a failure of the framework.
@cindex XFAIL, avoiding for POSIX
@item XFAIL
@sc{posix} 1003.3 does not incorporate the notion of expected failures,
so for strict @sc{posix}, @code{FAIL}, instead of @code{XFAIL}, is returned
for test cases which were expected to fail and did not. This means that
@code{FAIL} is in some sense more ambiguous than if @code{XFAIL} is also
used.
@item UNRESOLVED
@cindex ambiguity, required for POSIX
A test produced indeterminate results. Usually, this means the test
executed in an unexpected fashion; this outcome requires that a human
being go over results, to determine if the test should have passed or
failed. This message is also used for any test that requires human
intervention because it is beyond the abilities of the testing
framework. Any unresolved test should resolved to @code{PASS} or
@code{FAIL} before a test run can be considered finished.
Note that for @sc{posix}, each assertion must produce a test result
code. If the test isn't actually run, it must produce @code{UNRESOLVED}
rather than just leaving that test out of the output. With Greg this is
not a problem - any unexpected termination of a @code{greg-testcase}
procedure will produce @code{UNRESOLVED}.
Here are some of the ways a test may wind up @code{UNRESOLVED}:
@itemize @bullet
@item
A test's execution is interrupted.
@item
A test does not produce a clear result. This is usually because there
was a Guile exception thrown while processing the test.
This usually requires a human being to examine the output to
determine what really happened---and to improve the test case.
@item
A test depends on a previous test, which fails.
@item
The test was set up incorrectly and failed to produce a boolean value.
@end itemize
@item UNTESTED
A test was not run. This is a placeholder, used when there is no
real test case yet.
@end ftable
@noindent
The only remaining output message left is intended to test features that
are specified by the applicable @sc{posix} standard as conditional:
@ftable @code
@item UNSUPPORTED
There is no support for the tested case. This may mean that a
conditional feature of an operating system, or of a compiler, is not
implemented.
@end ftable
Greg uses the same output procedures to produce these messages for
all test suites, and these procedures are already known to conform to
@sc{posix} 1003.3. For a Greg test suite to conform to @sc{posix}
1003.3, you must set the variable @code{greg-posix} to be true (or run
the @file{greg} command with the @code{--posix} flag).
Doing this will ensure that non-posix extensions are not used.
@node Installation, Invoking Greg, Overview, Top
@chapter Installing Greg
@cindex installing
@subsection Requirements
@cindex requirements
Greg needs to have Guile installed. It should work with Guile-1.3
or later.
You need to have the @file{guile} program in your path in
order for the installation process to determine the proper locations
for things.
You can get Guile from any GNU ftp site.
The current version of Greg can normally be found on GNU ftp sites,
with documentation online at
@uref{http://www.gnu.org/software/greg/gregdoc.html}
or, for the bleeding edge - availably by anonymous cvs from the
GNUstep project (@url{http://www.gnustep.org/}) -
CVSROOT=":pserver:anoncvs@@cvs.net-community.com:/gnustep"@*
export CVSROOT@*
cvs login (password is `anoncvs')@*
cvs -z3 checkout guile/Greg@*
@subsection Building
@cindex building Greg
To build Greg -
Type @code{./configure} in the main Greg directory to configure for
your system.
Once configuration is complete, go into the `Library' subdirectory and
type @code{make install} to build and install things.
You should end up with -
A dynamic library, which can be dynamically linked into Guile with
@code{
(if (not (feature? 'greg-pty))
(dynamic-call "scm_init_greg" (dynamic-link "libgreg.so")))
}
a module defining Guile procedures and variables providing the main
test framework, which can be accessed using
@code{(use-modules (ice-9 greg))},
and a Guile script that you can use to run tests from the unix
command-line (@file{greg}).
You @emph{MUST} install Greg before you attempt to use it (or run it's
self-tests) because the Guile modules making it up must be in place in
the standard Guile directories before Greg can work.
Once Greg is installed, you can type @code{make check} in the Tests
directory to get Greg to test itself.
You can type @code{make} in the Documentation directory to build the
documentation in @emph{info}, @emph{html} and @emph{dvi} formats.
NB.@*
You must have the @file{makeinfo} program installed to build the
documentation in info format @*
You must have the @file{texi2html} program installed to build the
documentation in html format @*
You must have the @file{texi2dvi} program installed to build the
documentation in dvi format @*
@subsection Problems
@cindex problems
@cindex troubleshooting
Greg is quite simple, so there is not much to go wrong with it.
Of course, you must have a working copy of Guile installed, and you need
to make sure you ran the configure script to configure Greg for your system,
and installed Greg, but after that, most stuff should just work.
The single area where you are most likely to encounter problems is if you are
using Greg to test external programs run in a child process using the
@code{(greg-child)} procedure.
@itemize @bullet
@item
If you install Greg in a non-standard location, it's possible that guile
will not be able to find the @code{libgreg.so} dynamic library to load it.
In this case, you will need to change/set the @code{LD_LIBRARY_PATH}
environment variable (or whatever other environment variable your operating
system uses to list the places in which dynamic libraries may be found) to
include the directory in which the library has been installed.
@item
The C code for this procedure is quite system-dependent and
has only been tested on a few operating systems.
If you have problems with this, look at @file{greg.c} in the Library
directory.
@end itemize
Please attempt to make a patch to fix things on your
operating-system and send it to me - <rfm@@gnu.org> or to
<bug-gnustep@@gnu.org>
@node Invoking Greg, Writing tests, Installation, Top
@chapter Using @file{greg}
@cindex invoking
@cindex running
The Greg framework is designed to be used in two ways - as an embedded
system from within any application which is linked with the Guile library,
or stand-alone using the command-line @file{greg} driver script.
For both of these methods of usage the test cases are written the same
way and the expected output is the same.
@menu
* Output:: Expected output format from a Greg test
* Files and directories:: The layout of files and directories
* Embedded usage:: What does a Greg test case look like?
* Command-line usage:: Using the @file{greg} script.
@end menu
@node Output
@section Output
While Greg may produce more verbose output in response to various settings,
the basic output of a test run is a series of lines describing the
success/failure state of each testcase encountered, followed by a
summary of all test cases.
In `normal' mode, only unexpected results are displayed, but in `verbose'
output mode, results for @emph{all} results are displayed.
@file{greg} flags the outcome of each test case with a line consisting of
one of the following codes followed by a colon, a space, and then the
testcase description.
@table @code
@item PASS
@kindex PASS
@cindex successful test
@cindex test, successful
The most desirable outcome: the test succeeded, and was expected to
succeed.
@item UPASS
@kindex UPASS
@cindex successful test, unexpected
@cindex unexpected success
A pleasant kind of failure: a test was expected to fail, but succeeded.
This may indicate progress; inspect the test case to determine whether
you should amend it to stop expecting failure.
@item FAIL
@kindex FAIL
@cindex failing test, unexpected
@cindex test, failing
A test failed, although it was expected to succeed. This may indicate
regress; inspect the test case and the failing software to locate the bug.
@item XFAIL
@kindex XFAIL
@cindex expected failure
@cindex failing test, expected
A test failed, but it was expected to fail. This result indicates no
change in a known bug. If a test fails because the operating system
where the test runs lacks some facility required by the test, the
outcome is @code{UNSUPPORTED} instead.
@item UNRESOLVED
@kindex UNRESOLVED
@cindex test, unresolved outcome
Output from a test requires manual inspection; the test suite could not
automatically determine the outcome. For example, your tests can report
this outcome is when a test does not complete as expected.
@item UNTESTED
@kindex UNTESTED
@cindex untested properties
A test case is not yet complete, and in particular cannot yet produce a
@code{PASS} or @code{FAIL}. You can also use this outcome in dummy
``tests'' that note explicitly the absence of a real test case
for a particular property.
@item UNSUPPORTED
@kindex UNSUPPORTED
@cindex unsupported test
@cindex test, unsupported
A test depends on a conditionally available feature that does not exist
(in the configured testing environment). For example, you can use this
outcome to report on a test case that does not work on a particular
target because its operating system support does not include a required
subroutine.
@end table
@node Files and directories
@section Files and directories
@cindex files and directories
@cindex directories and files
A Greg test run expects to find files and directories in a certain layout
(modeled on that used by DejaGNU) - though it is possible to override this
DejaGNU compatibility feature and simply run the tests in a list of files.
The test source directory (normally your current directory) is expected to
contain one or more @dfn{tool directories}. Each tool directory
should contain one or more @dfn{test scripts}. In fact any file in a
tool directory which has a @file{.scm} extension is assumed to be
a Guile test script.
When a normal Greg test run is done, Greg goes through each tool
directory in turn and loads each test script in turn.
You may set the Guile variable @code{greg-tools} or use the @code{--tool ...}
command-line option to specify a list of tools directories to use rather than
assuming that @emph{all} subdirectories are tool directories.
If you do this, the tools are tested in the order in which they appear in
the list rather than the default order (ASCII sorted by name).
You may set the Guile variable @code{greg-files} or use the @code{--file ...}
command-line option to specify a list of file names to use rather than
assuming that @emph{all} @file{.scm} files in each tool directory are
test scripts.
If you do this, the files are loaded in the order in which they appear in
the list.
You may omit the @file{.scm} extension from filenames and Greg will supply
it for you if necessary.
You may set the Guile variable @code{greg-paths} to specify a list
of test files to be run directly, or simply list the files to be run
on the command-line.
Doing this overrides the @code{greg-tools} and @code{greg-files} variables, and
simply runs the files you list in the order you list them.
@cindex begin.grg
@cindex end.grg
@cindex cleanup
@cindex initialisation
As a (minor) complication to this simple layout, Greg permits the use of
@file{begin.grg} and @file{end.grg} scripts in both the main source directory
and in each tool directory. These scripts permit you to add any
initialisation and cleanup code you want. Typically (for non-embedded
testing) you would use a @file{begin.grg} script to start the application
to be tested.
If @file{begin.grg} exists in the main source directory, it will be loaded
before any tools are tested.
If @file{end.grg} exists in the main source directory, it will be loaded
after all the tools are tested.
If @file{begin.grg} exists in a tool directory, it will be loaded
before any test scripts in that directory are loaded.
If @file{end.grg} exists in a tool directory, it will be loaded
after all the test scripts in that directory are loaded.
NB. Even when you use the @code{greg-paths} variable to run one or more test
files directly, the @file{begin.grg} and @file{end.grg} files in your current
directory will be loaded.
@node Embedded usage
@section Embedded usage
@cindex embedded usage
@cindex module usage
Greg is designed primarily for embedded usage - any application that uses
Guile as it's scripting language should be able to use Greg to test itself.
To this end - Greg provides a Guile module containing definitions of
various procedures (used to run tests) and variables (used to modify the
behavior of a test run).
Before trying to use any part of Greg, You need to load the Greg module with
@code{(use-modules (ice-9 greg))}
@cindex run test procedure
The main procedure to run Greg tests is @code{(greg-test-run)}.
You can use this to run tests in much the same way as the @file{greg} script
is used to run tests from the command-line. The behavior of this
procedure is modified by setting the following top-level variables -
@cindex variables
@itemize @bullet
@item greg-debug
@cindex greg-debug
@cindex debug variable
The @code{greg-debug} variable is a boolean used to determine whether to
output extra debug information. The debug information is written to a
@file{.dbg} file. Test scripts may use the @code{greg-dlog} procedure to
output debug information.
@item greg-files
@cindex greg-files
@cindex files variable
The @code{greg-files} variable is a list of strings used to specify the names
of the test scripts to be run. If this list is empty, all the files with the
@file{.scm} extension in a tool subdirectory are loaded. If the list
contains names, the named scripts are loaded in the order in which they occur
in the list rather than the default order (ASCII sorted by name).@*
The exact names of scripts in this list will be used to match filenames,
and if no match is found a @file{.scm} extension will be added automatically
by Greg and a re-match attempted.
@item greg-obj-dir
@cindex greg-obj-dir
@cindex obj-dir variable
The @code{greg-obj-dir} variable is a string naming the directory in which
Greg expects to find binaries to be run as child processes.@*
By default, this is the current directory (@code{.}).
@item greg-out-dir
@cindex greg-out-dir
@cindex out-dir variable
The @code{greg-out-dir} variable is a string naming the directory in which
Greg will generate log and debug output files.@*
By default, this is the current directory (@code{.}).
@item greg-paths
@cindex greg-paths
@cindex paths variable
The @code{greg-paths} variable is a list of pathname of files to be run.
This is used where you don't want to use the normal tools hierarchy
and need to test specific files explicitly. The filenames are used exactly
as given. If you use this variable, you should set the @code{greg-tools}
variable to an empty list.
@item greg-posix
@cindex greg-posix
@cindex posix variable
The @code{greg-posix} variable is a boolean used to determine whether to
produce strict posix output, or permit non-posix extensions.
@item greg-src-dir
@cindex greg-src-dir
@cindex src-dir variable
The @code{greg-src-dir} variable is a string naming the directory in which
Greg will look for tool subdirectories and where it expects to find
the main initialisation and cleanup scripts
(@file{begin.grg} and @file{end.grg}) for a test run.@*
By default, this is the current directory (@code{.}).
@item greg-tools
@cindex greg-tools
@cindex tools variable
The @code{greg-tools} variable is a list of strings used to specify the names
of the tools to be tested (ie the names of the subdirectories of the main
source directory). If this list is empty, all the subdirectories of the
main source directory (normally @code{.}) are assumed to be valid tool
directories. If the list contains names, the named subdirectories are used
in the order in which they occur in the list.@*
@item greg-verbose
@cindex greg-verbose
@cindex verbose variable
The @code{greg-verbose} variable is a number used to determine the level of
detail produced in the log output. At zero (the default), only minimal
information is output (unexpected testcase results and a summary of the
number of testcases passed). At one, the result of each testcase is reported
and a more detailed summary is output. At two, entry and exit of tool
directories and test files is reported.
@end itemize
@node Command-line usage
@section Command-line usage
@cindex @file{greg} description
@cindex Greg test driver
@file{greg} is the executable test driver for Greg. This is a Guile script
that you can use to run tests from the command line.
The command-line options that you can pass to @file{greg} are listed below.
@cindex exit code from @file{greg}
@cindex @file{greg} exit code
@file{greg} returns an exit code of @code{1} if any test
has an unexpected result; otherwise (if all tests pass or fail as
expected) it returns @code{0} as the exit code.
This is the full set of command line options that @file{greg}
recognizes.
@cindex command line options
@cindex options
@cindex @file{greg} option list
@cindex option list, @file{greg}
@smallexample
greg --tool @var{tool} @dots{}
[ --debug ]
[ --file @var{script} @dots{} ]
[ --help ]
[ --objdir @var{path} ]
[ --outdir @var{path} ]
[ --posix ]
[ --srcdir @var{path} ]
[ -v | --verbose ] [ -V | --version ]
[ files to run ]
@end smallexample
@table @code
@item --tool @var{tool} @dots{}
@cindex selecting tests for a tool
@cindex @code{--tool} (@file{greg} option)
@var{tool} specifies what set of tests to run. It provides a list of
subdirectories (each corresponding to a tool) in which test scripts
can be found.
Initialisation code (including executable tool startup) for each tool may
be placed in @file{begin.grg} in the appropriate tool subdirectory.
Cleanup code may be placed in @file{end.grg}.
For example, including @code{--tool gcc} on the @file{greg} command
line runs tests from the @code{gcc} subdirectory.
The order in which the tools are tested will be the same as the order in
which the tool names occur on the command line.
@item --file @var{script} @dots{}
@cindex selecting a range of tests
@cindex tests, running specifically
@cindex naming tests to run
Specify the names of testsuites to run.
By default, @file{greg} runs all tests for the tool, but you can
restrict it to particular testsuites by giving the names of the @file{.scm}
Guile scripts that control them. You do not need to supply the
@file{.scm} file extension - it will be assumed.
@var{testsuite} may not include path information; use plain filenames.
The order in which the test scripts are run will be the same as the order in
which the script names occur on the command line.
@item --debug
@cindex @code{--debug} (@file{greg} option)
@cindex debug log for test cases
@cindex test cases, debug log
@cindex @file{tests.dbg} file
Turns on the @code{expect} internal debugging output. Debugging output
is displayed as part of the @file{greg} output, and logged to a file
called @file{tests.dbg}. The extra debugging output does @emph{not}
normally appear on standard output.
@item --help
@cindex @code{--help} (@file{greg} option)
@cindex help with @file{greg}
@cindex @file{greg}, listing options
Prints out a short summary of the @file{greg} options, then exits
(even if you also specify other options).
@item --objdir @var{path}
@cindex @code{--objdir} (@file{greg} option)
@cindex object directory
@cindex test programs, auxiliary
@cindex auxiliary test programs
Use @var{path} as the top directory containing any auxiliary compiled
test code. This defaults to @file{.}. Use this option to locate
pre-compiled test code. You can normally prepare any auxiliary files
needed with @code{make}.
@item --outdir @var{path}
@cindex output directory
@cindex @code{--outdir} (@file{greg} option)
@cindex log files, where to write
Write output logs in directory @var{path}. The default is @code{.}, the
directory where you start @file{greg}. This option affects only the
log and the debug files @file{@var{tool}.log} and
@file{@var{tool}.dbg}.
@item --srcdir @var{path}
@cindex source directory
@cindex @code{--srcdir} (@file{greg} option)
Use @var{path} as the top directory for test scripts to run.
@file{greg} looks in this directory for any subdirectory whose name
matches any toolname (specified with @code{--tool}).
@item --verbose
@itemx -v
@cindex @code{--verbose} (@file{greg} option)
@cindex @code{-v} (@file{greg} option)
@cindex turning on output
@cindex output, additional
By default, @file{greg} shows only the output of tests that produce unexpected
results; that is, tests with status @code{FAIL} (unexpected failure),
@code{UPASS} (unexpected success), or @code{ERROR} (a severe error in the test
case itself).
Specifying @code{--verbose} to see output for tests with status @code{PASS}
(success, as expected) and @code{XFAIL} (failure, as expected). It also
causes a more detailed summary to be displayed.@*
Specifying @code{--verbose} more than once causes more detail to be displayed.
@item --version
@itemx -V
@cindex @code{-V} (@file{greg} option)
@cindex @code{--version} (@file{greg} option)
@cindex version numbers
Prints out the version numbers of Greg, and Guile, then
exits without running any tests.
@end table
@node Writing tests, Index, Invoking Greg, Top
@chapter Writing tests
@cindex writing tests
@cindex testcase
@subsection getting started
@cindex getting started writing a test
The simplest way to get started is to write a file (say @samp{myTests}) and
type @code{greg --verbose myTests} to run the tests in it.
Your file might contain code like -
@cartouche
@smallexample
; A simple test that basic arithmetic works
(greg-expect-pass "one plus one is two"
(eq? (+ 1 1) 2)
)
(greg-expect-pass "one minus one is zero"
(eq? (- 1 1) 0)
)
@end smallexample
@end cartouche
And would produce output like -
@cartouche
@smallexample
PASS: one plus one is two
PASS: one minus one is zero
# of testcases attempted 2
# of expected passes 2
# of expected failures 0
# of unexpected passes 0
# of unexpected failures 0
# of unresolved testcases 0
# of unsupported testcases 0
# of untested testcases 0
@end smallexample
@end cartouche
@subsection types of testsuite
There are three types of situation where Greg may be used as a test
framework -
@itemize @bullet
@item
Embedded testing @*
In this case, you have an application that uses Guile as it's scripting
language - so you have provided an interface between the internals of
your program and the Guile interpreter.
Your testcases can use that interface to test your program directly, and
you run the tests by loading the Greg framework and then doing
@code{(greg-test-run)}
A testsuite for @dfn{embedded} use does not normally need @file{begin.grg}
or @file{end.grg} since the application to be tested will be running the
tests on itself - so it doesn't need to control child processes.
@item
Library testing @*
When you want to test a library, you need to write a tiny program to
start up the Guile command-line interpreter with the library to be tested
inked in to the program via a via a set of stubs. The resulting program
can then be used to run tests on the directly, just like normal embedded
testing.
There is a tool called G-Wrap (by Chris Lee), which will
generate stubs to import your library functions into Guile. The home
page of this tool is at @url{http://www.cs.cmu.edu/~chrislee/g-wrap/}
For testing Objective-C libraries, you should use the GNUstep-Guile
interface library (documented online at
@url{http://www.tiptree.demon.co.uk/gstep/guile/gstep_guile_toc.html}) which
provides a fairly complete interface between the Guile and Objective-C
languages (using the GNU objective-C runtime).
You can get hold of this as part of the GNUstep core package (from any GNU
ftp site) or, for the bleeding edge - by anonymous cvs as the
GNUstep-Guile package in the GNUstep project -
@smallexample
CVSROOT=":pserver:anoncvs@@cvs.net-community.com:/gnustep"@*
export CVSROOT@*
cvs login (password is `anoncvs')@*
cvs -z3 checkout guile@*
@end smallexample
@item
External testing@*
When you want to test an external program, you use Greg in much
the same way that you would use DejaGNU. That is - you run the
program to be tested as a separate child process and your test cases
consist of Guile code to send commands to the child process and wait
for responses from it.
The child process is normally started by a @file{begin.grg} script
using the @code{(greg-child)} procedure, though it is of course possible
to start and stop child processes in each test script.
@end itemize
@subsection testsuite file layout
A @dfn{testsuite} for a tool is a directory containing one or more
test script files and (optionally) @file{begin.grg} and @file{end.grg}
files to handle initialisation and cleanup.
Each script file has a @file{.scm} extension and contains Guile (Scheme)
code, but you do not need to know much about the Guile programming
language to write most tests.
A script file will contain one or more testcases - each of which
constitutes a test of a single well defined feature of the tool
that the script is meant to test. A testcase is always written
using the @code{greg-testcase} procedure, though this procedure could
be invoked by a convenience macro.
@subsection greg-expect-pass
@cindex greg-expect-pass
The @code{greg-expect-pass} macro is a shorthand method of writing
the most usual sort of testcase - where a fragment of Guile code is
run and is expected to return a @code{true} result.
It passes an assertion and a fragment of Guile
code that performs a test to the @code{greg-testcase} procedure -
@cartouche
@smallexample
; A simple test that basic arithmetic works
(greg-expect-pass "one plus one is two" (eq? (+ 1 1) 2))
is equivalent to -
(greg-testcase "one plus one is two" #t
(lambda ()
(eq? (+ 1 1) 2)
)
)
@end smallexample
@end cartouche
@subsection greg-expect-fail
@cindex greg-expect-fail
The @code{greg-expect-fail} macro is a shorthand method of writing a testcase
to confirm that a known bug is still present in the code being tested. Once
the bug is fixed, it would be altered to be a @code{greg-expect-pass} testcase.
@cartouche
@smallexample
; A test that basic arithmetic DOESN'T work!
(greg-expect-fail "one plus one is two" (eq? (+ 1 1) 2))
is equivalent to -
(greg-testcase "one plus one is two" #f
(lambda ()
(eq? (+ 1 1) 2)
)
)
@end smallexample
@end cartouche
@subsection greg-testcase
@cindex greg-testcase
The @code{greg-testcase} procedure takes three arguments -
@itemize @bullet
@item The assertion@*
This is a string describing the test to be performed. It should take the
form of an assertion that is expected to hold true.
@item The expected outcome@*
This is a boolean value - @code{#t} if the test is expected to return
@code{#t} (ie. the assertion is expected to be proved correct), and
@code{#f} if the test is expected to return @code{#f} (ie. the assertion
is expected to be proved to be incorrect).
@item The test itself@*
This is a @dfn{thunk} (a Guile procedure that takes no arguments) and is
normally a @dfn{lambda expression}. The return value of this procedure
should be @code{#t} if the assertion is proved correct, @code{#f} if it
is proved incorrect.
@end itemize
The Guile programming language permits the thunk to return in four
ways -
@itemize @bullet
@item Return the boolean value - @code{#t}@*
In which case, the test has succeeded, the assertion has proved correct,
and Greg will record either an expected success (PASS) or an unexpected
success (UPASS) depending on the expected outcome passed to
@code{greg-testcase}.
@item Return the boolean value - @code{#f}@*
In which case, the test has failed, the assertion has proved incorrect,
and Greg will record either an expected failure (XFAIL) or an unexpected
failure (FAIL) depending on the expected outcome passed to
@code{greg-testcase}.
@item Return a non-boolean value@*
In which case, the test is unresolved (UNRESOLVED).
@item Throw an exception@*
In which case Greg's behavior depends on the exception thrown -
@itemize @bullet
@item 'pass@*
Greg acts as if the test had returned the boolean @code{#t}
@item 'fail@*
Greg acts as if the test had returned the boolean @code{#f}
@item 'unsupported@*
Greg reports this case as unsupported (UNSUPPORTED)
@item 'untested@*
Greg reports this case as untested (UNTESTED)
@item (quit)@*
If the script calls the @code{(quit)} primitive - an exception is raised.
In this special case, the test is reported as unresolved (UNRESOLVED) and
no further tests are executed. The testrun is terminated.
@item any other exception@*
Greg reports this case as unresolved (UNRESOLVED)
@end itemize
@end itemize
As there are no other ways in which the thunk may be exited, it is
impossible for a testcase to produce a result that doesn't fit into the
framework (unless your testcase manages either to crash Guile or enter an
infinite loop - in which case you won't get @emph{any} output).
The value returned by the @code{greg-testcase} procedure is a boolean -
@code{#t} if the test resulted in an expected pass, @code{#f} otherwise.@*
You can use this return value to make the execution of subsequent testcases
dependent on the success of an earlier testcase.
@cartouche
@smallexample
;
; A testcase to check an instance of numeric addition
;
(greg-testcase "One plus One is two" #t
(lambda ()
(eq? (+ 1 1 ) 2)
))
;
; The above testcase will generate output -
; `PASS: One plus One is two'
;
@end smallexample
@end cartouche
The system provides hooks for general purpose procedures that are
automatically called immediately before and after each testcase is
executed. These procedures can be used to perform additional
logging or other housekeeping functions. -
@itemize @bullet
@item greg-case-begin
@cindex greg-case-begin
The @code{greg-case-begin} variable is a hook for a procedure to be
executed immediately before each testcase. It is initially defined
to be @code{(lambda () ())} which does nothing except return an
unused empty list. You can replace this with a procedure of your choice,
and use it to do any pre-test logging etc. When this procedure is called,
@code{greg-case-name} will contain the name of the current testcase.
@item greg-case-end
@cindex greg-case-end
The @code{greg-case-end} variable is a hook for a procedure to be
executed immediately before each testcase. It is initially defined
to be @code{(lambda (result) ())} which does nothing except return an
unused empty list. You can replace this with a procedure of your choice.
When this procedure is called, @code{greg-case-name} will contain the
name of the current testcase.
@end itemize
While a testcase is executing (or in the @code{greg-case-begin} or
@code{greg-case-end} procedures) there are a number of public procedures
that may be used to obtain information about the system -
@itemize @bullet
@item greg-casename
@cindex greg-casename
This procedure returns the name of the current testcase. It is valid during
execution of the testcase and during the pre and post test procedures.
@item greg-directory
@cindex greg-directory
The @code{greg-directory} procedure takes no arguments and simply returns
a string containing the absolute path name of the directory in which the
current testscript is located. This value can be used where a testscript
wishes to load data from local files in order to support a test.
The string returned by this method includes a trailing slash character,
so to refer to a file located in the same directory as the testscript
simply by appending the fielname.
eg @code{(string-append (greg-directory) "foo")}
@item greg-filename
@cindex greg-filename
This procedure returns the name of the current test file. It is valid during
execution of the testcase and during the pre and post test procedures.
@item greg-toolname
@cindex greg-toolname
This procedure returns the name of the current test tool. It is valid during
execution of the testcase and during the pre and post test procedures.
@end itemize
@subsection Multiple testcases
@cindex multiple testcases
It is normal to have more than one testcase in a file and this
produces no problems - the only thing to watch out for is communicating
information between testcases -
The scope of variables defined in the thunk in a @code{greg-testcase}
procedure call is that thunk - the variable will @emph{not} be
visible to the next testcase.
So - to pass information from one testcase to the next it is necessary to
define variables that can be seen in each testcase. The way to do this
is normally to define these variables at the start of the file and then
use the @code{set!} procedure within each testcase to set a value for a
variable to be passed to the next testcase.
@cartouche
@smallexample
(define arith-ok #f)
;
; A testcase to check an instance of numeric addition
;
(greg-testcase "One plus One is two" #t
(lambda ()
(if (eq? (+ 1 1 ) 2)
(begin (set! arith-ok #t) #t)
#f)
))
;
; A testcase to check arithmetic - only supported if we have addition.
;
(greg-testcase "X multiplied by 2 is X plus X" #t
(lambda ()
(if arith-ok
(eq? (+ 1 1) (* 1 2))
(throw 'unsupported))
))
@end smallexample
@end cartouche
Of course, if (as above) the only information you want to pass from a testcase
is whether the test succeeded or not, you can use the return value from the
@code{greg-testcase} procedure directly -
@cartouche
@smallexample
(if
(greg-testcase "One plus One is two" #t
(lambda ()
(eq? (+ 1 1 ) 2)
)
)
(greg-testcase "X multiplied by 2 is X plus X" #t
(lambda ()
(eq? (+ 1 1) (* 1 2))
)
)
(greg-dlog "Arithmetic operations not supported\n")
)
@end smallexample
@end cartouche
@subsection External tests
@cindex external tests
When Greg is used to test an external application, you usually want to run
that application as a child process on a pseudo-terminal and handle tests
sending a sequence of commands to the application and reading anticipated
output from the application.
@subsubsection Starting a child process
@cindex starting a child process
@cindex greg-child
Greg provides the @code{greg-child} procedure to start up a child process
on a pseudo-terminal. You would usually call this procedure in the
@file{begin.grg} file in your tool directory, but you could call it
at the start of each script to get a new child process for each script.
The @code{greg-child} procedure expects one argument (the name of
the program to be executed) followed by any number of additional
arguments which are the arguments to be passed to the child process.@*
If the program name does not begin with a slash, Greg will look in the
directory specified in @code{greg-obj-dir} to find it (by default the
current directory).@*
If you want your normal PATH to be searched for the program, you should
use -@*
@code{(greg-child "/bin/sh" "-c" "foo args")}@*
to get the shell to execute program @file{foo} with arguments @code{args}.
The @code{greg-child} procedure will automatically close down the I/O
channels to any process previously started and wait for that process to
die. If the old child process is badly behaved and will not die, this
can result in Greg hanging - in this case you may need to explicitly
kill the old child by another method before starting the new child
process (this is one of the uses of the @file{end.grg} script).
As a special case, you can use an empty string as the program name -
if you do this, another copy of the guile process will be created as
a child and the value returned by @code{greg-child} in the child process
will be a list containing the single number 0 (in the parent it will be
a list containing the input port, output port and process id of the child).
You can use this information to get the child copy of the process to
be the program under test. This is useful for embedded testing where you
want to test the I/O capabilities of the program.
NB. The @code{greg-child} procedure is implemented on top of the new
primitive @code{pty-child}. This primitive is used to create a new child
process on the end of a pseudo-terminal. Arguments and return values are
as for @code{greg-child}.
@itemize @bullet
@item
When successfully starting a new program a three item list is returned in the
parent - input port to read from the child, output port to write to the child,
process id of child.
@item
When returning in a child process because the program name was an empty string
a list of one item (zero) is returned.
@item
After a failure to create a child process, an empty list is returned.
@end itemize
@subsubsection Sending to a child process
@cindex sending to a child process
@cindex greg-send
The @code{greg-send} procedure is provided to send instructions to a
child process. This procedure takes one or more string arguments and
sends them to the child process (if one exists).
@subsubsection Reading from a child process
@cindex reading from a child process
@cindex greg-recv
The @code{greg-recv} macro is used to read data from a child process.
This procedure actually provides a simple front-end to the @code{expect}
module. You can use the @code{expect} module facilities directly if you
want more control than is offered by @code{greg-recv}.
The @code{greg-recv} macro expects one or more lists as arguments -
each list containing a string (a pattern to match)
and a result to return on a successful match. The value returned by
@code{greg-recv} is the result for the pattern actually matched.@*
If no pattern is matched within the timeout period then an exception
is raised, causing the testcase to return a FAIL result
(unless you use @code{(set! expect-timeout-proc xxx)} to
override Gregs timeout handler.@*
If no pattern is matched before an end-of-file is read, then an exception
is raised, causing the testcase to return a FAIL result
(unless you use @code{(set! expect-eof-proc xxx)} to
override Gregs end-of-file handler.@*
NB. The @code{expect-timeout-proc} and @code{expect-eof-proc} are saved
when a tool is tarted, and restored when it ends ... so if you want to
make changes to these procedures in multiple tools, you must do so in the
@code{begin.grg} and @code{end.grg} files for each tool.
In addition to setting up the above expect procedures, Greg also sets
the @code{expect-timeout} variable to a 15 second timeout, and sets the
@code{expect-char-proc} to be @code{greg-dlog} so that data read from
the child process is logged as debug output by default. You can of
course override this behavior in @code{begin.grg}.
The pattern matching is done with extended regular expressions,
usually with input split into lines so that a carat (^) at the start of an
expression matches the start of a line, and a dollar ($) matches the end
of a line.@*
This is convenient for testing programs that produce lines of output in
an expected format, as you can easily match the start and end of an
output line.
If you want to change this behavior to permit multi-line patterns and to
have the carat and dollar match the start of input and end of input
respectively, then you can use -
@cartouche
@smallexample
(set! expect-strings-compile-flags regexp/extended)
(set! expect-strings-exec-flags 0)
@end smallexample
@end cartouche
This pattern matching behavior is occasionally useful when you are
testing a program that produces output without clearly recognisable
individual lines.
NB. greg does not save and restore these values, so a change to them
effect all tools being tested until you change them back.
@subsubsection A complete external test
@cindex a complete external test
@cartouche
@example
;
; Run an interactive shell as a child process
;
(greg-child "/bin/sh" "-i")
;
; Set the shell prompt
;
(greg-send "PS1='% '\n")
;
; Now test that the shell echoes what we expect.
; If we have a timeout or an eof, we will get a failure result.
;
(greg-testcase "echo 'hello'" #t
(lambda ()
(greg-send "echo hello\n") ; Get it to send us something
(expect-strings
("hello\r\n% " #t)
)
)
)
@end example
@end cartouche
@node Index, , Writing tests, Top
@unnumbered Index
@printindex cp
@contents
@ifhtml
<P>
Return to <A HREF="/home.html">GNU's home page</A>.
<P>
Please send FSF & GNU inquiries & questions to
<A HREF="mailto:gnu@gnu.org"><EM>gnu@gnu.org</EM></A>.
There are also <A HREF="/home.html#ContactInfo">other ways to
contact</A> the FSF.
<P>
Please send comments on these web pages to
<A HREF="mailto:webmasters@gnu.org"><EM>webmasters@gnu.org</EM></A>,
send other questions to
<A HREF="mailto:gnu@gnu.org"><EM>gnu@gnu.org</EM></A>.
<P>
Copyright (C) 2000 Free Software Foundation, Inc.,
59 Temple Place - Suite 330, Boston, MA 02111, USA
<P>
Verbatim copying and distribution of this entire article is
permitted in any medium, provided this notice is preserved.
@end ifhtml
@bye
|