Migration from JUnit to TestNG
==============================

Author Bela Ban

Goals
-----

1. Reduce unnecessary channel creations, cluster joins and cluster leaves
   - Example: if we have 10 methods in a test class, we create and connect to a channel
     in setup() and disconnect and tear down the channel in tearDown() for *each* test method
   - Unless channel creation/tear down is tested (like in CloseTest), we can create a connect to the
     channel before the test, and disconnect and tear down the channel after all 10 methods have been executed
   - This will save us the cost of 9 channel creations/connects/disconnects

2. Run all the tests in separate threads
   - A thread pool (say 20 threads, depending on how many core we have) is configured
   - All tests are processed by different threads in that pool
   - The first 20 tests to be run are processed: each test is run by a separate thread
   - When a thread is done, it grabs the next test, until no more tests are available
   - All methods inside a given test are still executed *sequentially*, this is necessary as different methods within
     a test modify shared state, e.g. a view or the number of messages received, and are therefore dependent on
     running sequentially
   - We need to make sure we dynamically change mcast addresses / ports so that we don't have overlapping traffic 
   - Later, we might be able to take advantage of annotations which mark an entire test, such that it can be executed
     concurrently
   - This is similar to TestNG's parallel="methods" | "tests", but TestNG doesn't currently have a mode which allows for
     concurrent execution of different tests (with sequential execution of methods inside those tests)
   - We probably have to extend TestNG to do this:
     - parallel="tests", but provide a list of tests (one for each test class) to TestNG *programmatically*
   - The dependOn{Methods,Groups} option allows us to define an ordering between methods of a test. However, this
     requires detailed dependency analysis between the methods of a test. While this probably makes parallel
     processing even faster (as we don't need sequential processing for methods inside a test), we should do it
     in a second stage. This would allow us to use parallel="tests" or even "methods"
   - Use @Test(sequential=true) at the *class level*, so all methods within a class are executed on the same thread
   - Move instance variables into methods (unless they are channels)

x. Different stack configs (udp, tcp, mux-udp, mux-tcp etc)
   - As data providers ?
   - @Parameter annotation, e.g.

     @Parameter(name="channel.conf") // defined in the ChannelTestsBase (or whichever) super class
     public void setupStack(String config) {
         channel=createChannel(config);
     }

   - The @Parameter annotation is probably better than data providers, as we can run the stack dependent test suites
     as separate tests, with different parameters, e.g.:

     <suite name="JGroups test suite">
        <test name="UDP tests">
            <parameter name="channel.conf"  value="udp.xml"/>
        </test>
        <test name="TCP tests">
            <parameter name="channel.conf"  value="tcp.xml"/>
        </test>
        <test name="mux-udp tests">
            <parameter name="channel.conf"  value="mux-udp"/>
        </test>

        <test name="mux-tcp tests">
            <parameter name="channel.conf"  value="mux-tcp"/>
        </test>
        ...
     </suite>

x. Distributed execution across a cluster
   - Same as #2, but different tests are run on different hosts
   - The thread pool consists of different threads pools, hosted by different boxes
   - http://beust.com/weblog/archives/000362.html ?


Conversion
----------

#1 Change ANT build for tests
   - Provide a bunch of TestNG XML files, one for each config, e.g. testng-functional.xml, testng-udp.xml,
     testng-mux-udp.xml and so on
   - Change ANT targets to invoke the right TestNG XML file
   - All TestNG XML files have junit="true", so the unchanged tests can be run
   - Output uses the ReportNG reporter
   - [tbd] Convert into the old XML format, so JUnitReport can process those files

#2 Change unit test to TestNG
   - Tag @Test with group "functional" or "stack"
   - Modify ChannelTestBase, rename setUp() and teardown() and add @BeforeMethod and @AfterMethod to them
   - Remove 'extends TestCase'
   - Add @BeforeMethod to setUp() and @AfterMethod to tearDown()
   - Replace assertXXX() with org.testng.Assert.assertXXX()
   - OR: replace assertXXX() with assert <expr>



Todos:
------
- Convert all tests to TestNG (use TestNG's automatic converter or use awk.sed ?)
- Remove junit.jar, add testng.jar
- Write correct TestNG XML file
- Integrate with build.xml


Questions, issues
-----------------

- Identify useful group names and use those consistently
  - udp, mux-udp, tcp, mux-tcp ?
  - functional vs. stack-dependent ?
- How do we generate mcast addresses and ports such that traffic doesn't overlap ?
  --> Possibly a @BeforeTest annotation which modifies the stack config ?

- We're using system properties (mux.on, channel.conf etc) to configure stacks. This means we *cannot* run 2
  different stacks at the same time because the system properties are singleton wrt the JVM !
  --> Change from system properties to TestNG parameters ? Or DataProviders ?
  Solution: parameters:

  ChannelTestBase:

  public class ChannelTestBase {
    String channel_conf=null;

    @BeforeClass @Parameters(value="channel.conf")
    public void initialize(String channel_conf) {
        this.channel_conf=channel_conf;
    }

    protected Channel createChannel() throws ChannelException {
        return new JChannel(channel_conf);
    }


    Test (subclass of ChannelTestBase):

    @BeforeClass
    public void setUp() throws ChannelException {
        ch=createChannel();
        ch.connect("demo");
    }

    @AfterClass public void destroy() {
        ch.close();
    }
  

    Update 1: use of thread locals instead of instance variables:
    -------------------------------------------------------------

    Since the same thread invokes the before (@BeforeMethod) and after (@AfterMethod) methods, we can move boilerplate
    code to before- or after-methods, e.g.

    // instance variables
    private final ThreadLocal<Channel> ch=new ThreadLocal<Channel>();
    private final ThreadLocal<String>  PROPS=new ThreadLocal<String>();
    private final ThreadLocal<String>  GROUP=new ThreadLocal<String>();

     @BeforeMethod
    void init() throws Exception {
        String cluster_name=getUniqueClusterName("ChannelTest");
        GROUP.set(cluster_name);
        Channel tmp=createChannel(true, 2);
        String tmp_props=tmp.getProperties();
        PROPS.set(tmp_props);
        tmp.connect(GROUP.get());
        ch.set(tmp);
    }


    @AfterMethod
    void cleanup() {
        Channel tmp_ch=ch.get();
        Util.close(tmp_ch);
        ch.set(null);
        GROUP.set(null);
        PROPS.set(null);
    }

     @Test
    public void testViewChange() throws Exception {
        ViewChecker checker=new ViewChecker(ch.get());
        ch.get().setReceiver(checker);

        Channel ch2=createChannelWithProps(PROPS.get());
        try {
            ch2.connect(GROUP.get());
            assertTrue(checker.getReason(), checker.isSuccess());
            ch2.close();
            assertTrue(checker.getReason(), checker.isSuccess());
        }
        finally {
            Util.close(ch2);
        }
    }


    Update 2: use a unique channels
    -------------------------------

    In the above example, cluster names were unique for any given test (getUniqueClusterName()), and
    channels were also unique (createChannel(true, 2)).
    Unique channels are created by the superclass ChannelTestBase: it simply creates a normal channel, but then
    (if unique is true in createChannel()), looks at the config and changes it, e.g. mcast_addr and mcast_port, or
    bind_port for TCP.

    To join another channel to the unique channel for the test, we get the modified properties (Channel.getProperties()),
    store them in the PROPS thread local, and have the new channels use PROPS.