ROSE Compiler Framework/Testing

ROSE uses Jenkins to implement a contiguous integration software development process. It leverages a range of software packages to test its correctness, robustness, and performance.

make check rules

we leverage make check rules to do internal testing.

check exist status of pipeline command

In bash scripting, we can use pipelines | as follows:

command1 | command2 : the output of each command in the pipeline is connected to the input of the next command

each command is executed in its own subshell, exit status: the last command's exit status

To catch any command's return code, please use ${PIPESTATUS[0]}

For example: Using pipeline will only return the last command 'fold''s status. we add a test to catch the first command's return status

 ../autoPar -c $(srcdir)/$(@:.o=.c) | fold >$(@:.o=.out); test ${PIPESTATUS[0]} = 0

Benchmarks

The software used by the ROSE's Jenkins include:

SPEC CPU 2006 benchmark: a subset is supported for now
SPEC OMP benchmark: a subset is supported for now
NAS parallel benchmark: developed by NASA Ames Research Center. Both C (customized version) and OpenMP versions are used
Plum Hall C and C++ Validation Test Suites: a subset is supported for now
Jt++ - Java conformance testing: http://modena.us/

Modena Jt++ Test Suite

1. Clone the Modena test suite repository:

  $ git clone ssh://rose-dev@rose-git/modena

2. Autotools setup

  $ cd modena
  $ ./build.sh
  + libtoolize --force --copy --ltdl --automake
  + aclocal -I ./acmacros -I ./acmacros/ac-archive -I /usr/share/aclocal
  + autoconf
  + automake -a -c
  configure.ac:4: installing `./install-sh'
  configure.ac:4: installing `./missing'

3. Environment bootstrap

  $ source /nfs/apps/python/latest/setup.sh

4. Build and test!

  $ mkdir buildTree
  $ cd buildTree
  $ ../configure \
          --with-sqlalchemy=${HOME}/opt/python/sqlalchemy/0.7.5/lib64/python2.4/site-packages \
          --with-target-java-interpreter=java \
          --with-target-java-compiler=testTranslator \
          --with-target-java-compiler-flags="-ecj:1.6" \
          --with-host-java-compiler-flags="-source 1.6"

Jenkins

Using External Benchmarks

The way we set it up is to

In the benchmark, we change the benchmark's build system to call the ROSE tool (identityTranslator or your RTED tool) installed.
In the Jenkins test job,
- Build and install the tested ROSE, prepare environment variables.
- Go to the benchmark with modified build system. Build and run the benchmark.

Basically, the test job should simulate how a ROSE tool would be used by end-users, not by tweaking ROSE for each different benchmarks.

NAS Parallel Benchmarks

We have NPB as part of our regression tests.

To get the benchmark:

git clone rose-dev@rosecompiler1.llnl.gov:testsuite/npb-c-parallel.git

Within the benchmark, there is a make.def file with configuration about which compiler and options to use. The configuration should be correct but some path may be changed to point to your version of ROSE.

After that, type "make suite" to build the entire benchmark suite. Or type "make mg class=A" to just build the benchmark in question.