ROSE Compiler Framework/SPEC CPU 2006 benchmark
spec cpu 2006
editAssuming the source package is under /usr/casc/overture/ROSE/svn/spec_cpu2006_v1.1
installation
edit./install.sh -d /home/yourAccount/opt/spec_cpu2006
files and directories
edit/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench Subdirectories
- build data Docs exe run Spec src
benchmark summary
edit12+17= 29 benchmarks
12 Integer benchmarks: http://www.spec.org/cpu2006/CINT2006/
- 400.perlbench C Programming Language
- Derived from Perl V5.8.7. The workload includes SpamAssassin, MHonArc (an email indexer), and specdiff (SPEC's tool that checks benchmark outputs).
$ cloc-1.56.pl 400.perlbench
3057 text files.
1517 unique files.
484 files ignored.
http://cloc.sourceforge.net v 1.56 T=57.0 s (23.8 files/s, 26844.6 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Perl 1179 127322 661995 451954
C 109 13768 31736 192811
C/C++ Header 65 4202 9296 30785
XML 1 680 13 5357
Pascal 1 9 115 78
make 2 4 1 14
-------------------------------------------------------------------------------
SUM: 1357 145985 703156 680999
-------------------------------------------------------------------------------
- 401.bzip2 C Compression
- Julian Seward's bzip2 version 1.0.3, modified to do most work in memory, rather than doing I/O.
$ cloc-1.56.pl 401.bzip2/
56 text files.
43 unique files.
27 files ignored.
http://cloc.sourceforge.net v 1.56 T=1.0 s (17.0 files/s, 12473.0 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 10 1083 1131 5561
HTML 1 639 5 2837
C/C++ Header 3 210 174 617
XML 1 36 6 137
Perl 1 5 0 25
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 17 1974 1317 9182
-------------------------------------------------------------------------------
- 403.gcc C C Compiler
- Based on gcc Version 3.2, generates code for Opteron.
$ cloc 403.gcc/
-bash: cloc: command not found
[hudson-rose@hudson-rose-30]cloc-1.56.pl 403.gcc/
597 text files.
319 unique files.
26 files ignored.
http://cloc.sourceforge.net v 1.56 T=40.0 s (7.4 files/s, 46660.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Assembly 11 0 2794 1331673
C 159 58475 62060 370172
C/C++ Header 110 5890 9295 17385
XML 1 1242 6 3794
Teamcenter def 12 611 47 2897
Perl 1 4 0 55
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 295 66223 74203 1725981
-------------------------------------------------------------------------------
- 429.mcf C Combinatorial Optimization
- Vehicle scheduling. Uses a network simplex algorithm (which is also used in commercial products) to schedule public transport.
$ cloc-1.56.pl 429.mcf/
94 text files.
64 unique files.
37 files ignored.
http://cloc.sourceforge.net v 1.56 T=2.0 s (19.5 files/s, 2090.0 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 22 526 345 2501
C/C++ Header 14 221 216 204
XML 1 24 10 95
Perl 1 5 0 26
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 39 777 572 2831
-------------------------------------------------------------------------------
- 445.gobmk C Artificial Intelligence: Go
- Plays the game of Go, a simply described but deeply complex game.
$ cloc-1.56.pl 445.gobmk/
3110 text files.
1543 unique files.
2767 files ignored.
http://cloc.sourceforge.net v 1.56 T=37.0 s (4.3 files/s, 7464.7 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 134 25940 25404 218416
C/C++ Header 22 1048 1595 3478
XML 1 49 6 139
Perl 1 7 11 95
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 159 27045 27017 222133
-------------------------------------------------------------------------------
- 456.hmmer C Search Gene Sequence
- Protein sequence analysis using profile hidden Markov models (profile HMMs)
$ cloc-1.56.pl 456.hmmer/
235 text files.
160 unique files.
37 files ignored.
http://cloc.sourceforge.net v 1.56 T=4.0 s (33.0 files/s, 17054.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 114 3946 22853 38230
C/C++ Header 15 324 1045 1413
XML 1 75 6 212
Perl 1 4 0 102
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 132 4350 23905 39962
-------------------------------------------------------------------------------
- 458.sjeng C Artificial Intelligence: chess
- A highly-ranked chess program that also plays several chess variants.
$ cloc-1.56.pl 458.sjeng/
69 text files.
45 unique files.
18 files ignored.
http://cloc.sourceforge.net v 1.56 T=2.0 s (14.0 files/s, 7162.5 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 21 2152 1068 10429
C/C++ Header 4 123 36 397
XML 1 17 6 56
Perl 1 6 0 28
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 28 2299 1111 10915
-------------------------------------------------------------------------------
- 462.libquantum C Physics / Quantum Computing
- Simulates a quantum computer, running Shor's polynomial-time factorization algorithm.
$ cloc-1.56.pl 462.libquantum/
86 text files.
54 unique files.
19 files ignored.
http://cloc.sourceforge.net v 1.56 T=1.0 s (36.0 files/s, 4694.0 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 18 798 492 2302
C/C++ Header 15 267 243 394
XML 1 40 6 111
Perl 1 7 0 27
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 36 1113 742 2839
-------------------------------------------------------------------------------
- 464.h264ref C Video Compression
- A reference implementation of H.264/AVC, encodes a videostream using 2 parameter sets. The H.264/AVC standard is expected to replace MPEG2
$ cloc-1.56.pl 464.h264ref/
191 text files.
109 unique files.
23 files ignored.
http://cloc.sourceforge.net v 1.56 T=3.0 s (28.7 files/s, 17554.0 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 44 5968 7718 33063
C/C++ Header 39 899 1118 3409
XML 1 71 6 321
Perl 1 6 0 76
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 86 6945 8843 36874
-------------------------------------------------------------------------------
- 471.omnetpp C++ Discrete Event Simulation
- Uses the OMNet++ discrete event simulator to model a large Ethernet campus network.
$ cloc-1.56.pl 471.omnetpp/
333 text files.
178 unique files.
22 files ignored.
http://cloc.sourceforge.net v 1.56 T=4.0 s (39.2 files/s, 12109.5 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C++ 85 5087 3697 22603
C/C++ Header 69 2684 9788 4044
XML 1 122 6 357
Perl 1 4 0 39
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 157 7898 13492 27048
-------------------------------------------------------------------------------
- 473.astar C++ Path-finding Algorithms
- Pathfinding library for 2D maps, including the well known A* algorithm.
$ cloc-1.56.pl 473.astar/
107 text files.
64 unique files.
42 files ignored.
http://cloc.sourceforge.net v 1.56 T=1.0 s (33.0 files/s, 9131.0 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C++ 22 949 192 5972
C/C++ Header 8 544 62 1277
XML 1 22 6 56
Perl 1 9 0 35
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 33 1525 261 7345
-------------------------------------------------------------------------------
- 483.xalancbmk C++ XML Processing
- A modified version of Xalan-C++, which transforms XML documents to other document types.
$ cloc-1.56.pl 483.xalancbmk/
3665 text files.
1820 unique files.
19 files ignored.
http://cloc.sourceforge.net v 1.56 T=75.0 s (24.0 files/s, 37920.6 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
XML 4 194172 8 2074361
C++ 829 54318 72591 202976
C/C++ Header 944 35917 120959 80316
C 19 811 2399 2982
XSD 2 86 0 789
Perl 1 16 14 767
XSLT 2 58 0 500
make 1 1 1 5
-------------------------------------------------------------------------------
SUM: 1802 285379 195972 2362696
-------------------------------------------------------------------------------
17 Floating point benchmarks: http://www.spec.org/cpu2006/CFP2006/
- 410.bwaves Fortran Fluid Dynamics Computes 3D transonic transient laminar viscous flow.
- 416.gamess Fortran Quantum Chemistry. Gamess implements a wide range of quantum chemical computations. For the SPEC workload, self-consistent field calculations are performed using the Restricted Hartree Fock method, Restricted open-shell Hartree-Fock, and Multi-Configuration Self-Consistent Field
- 433.milc C Physics / Quantum Chromodynamics A gauge field generating program for lattice gauge theory programs with dynamical quarks.
- 434.zeusmp Fortran Physics / CFD ZEUS-MP is a computational fluid dynamics code developed at the Laboratory for Computational Astrophysics (NCSA, University of Illinois at Urbana-Champaign) for the simulation of astrophysical phenomena.
- 435.gromacs C,
Fortran Biochemistry / Molecular Dynamics Molecular dynamics, i.e. simulate Newtonian equations of motion for hundreds to millions of particles. The test case simulates protein Lysozyme in a solution.
- 436.cactusADM C,
Fortran Physics / General Relativity Solves the Einstein evolution equations using a staggered-leapfrog numerical method
- 437.leslie3d Fortran Fluid Dynamics Computational Fluid Dynamics (CFD) using Large-Eddy Simulations with Linear-Eddy Model in 3D. Uses the MacCormack Predictor-Corrector time integration scheme.
- 444.namd C++ Biology / Molecular Dynamics Simulates large biomolecular systems. The test case has 92,224 atoms of apolipoprotein A-I.
- 447.dealII C++ Finite Element Analysis deal.II is a C++ program library targeted at adaptive finite elements and error estimation. The testcase solves a Helmholtz-type equation with non-constant coefficients.
- 450.soplex C++ Linear Programming, Optimization Solves a linear program using a simplex algorithm and sparse linear algebra. Test cases include railroad planning and military airlift models.
- 453.povray C++ Image Ray-tracing Image rendering. The testcase is a 1280x1024 anti-aliased image of a landscape with some abstract objects with textures using a Perlin noise function.
- 454.calculix C,
Fortran Structural Mechanics Finite element code for linear and nonlinear 3D structural applications. Uses the SPOOLES solver library.
- 459.GemsFDTD Fortran Computational Electromagnetics Solves the Maxwell equations in 3D using the finite-difference time-domain (FDTD) method.
- 465.tonto Fortran Quantum Chemistry An open source quantum chemistry package, using an object-oriented design in Fortran 95. The test case places a constraint on a molecular Hartree-Fock wavefunction calculation to better match experimental X-ray diffraction data.
- 470.lbm C Fluid Dynamics Implements the "Lattice-Boltzmann Method" to simulate incompressible fluids in 3D
- 481.wrf C,
Fortran Weather Weather modeling from scales of meters to thousands of kilometers. The test case is from a 30km area over 2 days.
- 482.sphinx3 C Speech recognition A widely-known speech recognition system from Carnegie Mellon University
language summary
editC : 14 benchmarks/ 12 real ones excluding specrand
400.perlbench Programming Language 401.bzip2 Compression 403.gcc C Compiler 429.mcf Combinatorial Optimization 433.milc Physics / Quantum Chromodynamics 445.gobmk Artificial Intelligence: Go 456.hmmer Search Gene Sequence 458.sjeng Artificial Intelligence: chess 462.libquantum Physics / Quantum Computing 464.h264ref Video Compression 470.lbm Fluid Dynamics 482.sphinx3 Speech recognition 998.specrand 999.specrand
C++: 7 benchmarks : namd dealII soplex povray omnetpp astar xalancbmk
444.namd 447.dealII 450.soplex 453.povray 471.omnetpp 473.astar 483.xalancbmk
Fortran: 6 benchmarks: bwaves gamess zeusmp leslie3d GemsFDTD tonto
410.bwaves 416.gamess 434.zeusmp 437.leslie3d 459.GemsFDTD 465.tonto
Mixed C and Fortran: 4: gromacs cactusADM calculix wrf
435.gromacs 436.cactusADM 454.calculix 481.wrf
configuration
edit- enviornment Source the shrc or cshrc in /home/youAccount/opt/spec_cpu2006 to set up your environment for the benchmark.
. ./shrc
- configuration prepare config/tux268-gcc42.cfg set compilers and flags based on existing examples there
Table summary
- output_format = asc, pdf, Screen, html
use
edithttp://www.spec.org/cpu2006/Docs/runspec.html#section3.1actions
- build only:
bash-3.00$ which runspec /home/youAccount/opt/spec_cpu2006/bin/runspec bash-3.00$ runspec --config=tux268-gcc42.cfg --action=build --tune=base bzip2
clean up first
- runspec --config=rose-linux64-gcc41 --tune=base --size=test --iterations=1 --noreportable --action=clobber all
- Run with test data set default is --size=ref : run with reference data set
runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2 runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2
- run all integer or fp benchmarks 12 total integer
runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable int // total 12 400.perlbench -- 3.78 -- S 401.bzip2 -- 7.52 -- S 403.gcc -- 1.42 -- S 429.mcf -- 5.37 -- S 445.gobmk -- 20.0 -- S 456.hmmer -- 5.35 -- S 458.sjeng -- 5.00 -- S 462.libquantum -- 0.0951 -- S 464.h264ref -- 21.7 -- S 471.omnetpp -- 0.545 -- S 473.astar -- 10.7 -- S 483.xalancbmk -- 0.115 -- S Est. SPECint_base2006 -- Est. SPECint2006 Not Run runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable fp // total 17 410.bwaves -- 12.4 -- S 416.gamess -- 0.510 -- S 433.milc -- 8.71 -- S 434.zeusmp -- 25.2 -- S 435.gromacs -- 1.74 -- S 436.cactusADM -- 4.66 -- S 437.leslie3d -- 32.7 -- S 444.namd -- 15.6 -- S 447.dealII -- 22.6 -- S 450.soplex -- 0.0263 -- S 453.povray -- 0.874 -- S 454.calculix -- 0.0546 -- S 459.GemsFDTD -- 4.52 -- S 465.tonto -- 1.35 -- S 470.lbm -- 13.6 -- S 481.wrf -- 5.66 -- S 482.sphinx3 -- 2.26 -- S Est. SPECfp_base2006 -- Est. SPECfp2006 Not Run
test gcc/g++/gfortran first then build only
cleanup/clobber
editclean up: --action=clean or clobber then identityTranslator
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable gobmk
data set
editReportable runs ensure that your binaries can produce correct results with the test and train workloads (this run rule requirement is new as of CPU2006 V1.0) and then run the ref workload 3 times for the actual measurements.
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=ref --iterations=1 --noreportable perlbench
data sizes
- test: smallest data set
- train: test the ability to run more than one size in a single invocation
- ref (default): run 3 times for reportable results
selective run
editrunspec --action=clobber int fp runspec --config=tux268-intel.cfg --tune=base --size=test --iterations=1 --noreportable bzip2 runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2 runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2 bash-3.00$ pwd /home/yourAccount/opt/spec_cpu2006 bash-3.00$ runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable fp
modified source run
editChanged /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench/src/spec_config.h
to have #include <stdbool.h>
Have to add a line in the config file (eg. config/tux268-rose-gcc42.cfg) to avoid corrupted source warning
strict_rundir_verify = 0
debug
editpwd
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench/src
/home/yourAccount/opt/roseLatest/bin/identityTranslator -c -o sv.o -DSPEC_CPU -DNDEBUG -DPERL_CORE -O2 -DSPEC_CPU_LINUX_IA32 sv.c Adding argv[3] = sv.o to p_objectFileNameList Using strcmp(): This was not a valid string (buffer = define returned) Using strcmp(): This was not a valid string (buffer = define returned) Error: Unknown cppIndentifier = identityTranslator: ../../../../rose/src/frontend/SageIII/rose_attributes_list.C:1676: bool ROSEAttributesList::isCppDirective(const std::string&, PreprocessingInfo::DirectiveType&, std::string&): Assertion `false' failed. Aborted
results verification
edit/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/401.bzip2/run/run_base_test_i386-m32-gcc42-nn.0001
- .mis
bash-3.00$ cat dryer.jpg.out.mis
0008: Compressed data 1434910 bytes in length
Compressed data 1434894 bytes in length
^
0013: Compressed data 1127895 bytes in length
Compressed data 1127995 bytes in length
^
0018: Compressed data 1067335 bytes in length
Compressed data 1067427 bytes in length
^
run manually
editeach benchmark has a command line file:
run_base_test_amd64-m64-gcc41-nn.0000/speccmds.cmd
sphinx_livepretend_base.amd64-m64-gcc41-nn ctlfile . args.an4
403.gcc ~/opt/spec_cpu2006/benchspec/CPU2006/403.gcc/run/run_base_test_amd64-m64-gcc41-nn.0000]
[yourAccount@tux322]cat speccmds.cmd -C /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/403.gcc/run/run_base_test_amd64-m64-gcc41-nn.0000 -o cccp.out -e cccp.err ../run_base_test_amd64-m64-gcc41-nn.0000/gcc_base.amd64-m64-gcc41-nn cccp.i -o cccp.s # ignore -C etc. focus on the final part! gdb -args ../run_base_test_amd64-m64-gcc41-nn.0000/gcc_base.amd64-m64-gcc41-nn cccp.i -o cccp.s
ROSE identityTranslator
editformal text
editWe use ROSE's identityTranslator. Like any other ROSE-based translators, it is designed to automatically detect input source file types and invoke the right EDG capability.
We do test ROSE on a subset of SPEC CPU2006 as part of our daily regression test. ROSE can fully support 10 out 12 spec C benchmarks (compiling them and having verified execution results). The 10 C benchmars are perlbench, bzip2, mcf, milc, gobmk, hmmer, sjeng, libquantum, lbm, and sphinx3. gcc and h264ref have some minor issues (mostly related to ROSE's handling of macro calls) but we are working on this.
The way we compile SPEC using ROSE is: 1) install (via make install) ROSE to a directory, like ROSE_INS (specified via configure --prefix=ROSE_INS).
2) set up the environment for using the installed ROSE, identityTranslator is located under ROSE_INS/bin PATH=$ROSE_INS/bin:$PATH LD_LIBRARY_PATH=$ROSE_INS/lib:$LD_LIBRARY_PATH export PATH LD_LIBRARY_PATH
3) preparing an SPEC configuration file (e.g. spec_installation_path/config/rose.cfg) to use ROSE A set of relevant options in rose.cfg are:
#We want to the test to abort on errors and report immediately ignore_errors = no # we want have ascii and table-based output (Screen) for results output_format = asc, Screen #The result is not intended for official reports to the SPEC organization reportable = 0 # compilers to compile benchmarks CC = identityTranslator CXX = identityTranslator FC = identityTranslator # compilation options: turn off ROSE ’s EDG frontend warnings # since we are not interesting in fixing the benchmarks COPTIMIZE = -O2 --edg:no_warnings CXXOPTIMIZE = -O2 --edg:no_warnings FOPTIMIZE = -O2 --edg:no_warnings
4) finally you can test ROSE on spec
cd $SPEC_CPU2006_INS # set up environment to run spec . ./shrc # clean up previous object/executable files for all floating point and integer benchmarks runspec --config=rose.cfg --tune=base --size=test --iterations=1 --noreportable fp int --action=clobber runspec --config=$SPEC_CPU2006_CONFIG --tune=base --size=test --iterations=1 --noreportable \ perlbench bzip2 mcf milc gobmk hmmer sjeng libquantum lbm sphinx3
12 C benchmarks
edit10/12 passed now
Passing set: runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable perlbench bzip2 mcf milc gobmk hmmer sjeng libquantum lbm sphinx3
perlbench: pass !!! compilation -D_GNUG__ problem bool problem
bzip2: pass !!! line continuation for two empty line problem: compilation but wrong results
gcc : noncompilable rose_x pending on a bug 347?, Unparse_ExprStmt::unparseEnumDeclStmt() SgInitializedName should be moved to located node
test case test2009_05.c
mcf : pass!!!
milc: pass!!!
gobmk : pass!!! file path issue include/liberty.h vs. engine/liberty.h rose_out.c engine/rose_out.c
hmmer : pass!!! EDG frontend failure: implicit C function decl
sjeng : pass!!!
libquantum:pass!!!
h264ref: noncompilable rose_x recursive macro replace after macro expansion:
lbm : pass!!!
sphinx3: pass!!! __FILE__ macro expansion problem. file name vs. full path+filename
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable sphinx3
/home/yourAccount/opt/spec_cpu2006/bin/specinvoke -d /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482.sp hinx3/run/run_base_test_i386-m32-gcc42-nn.0000 -e speccmds.err -o speccmds.stdout -f speccmds.cmd -C
/home/yourAccount/opt/spec_cpu2006/bin/specinvoke -E -d /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482 .sphinx3/run/run_base_test_i386-m32-gcc42-nn.0000 -c 1 -e compare.err -o compare.stdout -f compare.c md
*** Miscompare of an4.log; for details see
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482.sphinx3/run/run_base_test_i386-m32-gcc42-nn.0 000/an4.log.mis
Error: 1x482.sphinx3
7 C++ benchmarks
editrunspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable namd dealII soplex povray omnetpp astar xalancbmk
References
edit- Henning, J.L., SPEC CPU2000 Measuring CPU Performance in the New Millennium, 2000
- http://www.spec.org/cpu2006/