Wikibooks

ROSE Compiler Framework/ROSE tools

< ROSE Compiler Framework

Overview

edit

ROSE is a compiler framework to build customized compiler-based tools. A set of example tools are provided as part of the ROSE release to demonstrate the use of ROSE. Some of them are also useful for daily work of ROSE developers.

We list and briefly explain some tools built using ROSE. They are installed under ROSE_INSTALLATION_TREE/bin .

  • some tools are not installed by default. You may have to cd into the tool's subdirectory and type "make install"

Any ROSE tool(translator) works like a compiler. You have to provide all necessary compilation flags to make it work properly. You have to specify some include paths and/or some macro to be defined.

  • One way to use rose tools is to replace your default compiler (gcc for example) with the rose tool command (identityTranslator, outline, etc) in your Makefile. So the tool will be called with all correct flags when you type "make".
  • Alternatively, if you are only interested in processing a single file. You can manually watch the full compilation command line (e.g. gcc .... -c ) used to compile that single file during a normal "make". Then replace your compiler (gcc) with the rose tool you are interested in (e.g. outline) in the command line.

Prerequisites

edit

You have to install ROSE first, by typing configure, make, make install, etc.

You also have to set the environment variables properly before you can call ROSE tools from command line.

For example: if the installation path (or --prefix path in configure) is /home/opt/rose/install, you can have the following script to set the environment variables using bash: 

ROSE_INS=/home/opt/rose/install
export ROSE_INS 

PATH=$ROSE_INS/bin:$PATH
export PATH

LD_LIBRARY_PATH=$ROSE_INS/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH

All ROSE tools are installed under the path specified by the --prefix option.

identityTranslator

edit

Source: http://www.rosecompiler.org/ROSE_Tutorial/ROSE-Tutorial.pdf (chapter 2)

This is the simplest tool built using ROSE. It takes input source files , builds AST, and then unparses the AST back to compilable source code. It tries its best to preserve everything from the input file.

Uses

edit

Typical use cases

  • without any options, test if ROSE can compile your code: replace the compiler used by your Makefile with identityTranslator
  • turn on some built-in analysis, translation or optimization phases, such as -rose:openmp:lowering to support OpenMP
    • type "identityTranslator --help" to see all options
  • debug a ROSE-based translator: the first step is often to use identityTranslator to rule out if it is a compilation problem using ROSE
  • use the source of the identityTranslator as a start point to add custom analysis and transformation. The code in the identityTranslator is indeed the minimum code required for almost all kinds of ROSE-based tools.

Source code

edit

identityTranslator.c 

#include "rose.h"
int main(int argc, char *argv[]){
	// Build the AST used by ROSE
	SgProject *project = frontend(argc, argv);

	// Run internal consistency tests on AST
	AstTests::runAllTests(project);

	// Insert your own manipulation of the AST here...

	// Generate source code from AST and call the vendor's compiler
	return backend(project);
}

Plugin

edit

Starting from Version 0.9.9.83, ROSE has a new feature to support external plugins. It borrows the design and implementation of Clang Plugins. The interface is very similar to what Clang has, with some simplification and improvements.

With this feature, you can develop your ROSE-based tools as dynamically loadable plugins. Then you can use command line options of ROSE's default translator, identityTranslator (or another ROSE translator), to

  • load shared libraries containing the plugins,
  • specify actions to be executed,
  • as well as pass command line options to each action.

See more at:

Limitations

edit

Due to limitations of the frontends used by ROSE and some internal processing, identityTranslator cannot generate 100% identical output compared to the input file.

Some notable changes it may introduce include:

  • "int a, b, c;" are transformed to three SgVariableDeclaration statements,
  • macros are expanded.
  • extra brackets are added around constants of typedef types (e.g. c=Typedef_Example(12); is translated in the output to c = Typedef_Example((12));)
  • Converting NULL to 0.

AST dot graph generators

edit

Tools to generate AST graph in dot format. There are two versions

  • dotGenerator: simple AST graph generator showing essential nodes and edges
  • dotGeneratorWholeASTGraph: whole AST graph showing more details. It provides filter options to show/hide certain AST information.

command line: 

 dotGeneratorWholeASTGraph  yourcode.c  // it is best to avoid include any header into your sample code to have a small enough tree to visualize!

./dotGeneratorWholeASTGraph -rose:help | more
   -rose:help                     show this help message
   -rose:dotgraph:asmFileFormatFilter              [0|1]  Disable or enable asmFileFormat filter
   -rose:dotgraph:asmTypeFilter                    [0|1]  Disable or enable asmType filter
   -rose:dotgraph:binaryExecutableFormatFilter     [0|1]  Disable or enable binaryExecutableFormat filter
   -rose:dotgraph:commentAndDirectiveFilter        [0|1]  Disable or enable commentAndDirective filter
   -rose:dotgraph:ctorInitializerListFilter        [0|1]  Disable or enable ctorInitializerList filter
   -rose:dotgraph:defaultFilter                    [0|1]  Disable or enable default filter
   -rose:dotgraph:defaultColorFilter               [0|1]  Disable or enable defaultColor filter
   -rose:dotgraph:edgeFilter                       [0|1]  Disable or enable edge filter
   -rose:dotgraph:expressionFilter                 [0|1]  Disable or enable expression filter
   -rose:dotgraph:fileInfoFilter                   [0|1]  Disable or enable fileInfo filter
   -rose:dotgraph:frontendCompatibilityFilter      [0|1]  Disable or enable frontendCompatibility filter
   -rose:dotgraph:symbolFilter                     [0|1]  Disable or enable symbol filter
   -rose:dotgraph:emptySymbolTableFilter           [0|1]  Disable or enable emptySymbolTable filter
   -rose:dotgraph:emptyFunctionParameterListFilter [0|1]  Disable or enable emptyFunctionParameterList filter
   -rose:dotgraph:emptyBasicBlockFilter            [0|1]  Disable or enable emptyBasicBlock filter
   -rose:dotgraph:typeFilter                       [0|1]  Disable or enable type filter
   -rose:dotgraph:variableDeclarationFilter        [0|1]  Disable or enable variableDeclaration filter
   -rose:dotgraph:variableDefinitionFilter         [0|1]  Disable or enable variableDefinitionFilter filter
   -rose:dotgraph:noFilter                         [0|1]  Disable or enable no filtering
Current filter flags' values are: 
         m_asmFileFormat = 0 
         m_asmType = 0 
         m_binaryExecutableFormat = 0 
         m_commentAndDirective = 1 
         m_ctorInitializer = 0 
         m_default = 1 
         m_defaultColor = 1 
         m_edge = 1 
         m_emptySymbolTable = 0 
         m_expression = 0 
         m_fileInfo = 1 
         m_frontendCompatibility = 0 
         m_symbol = 0 
         m_type = 0 
         m_variableDeclaration = 0 
         m_variableDefinition = 0 
         m_noFilter = 0

More information about how to use the tools can be found at How_to_visualize_AST

AST Outliner

edit

Basic concept: outlining is the process of replacing a block of consecutive statements with a function call to a new function containing those statements. Conceptually, outlining the inverse of inlining.

ROSE provide a builtin translator called AST outliner, which can outline a specified portion of code and generate a function from it.

  • Official documentation for the AST outliner is located in Chapter 37 Using the AST Outliner with the ROSE Tutorial. pdf.
  • Supplemental information can be found here at ROSE Compiler Framework/outliner

KeepGoingTranslator

edit

Often a ROSE-based tool may encounter some issues when processing a large scale applications. Users may want to let the tool keep running until the processing is finished and they can check the final results about how many source files are successfully processed or not. This is similar to the GNU make's -k option, by using which make will try to compile every file that can be tried and show as many compilation errors as possible.

ROSE has a builtin -rose:keep_going support. If this feature is turned on and an error occurs, ROSE will simply run your backend compiler on your original source code file, as is, without modification.

To further simplify the use of -rose:keep_going, we provide the Rose::KeepGoing namespace, which internally

  • uses -rose:keep_going and keeps track of successfully processed or failed files, and
  • save such information into log files.

An example translator named KeepGoingTranslator is created to demonstrate the use of this namespace.

In summary, there are three levels of keep going support

  • GNU make's -k option as part of the build system
  • ROSE's -rose:keep_going option, independent from the build system.
  • Rose::KeepGoing namespace with additional logging support

To use the builtin support for logging messages for each file 


# specify where to store success and error logs
  Rose::KeepGoing::report_filename__fail = boost::filesystem::path(getenv("HOME")).native()+"/mytool-failed_files.txt";
  Rose::KeepGoing::report_filename__pass = boost::filesystem::path(getenv("HOME")).native()+"/mytool-passed_files.txt";


# in your tool's traversal, add log messages 
    SgSourceFile* file = getEnclosingSourceFile(forloop);
    string s(":");
    string entry= forloop->get_file_info()->get_filename()+s+oss.str(); // add full filename to each log entries
    Rose::KeepGoing::File2StringMap[file]+= entry;


# in your main(), handle errorss

int
main ( int argc, char* argv[])
{
 ...
  vector<string> argvList(argv, argv+argc);
  argvList = commandline_processing (argvList);


   if (CommandlineProcessing::isOption (argvList,"-E","",false))
   {
     preprocessingOnly = true;
     // we should not put debugging info here. Otherwise polluting the generated preprocessed file!!
   }

  SgProject* project = frontend(argvList);
  ROSE_ASSERT (project != NULL);


  // register midend signal handling function
  if (KEEP_GOING_CAUGHT_MIDEND_SIGNAL)
  {
    std::cout
      << "[WARN] "
      << "Configured to keep going after catching a "
      << "signal in myTool"
      << std::endl;
    Rose::KeepGoing::setMidendErrorCode (project, 100);
    goto label_end;
  }
  else
  {
     // Your own traversal for analysis or transformation
    RoseVisitor visitor;

    SgFilePtrList file_ptr_list = project->get_fileList();
    for (size_t i = 0; i<file_ptr_list.size(); i++)
    {
      SgFile* cur_file = file_ptr_list[i];
      SgSourceFile* s_file = isSgSourceFile(cur_file);
      if (s_file != NULL)
      {
        visitor.traverseWithinFile(s_file, preorder);
      }
    }

  }

label_end:


# write log files
 
  int status = backend(project);
 // important: MUST call backend() first, then generate reports.
 // otherwise, backend errors will not be caught by keep-going feature!!

// We want the reports are generated with or without keep_going option
// if (MyTool::keep_going)
  {
    std::vector<std::string> orig_rose_cmdline(argv, argv+argc);
    Rose::KeepGoing::generate_reports (project, orig_rose_cmdline);
  }

  return status;
}

Inliner

edit

see more at ROSE_Compiler_Framework/Inliner

The ROSE Inliner inlines functions at function callsites.

Official documentation about the Inliner is

Source code

Test directory with an example translator and test input files

By looking into Makefile.am, the example translator's source code will generate an executable named "inlineEverything" in your buildtree.

This is the tool you can try to inline your sample code.

The same Makefile.am's make check rules contain sample command lines to use the tool.

call graph generator

edit

source code of this tool

Command line: 

 buildCallGraph -c yourprogram.cpp

Control flow graph generator

edit

Command line: 

 virtualCFG -c yourprogram.c

autoPar

edit

This is a tool which can automatically insert OpenMP pragmas into input serial C/C++ codes

This tool is an implementation of automatic parallelization using OpenMP. It is used to explore semantics-aware automatic parallelization, as described in one of our paper* .

  • The source files are currently located in rose/projects/autoParallelization.
  • A standalone executable program (named autoPar ) is generated and installed to the installation tree of ROSE (under ROSE INS/bin).
  • you can test the tool in rose_build_tree/projects/autoParallelization by typing "make check"
  • There is a section in ROSE manual pdf: 12.7 Automatic Parallelization

Publications

  • Chunhua Liao, Daniel J. Quinlan, Jeremiah J. Willcock and Thomas Panas, Semantic-Aware Automatic Parallelization of Modern Applications Using High-Level Abstractions, Journal of Parallel Programming, Volume 38, Numbers 5-6, 361-378, August 23. 2010 LLNL-JRNL-421803
  • Chunhua Liao, Daniel J. Quinlan, Jeremiah J. Willcock and Thomas Panas, Extending Automatic Parallelization to Optimize High-Level Abstractions for Multicore, In Proceedings of the 5th international Workshop on OpenMP: Evolving OpenMP in An Age of Extreme Parallelism (Dresden, Germany, June 03 - 05, 2009).

More info is at autoPar

abstract handles

edit

This work is used in the autotuning and also other tools that pass references to source code as part of an interface.

  • essentially it defines ways to create a string to uniquely identify a language construct in your source code
  • any tool then can locate the corresponding node in AST and do the targetted analysis or transformation for you.

Key info


See more at

Loop translators

edit

List of individual loop translators which perform transformation only. So callers use make sure the transformations are semantically correct for your input code

Example command lines to run these loop translators can be found in Makefile.am

  • e.g.: loopUnrolling -rose:loopunroll:abstract_handle 'Statement<position,5>' -rose:loopunroll:factor 3


There is another integrated loopProcessor, which relies on sophisticated analyses to drive a range of loop optimizations.

Declaration move tool

edit

This tool will move variable declarations to their innermost possible used scopes.

For a declaration, find the innermost scope we can move it into, without breaking the code's original semantics.

  • For a single use place, move to the innermost scope.
  • For the case of multiple uses, we may need to duplicate the declarations and move to two scopes if there is no variable reuse in between, otherwise, we move the declaration into the innermost common scope of the multiple uses.


User instructions: The translator accepts the following options:

  • -rose:merge_decl_assign will merge the moved declaration with an immediately followed assignment.
  • -rose:aggressive  : turn on the aggressive mode, which will move declarations with initializers, and across loop boundaries. A warning message will be sent out if the move crosses a loop boundary. Without this option, the tool only moves a declaration without an initializer to be safe.
  • -rose:debug, which is turned on by default in the testing. Some dot graph files will be generated for scope trees of variables for debugging purpose.
  • -rose:keep_going will ignore assertions as much as possible (currently on skip the assertion on complex for loop initialization statement list). Without this option, the tool will stop on assertion failures.
  • -rose:identity will turn off any transformations and act like an identity translator. Useful for debugging purposes.
  • -rose:trans-tracking will turn on the transformation tracking mode, showing the source statements of a move/merged declaration

source code: https://github.com/rose-compiler/rose-develop/blob/master/tests/roseTests/astInterfaceTests/moveDeclarationToInnermostScope.C

  • Note: moved to tests/nonsmoke/functional/roseTests/astInterfaceTests recently

tests: make move_diff_check defined in https://github.com/rose-compiler/rose-develop/blob/master/tests/roseTests/astInterfaceTests/Makefile.am


See more at Declaration move tool

Arithmetic intensity measuring tool

edit

Arithmetic intensity measuring tool : measure arithmetic intensity (FLOPS/Memory) of loops

MPI code generator

edit

A set of code generation functions used to support translating high level domain specific languages into MPI code, with some runtime support.

Compilation Database

edit

A utility program to convert a compilation database json file into a single makefile.

The motivation is to avoid hacking into users' build systems in order to invoke a ROSE-based tool.

With the single makefile, we can freely replace the compiler name and options without understanding the complex build systems.

Details at: https://github.com/rose-compiler/rose/tree/develop/utilities/compilationDatabase2Makefile


Steps to use it

  • generate compilation_database.json file
  • convert it to a makefile
  • rm all object files first from your build tree!!
    • make -f your-makefile clean
  • make -k -f your-makefile all # -k means keep going if somethings goes wrong

TODO

edit

refactor the tool translators

edit

Refactor the tools into a dedicated rose/tools directory. So they will always be built and available by default, with minimum dependency on other things, like which languages are turned on or off (when applicable of course)

Our current idea is we should separate translators used as examples or tutorials AND translators used for creating end-user tools.

  • For tutorial translators, they should NOT be installed as tools by default. Their purpose is to be included in Manual or Tutorial pdf files to illustrate something to developers by examples. Examples should be concise and to the point.
  • On the other hand, translators used to build end-user tools should have much higher standard to accept command options for different, even advanced features. These translators can be very sophisticated since they don't have the page limitation as tutorial examples do.
Retrieved from "https://en.wikibooks.org/w/index.php?title=ROSE_Compiler_Framework/ROSE_tools&oldid=3701302"