ROSE Compiler Framework/Source Code Repository

Motivation

Why are we considering this?

• A single monolithic git repository for ROSE becomes too large
  • not productive to clone big repo if people only care about a single portion of it
• We have many collaborative projects using ROSE. Many project contributors are forced to live with the same high-standard as the core ROSE contributors for commits
  • Same rigid regression tests by Jenkins for different platforms, different versions of GCC, boost, etc. This is not always necessary for some projects.
• We have some LaTeX documentation and HTML web pages within ROSE's git repository. They essentially can be standalone. But even minor fixing typo for them will require lengthy Jenkins tests (~10 hours or more) to be made effective. This really turns down our appetite to add/update our documents.

Goals

Have lean, clean, and well organized source code repositories for the ROSE project

• lean (<=100 MB): only essential things should be stored in the central core repository
• clean: only code-reviewed and tested things are allowed be added into central repositories
• well organized: intuitive, standardized directory layout

Facilitate collaborations

• allowing incremental/alternative testing requirements for collaborative, experimental projects

Other requirements: mostly already taken care of by using git

• secure
• backup

TODO

• Find out how other big projects manage multiple repositories ??
• remove unnecessary big files: remove from the repository and from the history
• split the single repository into smaller ones.
  • Allow users to download/clone only the things they care.
  • make sure they all work together and tested as a whole when needed
• prepare sample repositories to demonstrate how different types of repositories should be used

Risks and Mitigation

The biggest concern is that by having multiple repositories,

• it is hard to make sure non-core repositories always work with the core ROSE repository
• Or changes to the core repository may break other repositories without being noticed
• make sure the accurate linking between a project and the version of librose.so

How to prevent these problem?

• Commits to ROSE core repo will trigger tests of all official projects. Only commits passing all project tests will be accepted into the core repo.
• Similarly, changes to any official project will trigger tests against latest ROSE. Only commits passing all ROSE tests will be accepted into the individual project
• Better versioning of librose.so: non-official (external) projects can easily detect if a version of librose.so is compatible.

Plan

Core repository

The absolutely necessary things for 80% ROSE users should be put into the core repository of ROSE.

It should contain only

• the source files and headers used to build librose.so
• built-in tools for users to use out-of-box
• Doxygen documentation

Submodule:

• Core tests

Questions

• How about some essential documents: installation guide for example
• and some tests which are tightly coupled with some features in ROSE?

Docs repository

Why do we want to separate some documentations out of the ROSE repository?

• Right now, any changes (even fixing typos) to documentations within ROSE require the same rigid regression tests by Jenkins. This is not necessary and wasteful.

Projects repository/repositories

We may have two choices:

• Having a single monolithic repository for all projects.
• Allowing individual repository for each project. So people can just clone what they want. Each project can have its own lifecycle (abandoned, merged into rose, separated for ever, etc.)

I think in a long run, most projects should be treated as

• plugins (or augmentative library) of ROSE (librose.so).
• additional tools, as compared to built-in tools of ROSE

They are important and useful, but they somehow are not merged into the core of ROSE due to various reasons.

The current idea:

• Individual projects should be separate repositories, configurable "--with-rose=/path/to/rose/installation" to make the ROSE-to-project connection.
• Collections of all official projects should be contained in a monolithic repository (e.g. "rose-projects"), where each project is simply a submodule

To be discussed

• We may allow users to contribute projects which only use simple Makefile. Staff members or paid interns can be in charge of converting the build system to use autotools.

Tests repository

Common test input files, benchmarks we use to test robustness of the core of ROSE.

Projects can also leverage the same set of tests.