Creating a Nightly Build
The Nightly Build
- SSH Account for the AROS Project on sourceforge.net
- Space to store all the required source for your chosen packages,
- Space to store the generated binaries and log files.
- A valid svn account for https://svn.aros.org/ with read-access.
- An email account subscribed to the AROS dev mailing list.
- The location the nightly scripts are stored in
- <Sourceforge SSH Account Name>
- A valid SSH account on sourceforge with access to AROS.
- <Nightly-Build User Name>
- The local user account you will build binaries with.
- <Dev Mailing List eMail Address>
- An email account with access to the AROS Dev mailing list.
- This will be used to send the log to the mailing list.
- <Your Servers Config-Name>
Nightly Package Names
The build script uses package names to identify what to build for a given server. the packages tell the build script which files in <SRCDIR>/pkg/ to run, which in turn creates the chosen binaries.
Valid Package Names
Files which are common to all platform generally have a name that is just their target, with all as the build target. Currently defined ones are
Common Package Names
sources.svn.all changelogs.svn.all documentation.svn.all
Arch Specific Package Names
sdk.svn.pc-i386 sdk.svn.pc-x86_64 contrib.svn.pc-i386 contrib.svn.pc-x86_64 bootiso-grub.svn.pc-i386 bootiso-grub2.svn.pc-i386 bootiso-grub2.svn.pc-x86_64 bootfloppy.svn.pc-i386 hosted.svn.linux-i386 hosted.svn.linux-x86_64 hosted.svn.linux-ppc
bootiso generic routine used by the arch specific nightly targets hosted sdk
Prepairing the system
Firstly you will want to set up a user account on your system that will be used to perform the build. create a user
adduser <Nightly-Build User Name>
The first step is to obtain the nightly build scripts. cd to the location you will perform the build in and checkout the scripts as follows -:
cd /home/<Nightly-Build User Name> mkdir -p MyNightlyBuildDir cd MyNightlyBuildDir svn checkout https://svn.aros.org/svn/aros/trunk/AROS/scripts/nightly
you will have a new subdirectory called nightly that contains all the scripts used to build the nightly binaries. from here on in we will refer to this directory as <SCRIPTDIR>.
now cd into that directory and make a directory tree to hold the AROS source
cd <SCRIPTDIR> mkdir -p Source mkdir -p Source/Contrib mkdir -p Source/Documentation mkdir -p Source/Necessary
Finaly create an initial cache of the svn files used to build AROS
IMPORTANT: You must set the correct LOCALE for your system at this point!!
type the following -:
cd Source export LANG=en_US.ISO8859-1
then checkout all the source repositories needed to perform your nightly build packages.
svn checkout https://svn.aros.org/svn/aros/trunk/AROS cd Contrib svn checkout https://svn.aros.org/svn/aros/trunk/contrib cd ../Necessary svn checkout https://svn.aros.org/svn/aros/trunk/contrib/necessary
You may omit complete contrib if you don't want to bundle it within your archive. Just leave the corresponding directory empty.
Prepairing your servers config file
Create a plain textfile in the cfg directory called <Your Servers Config-Name> Open it in a text editor and add the packages you'd like to build.
For example ..
CFG_PACKAGES=(sources.svn.all documentation.svn.all sdk.svn.pc-x86_64 bootiso-grub2.svn.pc-x86_64)
To enable sending an email report of the build results to the dev mailing list add the following lines ..
CFG_MAIL_ENABLE=yes CFG_MAIL_CC=<Dev Mailing List eMail Address>
And to enable uploading the binaries so they may be listed on AROS.org add ..
CFG_UPLOAD_ENABLE=yes CFG_DS_USER=<Sourceforge SSH Account Name> CFG_BACKUP_ENABLE=no
Performing your first run
The first time you check the system is operational ... perform a manual dry run by doing the following
cd <SCRIPTDIR>; ./build <Your Servers Config-Name>
And watch as it displays all your build output.
If everything went well and you have enabled the email logging in your server config, you should see a confirmation email on the AROS developer mailing list confirming the build completed successfully. If you have also enabled uploading the files, and have a valid AROS SSH account on sourceforge, the binaries should be uploaded into the temporary storage location ready to be moved into the live folders on the AROS.org download pages
Setting it up to run automatically
You will need to create a cron job for the build step we performed doing the dry run. You will want to add a command that is performed once every 24hrs, which runs the build script with your server's configuration and uploads the created archives to AROS.org.
0 2 * * * ( cd <SCRIPTDIR> && ./build <Your Server's Config-Name> ) > /dev/null
This will make the build machine attempt to build and upload AROS at 2am. All output will be directed to NULL (except of course the generated logfiles)
Adding the binaries to the AROS.org site
If you are simply providing binaries for existing AROS targets, the previous steps should be enough to add the files to the AROS.org site. If however you wish to provide new flavours of AROS then you will need to update the AROS.org site with information about them. Read on...
The scripts and page definitions to list the binaries on AROS.org are maintained in svn at https://svn.aros.org/svn/aros/trunk/documentation
To edit the files you'll need to checkout the repository and customise to suit.
listing new downloads
If you are providing binaries for AROS targets that don't currently exist, you will need to update the files which list them on AROS.org.
Edit the db/download-descriptions/<Language> files to list descriptions of the packages you build. make sure and follow the naming convention used by the existing packages.
Corrupt localised filenames in the binaries
This usually means the source files used have utf8 filenames in the source repository caches resulting from the initial checkout being done when the system locale was set to utf8.
to fix it change directory into each source directory in turn and delete all files and directories EXCEPT the top level .svn directory.
now type ..
.. and finally do an "svn update" in each directory.
Maintaining the Main Nightly Build Machine
The main nightly build machine has the additional responsibility of running scripts to move the uploaded binaries to the live AROS.org download pages and clean up the upload directory. Other build machines don't need to run these scripts.
Moving the uploaded binaries to the live AROS.org download pages.
You will need to create a cron job that runs at regular intervals during the day, currently every hour. This job however isn't performed locally, and will require an AROS SSH account on sourceforge. Its goal is to move the archives uploaded by the various build machines into the correct folder on the AROS.org site so that they become available for download. It also deletes any old versions of these files so that only the most up-to-date version is available (there isn't enough space to keep multiple versions of each file).
00 * * * * ssh <Sourceforge SSH Account Name> 'bash -l -c aros/scripts/nightly-update' > /dev/null
This will make the server transfer any files present in the upload dirextory to the AROS.org site once an hour. Again all output is sent to NULL.
This step removes incomplete or broken nightly build files from the upload directory.
51 7 * * * ssh <Sourceforge SSH Account Name> 'bash -l -c aros/scripts/nightly-clean' > /dev/null
This will clean up the uploads directory on the AROS.org server at 7:51am each day.