JPackage.org Build Process

Our Build process is driven by a CVS repository that, combined with some Makefiles and other scripts facilitate the workflow of package management.

For each JPP package there is a CVS module, with the SRPM name, that you can checkout from CVS with:

cvs -d :ext:jpackage@jpackage.org:/home/projects/jpackage/cvs co <package name>

Using that checkout, one drives the process by doing changes, commiting them to CVS, and then using a series of "make <target>" commands.

At the present moment, one needs accounts in the JPackage.org main server and in the build server, not to mention commit access to JPackage CVS in order to be able to maintain packages, so currently only the core JPackage members are able to use the system fully (you can still checkout, and some limited privileges can perhaps be granted). A per-package privilege will be implemented in a near future.

Acknowledgements: Besides myself, several JPackage core members helped setting this up, including Jason Corley and Deepak Bhole, but we also got lots of help from Jesse Keating, from the Fedora community, and Mike Bonnet, from Red Hat. Most of the Makefiles and scripts used are modified versions of the Koji-based setup used by Fedora, so thanks also for the people who made Koji open source. Last but not least, Joe Sauer who set up a build machine for us.

Jpackage Build 101 -- a brief tutorial

I usually check out all my packages under a directory created for just that purpose. For instance, if I want to upgrade ant, I'd do:

mkdir -p jpackage/DIST
cd jpackage/DIST
cvs -d :ext:jpackage@jpackage.org:/home/projects/jpackage/cvs co ant

You will notice two things: a directory called "common" is also checked out; there are several subdirectories under jpackage/DIST/ant

The subdirectories mean branches like JPP-5 for Jpackage 5.0, JPP-1.7 for JPackage 1.7 etc. The "devel" directory is for the yet unreleased *next* JPP release (JPackage 6.0 at the time of writing).

For the purposes of this tutorial, lets assume I want to make a change (update, fix some packaging detail, etc.) in the ant package we have for JPackage 5.0.

So, I go into that subdirectory

cd JPP-5
(I am now at jpackage/DIST/ant/JPP-5)

I can then edit my ant.spec file and make changes to it. To keep it simple, lets suppose that is all I had to do this time. When I am happy
with my changes (of course, the release number has to be bumped with any changes) I can commit them:

cvs commit -m "Fix typo in spec file" ant.spec

If this was indeed the only change, I am ready to tag this NVR (name-version-release) for building:

make tag

A tag is created that includes all files needed to create an SRPM for this NVR.

All I have to do now is build in the official JPP build server:

make build

The log will give me an idea of how the process goes. As I only fixed a typo my build will probably succeed, and will give me the path to the result files in the build server. At this point, the build results are still in the build server, and are not visible by other builds. You can log in the build server and examine the RPMs generated and the logs in that path printed out, or copy them from there with scp. When you are happy with the results and want to use them in future builds and make them available in the Jpackage repositories, you must upload.

Note: If I wanted to play with the SRPM locally first, I could use:

make srpm

and the SRPM would be added to my current directory (jpackage/DIST/ant/JPP-5).

To upload the results of the build, all I have to do is:

make rpmupload

If all goes well, the package is updated in the JPackage master repository copy and it will soon propagate into the mirrors. Although it make take some time for a yum update to see the new version of the package, any subsequent builds in the build system will see the new version I just uploaded (immediately after my make rpmupload).

And that is it! At least for this simple case.

Lets now examine things that can deviate from the trivial case, like adding a patch, replacing the source tar ball, and even adding a brand new package. Creating new branches require core members privileges, so it is described elsewhere -- if you need one created for a package, discuss that need on the list and a core member will do it for you.

Adding a text file

Test files that are needed for the build like "Patch" and other small "Source" files are kept in CVS itself together with the .spec file.

Lest say I want to add a patch for Ant. The patch file, as per convention, is something called "ant-somefix.patch".

Before I work on a package I have already checked out I always update the check out first from the package top directory (this is particularly important at the present moment as we are still enhancing the build scripts).

cd jpackage/DIST/ant
cvs up -d -P
cd JPP-5
cp ~/ant-somefix.patch .
cvs add ant-somefix.patch
vi ant.spec
- Release: <old release>
+ Release: <new release>
+Patch1: ant-somefix.patch
+%patch1
+* Tue Dec 18 2007 Fernando Nasser <fnasser@redhat.com> <new NVR here>
+- Add patch for somefix
:q
cvs commit -m "Add patch for somefix"
(the .spec file is committed and the ant-somefix.patch is effectively added)

Now, it is the same process as before:

make tag
make build

Replacing a source tar ball

Unlike the text files, any binary and large files are kept in a lookaside repository, which is still managed with help of CVS and the Makefile scripts (and some other scripts you don't see, as they live in CVSROOT).

The files stored in the lookaside repository are listed, with their MD5 signatures, in the "sources" file, that is kept together in CVS alongside the .spec file and other small text files used in the build.

Lets suppose I have an update for Ant. I download the ant source tar ball from the ant web site, check the MD5, etc. and store it in my jpackage/DIST/ant/JPP-5 directory.

To replace the old source tar ball with this new one I do:

make new-source FILES=apache-ant-1.6.5-src.tar.gz

The I edit the ant.spec file and make the changes as one does for upgrades and commit:

cvs commit -m "Upgrade to 1.7.1"
(not only the spec file, but the sources file as well will be comitted)

And, as usual:

make tag
make build

Done!

      

A new package is born: importing the SRPM for a new package

This _must_ be dome from the commons directory. If I did not had any package checked out yet (which brings commons automatically) I could obtain it with:

cvs -d :ext:jpackage@jpackage.org:/home/projects/jpackage/cvs co commons

Well, I have it, so I just go into it and make sure it is up to date:

cd jpackage/DIST/common
cvs up -d -P

I should have in my system the SRPM for the package I want to add, and as I am a nice packager, I probably buit it locally on a machine with JPP 5.0 installed and it builds fine. So I am ready to import it:

./cvs-import.sh -m "Import this new package I created for JPackage"
~/topdir/SRPMS/mynewpackage-1.2.0-1.jpp5.src.rpm

Note that I've added the package to the CVS repository, not to my CVS sandbox. I have to checkout to get it:

cd ..
(now at jpackage/DIST)
cvs -d :ext:jpackage@jpackage.org:/home/projects/jpackage/cvs co
mynewpackage
cd mynewpackage
(now at jpackage/DIST/mynewpackage)

You will notice that there is only the devel directory in there, which
makes sense, if it is a new package should be built for the next release, not for one already released. In rare ocasions, it may be necessary to add a new package to an old release as a new dependency to a package that must be updated for security reasons, for instance. Just ask for the branch you need (explaining the reason to add a package to a released JPackage release) in the discussion list.

Note: We are having to work in parallel on JPP 5 and 6, so exceptionally we are doing the JPP 5 work on a JPP-5 branch, not in devel, which will be used for JPP-6. Just ask in the list for the JPP-5 branch if it is a new package for the yet to be release JPackage 5.0.