Centralising your tools in a custom repository - Part 1

At almost every software company I’ve been involved in, used custom-made tools for various tasks. These tools range from simple shell-scripts for search&replacing data to large deployment-script or even programs that take care of administrative tasks like monitoring, log aggregation and so on. At the good companies, these tools are maintained inside a software repository system SVN or GIT. But tools like these need to be deployed to a lot of different server and development environments, both in internal and external networks. Creating SVN checkouts is a possibility, but what about things like dependencies or easy up- or downgrades. And do you want external systems give access to your source code repository? And how do you make sure everybody uses the latest release with the newest features and bug fixes?

One of the best answers is something almost every Linux user is already familiar with: package managers. They are easy to use, it’s fast and can do so much more work for you than simple repository checkouts ever can do. Using Linux packages managers as the preferred way of deployment for software is not yet discovered by the php-community. Strange actually, since package managers are capable of handling issues that most deployment-tools are struggling with like dependencies, upgrades, downgrades and multi-environment setups.

Since this subject is pretty large to handle in a single blog post, it is setup in 3 different chapters. First, we begin by creating a package from scratch. In the next chapter, we show how to collect and maintain these packages inside your customer repository and in the last chapter we will talk about connecting your software repository to your package manager and how to deal with multiple repository-formats.

Packages and their repositories

A package consists of a collection of files, (de)intialization scripts and some global information. When you install a package, it will copy the package-contents to your system (see it as unzipping an archive), and afterward (or even before) it can run some initialization scripts. For instance, a mysql-server package can create a mysql-user on your system, initialize databases, set root-passwords etc.

A package repository consists of packages which can be automatically downloaded by all kind of different package tools on various platforms. These tools will find out which packages there are inside a package repository, what other packages needs to be installed before this package functions (it’s dependencies), what a package provides, if there are any updates for a specified package etc. The nice thing about this system is that with a few very simple commands you can control all the software updates on your systems without any hassle.

The closest match for somebody not familiar with package managers is PHP’s PEAR and PECL libraries. You can download software from the PEAR library and it automatically installs everything it needs in order to function. When updates are available, you can easily update the packages.

A “normal” package repository is just like PEAR, but just for all kind of software (and other things, which you will find out later on).

However, there are different kind of repository and package formats. The examples we use here are for the RPM format which is used for RedHat, Fedora, CentOS and others. The “other” big flavor out there is DEB, which is used in Ubuntu and Debian. In the last chapter we will talk about mixing both repositories.

Setting up your source

Before setting up a package repository, you must have packages you want to place inside. The first thing you must do before you can create a package is gather all the files and place them all inside a temporary directory. The structure of this directory does not matter but I like to have it clean and simple, something like this:

/src   (source code tarballs)
/doc   (documentation)
/man   (man pages)
/bin   (binary files)
/conf  (configuration files)

As said, the structure doesn’t really matter. In this article, I will try to create a example package : changecase. This is a (very simple) shell script that will change the case of the default input. It’s a simple shell-script without any bells and whistles in order for you to understand the concept of packaging.

$ echo "HelLLO World" | changecase

outputs

HELLO WORLD

or

hello world

when the setting TO_UPPER is set to 0 in the /etc/changecase.conf

It has absolutely no real-life purposes, since this can be achieved must easier with other tools, but at least it will show you how packaging works.

You can find the source code for this tool at github: https://github.com/jaytaph/changecase

It has got a simple shell-script in the /bin directory, a manual page in the /man directory and some documentation.

Creating a spec file

The most important file you will need is called the .SPEC file. This file will be the main configuration of your package and contains everything: the information about the package, the commands to run in various stages while building the packages and which files actually will be packed and where they will be placed.

The example .spec file (changecase.spec)

Summary: NoxLogic Case Changer
Vendor: NoxLogic
Name: changecase
Version: 1.0
Release: 1
Source0: changecase-%{version}.tar.gz
License: GPL
Group: NoxLogic
BuildArch: noarch
BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot
Requires: coreutils
%description
This is a simple script that will change the case of the input.

%prep
%setup

%install
mkdir -p $RPM_BUILD_ROOT/usr/local/noxlogic/bin
mkdir -p $RPM_BUILD_ROOT/usr/share/man/man1
mkdir -p $RPM_BUILD_ROOT/etc
cp bin/changecase $RPM_BUILD_ROOT/usr/local/noxlogic/bin
cp man/changecase.1 $RPM_BUILD_ROOT/usr/share/man/man1
cp conf/changecase.conf $RPM_BUILD_ROOT/etc/changecase.conf

%clean
rm -rf $RPM_BUILD_ROOT

%files
%doc doc/*
%dir /usr/local
%dir /usr/local/noxlogic
%dir /usr/local/noxlogic/bin
%config /etc/changecase.conf
/usr/share/man/man1/changecase.1.gz
/usr/local/noxlogic/bin/changecase

Most of these fields are pretty self explanatory, but we will highlight a few:
Name
This is the name of the software package.

Version
This is the main version of your package. As soon as you have a new release you should update the version number. This will result in a new rpm file: (rpmfile-X.Y.rpm).

Release
If you have a small release (bug fix for instance), you can opt for increasing the release. This will result in a rpmfile-X.Y-Z.rpm

Source / Source0
This is the source file that will be used for creating the package (will be explained later). If you like, you can have multiple source files (source1, source2 etc). The 0 on source0 is optional, so you could also use source as the directive.

BuildArch
Depending on your software, it can be either binaries which are architecture-specified (x86-64bit, 32bit, powerpc etc), or when you have shell-script, php files or things that are not specified for a architecture, you can use noarch.

Requires Since this shell script requires the “tr” program, we can require the package that holds this program. The program can be found in the coreutils package. If a system does not have this package installed (and thus, does not have the “tr” program), install our package will trigger installation of the coreutils package so we know our program will function properly.

Other uses could be that a “php5-mysql” package which holds the mysql-extension for PHP would require both the PHP package and the “mysql-client library” packages. This is one of the biggest advantages of installing packages: dependencies will be automatically be added and installed for you.

%prep, %build, %install, %clean
These are the 4 “steps” that are involved in compiling binaries from source codes. When dealing with packages that doesn’t require compilation (for instance, shellscripts or php scripts). You could skip most of these steps. The %clean will be used to cleanup the build environment and should most of the time something like this:

rm -rf $RPM_BUILD_ROOT

It should be noted that these 4 commands are only executed when CREATING a package, not when INSTALLING it.

In our spec-file, we use a macro in the %prep phase named %setup. This automatically untars our sourcecodes and sets everything up which saves us time typing all those commands ourself. But if you need to setup in a different way, feel free to use your own way.. RPM is flexible..

%pre, %post, %preun, %postun

These scripts gets executes when you as an end user installs (or uninstalls) a package. Before installation of package, the %pre will be run. It’s not really used a lot since most of the things will be done in the %post. The %post will be run AFTER installation. They can be used for adding new users to the system, creating caches, setting up symlinks etc. %preun and %postun are executed before and after uninstalling a package. It normally will revert everything that is done inside the %pre and %post (removing users, removing caches etc).

%files
This will be the place where we will decide which files needs to be added to the package and where it will be places when installing a package.

%files
%doc doc/*
%dir /usr/local
%dir /usr/local/noxlogic
%dir /usr/local/noxlogic/bin
/usr/share/man/man1/changecase.1
/usr/local/noxlogic/bin/changecase
%config /etc/changecase.conf

This will do the following:

1. Add all files in the ‘doc’ directory to the documentation directory. This is most likely /usr/share/doc/-x.y
2. Create a directory /usr/local (if it does not exists already)
3. Create a directory /usr/local/noxlogic (if it does not exists already)
4. Create a directory /usr/local/noxlogic/bin (if it does not exists already)
5. Copy the /etc/changecase.conf into the /etc. This will take care in case the file is already present or modified etc.
6. Copy ‘changecase’ into the /usr/local/noxlogic/bin directory
7. Copy the manual page into the manual directory (there is no special directive for this, which is a shame I guess)

Steps 2-4 ensures us that the directory we need to place files into actually exists. Since it doesn’t create parent directories, we need to add directory by directory.

Building the package

To build the package, you must go through a few steps:

First you have to make sure you have the rpm-development tools installed on your system. Not surprisingly, this only works on RPM (YUM) based systems like centos, fedora or redhat:

yum install rpm-devel

It shouldn’t be a supprise to anyone that all these buildtools are packaged as well.

The next step is to create a “build environment” for your RPMs. This can be done by creating a few directories:

mkdir -p ~/buildroot/{BUILD,RPMS,SOURCES,SPECS,SRPMS,tmp}
mkdir -p ~/buildroot/RPMS/{i386,i686,noarch}

The buildroot is pretty simple but we need rpmbuild to give some information about our new buildroot. Create a ~/.rpmmacro file with the following contents:

%packager Joshua Thijssen
%_topdir ~/buildroot
%_tmppath ~/buildroot/tmp

This effectively tells rpmbuild program that you are the packager (me, in this case), and we set the top directory and a temporary (build) directory. There are more things to place into this file, but it will suffice for now.

Next up, tar sources into SOURCES.

This step will be a bit tricky: the source(0) directive in the .SPEC file, decides which source-files it needs. In our case, it’s changecase-1.0.tar.gz (the {$version} tag can be found a few lines before the source-directive). This means that every major or minor build needs a separate source file. (you don’t have to do this, if you don’t want to, but I like it this way).

You can tar the file like this:

• Copy the changecase files into the directory changecase-1.0
• Create a tarball by issuing: tar cvzf changecase-1.0.tar.gz changecase-1.0

After creating the tarball, you need to copy this into the SOURCES directory.

The next step is to copy the changecase.spec file from the source into the SPECS directory.

After all these steps, you can try to build your first RPM-package:

cd ~/buildroot
rpmbuild -v -ba SPECS/changecase.spec

It will show you all kind of info, and if everything worked correctly, you have got the following files:

• SRPMS/changecase-1.0-1.src.rpm (the source rpm package)
• RPMS/noarch/changecase-1.0-1.noarch.rpm (the actual (binary) package)

The source package allows you to recreate the rpm. Normally, this means it is a simple package with the .spec file and the actual source tarball(s). You don’t really need it but it allows others to create the rpm in for instance other formats.

The RPM/noarch directory holds the actual rpm package. It has been placed in the /noarch directory since the specfile told the builder that there is no specific architecture needed (binaries compiled for i386 processors would be placed in /i386, which allows us to create binaries specifically for your processor/architecture).

Displaying information about the package can be done with the following command:

jthijssen@guybrush$ rpm -qip changecase-1.0-1.noarch.rpm
Name        : changecase                   Relocations: (not relocatable)
Version     : 1.0                               Vendor: NoxLogic
Release     : 1                             Build Date: Mon 22 Nov 2010 01:26:10 PM CET
Install Date: (not installed)               Build Host: guybrush.noxlogic.nl
Group       : NoxLogic                      Source RPM: changecase-1.0-1.src.rpm
Size        : 557                              License: GPL
Signature   : (none)
Packager    : Joshua Thijssen
Summary     : NoxLogic Case Changer
Description :
This is a simple script that will change the case of the input.

Manual Installation

When you have finally created and signed your package, it’s time to let it out into the world. Since we have to wait until the next chapter to setup a repository, we will manually install the package onto your system.

rpm --install changecase-1.0-1.noarch.rpm

This will install the pacakge onto your system. To verify that your package is actually installed try to following:
yum list

Updating your pacakge is almost as easy:

rpm --upgrade rpmfile.rpm

If you do not need your sofware anymore, removing it can be done with the following:

rpm --erase <packagename>

Note that we have to use the package NAME instead of the RPM filename when erasing the package from your system. If there are files made by your software, or if you have made modifications in configuration files, those files will NOT be removed.

Signing your packages

When your packages will be publicly available, I advice you to sign the packages. This way we can make sure that the (signed) packages really originates from you and that they are not tampered with by a third party. Since signing is so easy, I’d recommend signing the packages even if you are not planning to deploy them publically.

Creating a GPG key

Before you can sign packages, you must have a (secret) key. Creating the key is simple:

gpg --gen-key

Make sure you add a pass-phrase. Adding one will make sure that if your key gets compromised, nobody will be able to sign packages with it since they don’t know the correct pass-phrase. After creating a key, you must export (and armour) the key so others can validate against it:

gpg --export --armour jthijsen@noxlogic.com > RPM-GPG-KEY-jthijssen

This file must be exported to the server where you install your packages (read the last chapter on how to automate this).

You should add your signing name and application into your .rpmmacros file

%_signature gpg
%_gpg_name Joshua Thijssen <jthijssen@noxlogic.com>
%_gpgbin /usr/bin/gpg

To sign a package, just use:

rpm --addsign changecase-1.0-1.noarch.rpm

When issuing a rpm -qip changecase-1.0-1.noarch.rpm again, you’ll see that the “signature” has changed:

Name        : changecase                   Relocations: (not relocatable)
Version     : 1.0                               Vendor: NoxLogic
Release     : 1                             Build Date: Mon 22 Nov 2010 01:26:10 PM CET
Install Date: (not installed)               Build Host: guybrush.noxlogic.nl
Group       : NoxLogic                      Source RPM: changecase-1.0-1.src.rpm
Size        : 557                              License: GPL
Signature   : DSA/SHA1, Mon 22 Nov 2010 01:46:08 PM CET, Key ID 2194ae1c159456fa
Packager    : Joshua Thijssen
Summary     : NoxLogic Case Changer
Description :
This is a simple script that will change the case of the input.

Conclusion

This concludes the first chapter of creating a package repository. We just barely scratched the package/.spec surface but it should be enough to get you going. A very good page of find more information about RPM and the SPEC standards can be found here.

Watch out for part 2, where we will finally setup a repository where we can add packages too. For now, find your tools you want to package, create spec-files and build the packages and install them away!

Read more in part 2