Introduction

This document is a LFS hint.

AUTHOR: Pierre Hebert
DATE: 2007-07-31
LICENSE: GNU Free Documentation License Version 1.2
SYNOPSIS: TRIP, a TRIvial Packager for LFS (and possibly other linux systems)
DESCRIPTION: Trip is a trivial package manager. Its aim is to provide a secure way to build and install packages from sources, without hassle and with as few dependencies as possible. Trip will record all created, modified or deleted files during the "make install" step, while preventing the root filesystem to be altered. This fine and safe monitoring has been made possible thanks to unionfs. Trip make use of unionfs when constructing a package. Basically it puts a temporary filesystem on top of the root filesystem. This temporary filesystem will receive a copy of all created/modified files, while the underlying root filesystem remains untouched. In no case you will overwrite a file by mistake, or create a conflict without being alerted. Trip package format is almost as simple as possible : it consists in a tarball, meaning that you can always verify and extract its content with common tools. Trip is a bash shell script and does not rely on other tools than those already present in the /tools directory when building LFS... and unionfs, which is at time of writing still not in mainline Linux kernel. Fortunately this module is widely available though. Trip has been designed with LFS in mind, but can also be used on other systems.
PREREQUISITES:
- some knowledge about the linux/unix problematic of package management (suggested reading among other documents : http://www.linuxfromscratch.org/hints/downloads/files/fakeroot.txt from Tushar Teredesai)
- a linux 2.6 system with unionfs >= 1.1.3
- optional : you should have already built a LFS system yourself, in case you plan to use trip to build and package a new LFS system. Some explanations about this process are given later in this hint and may be enough, but the LFS book is far better written and more instructive.

What trip is, and what trip is not

Trip is a simple package manager, mainly designed for LFS users. It is a quite low level tool, probably with bugs and rough edges. It will safely monitor file alterations during compilation and installation from a source package. It will manage packages with tarball and text files. It will let the user do what it wants to do, no matter if that is a mistake or not.

Trip is not RPM nor DPKG (to name only two). It will not fetch packages automatically for you from internet. It will not check dependencies. It will not split packages into various pieces automatically (executables, devel, libs, etc.). It will not allow you to upgrade your whole system automagically.

All that said, trip won't be in you way when you decide to do something, and trip will not require tons of dependencies to run.

Trip usage in short

Trip -h should give something like that :


+ Trip, a Trivial Package manager, using Unionfs. You are using version 0.3
+ usage : trip -i,   --install <binary package file> [--no-conflict]
+                    (install from a pre-built binary package)
+              -u,   --uninstall <package name>
+                    (uninstall a package installed with trip)
+              -b,   --build <source package directory>
+                    (create a binary package from sources)
+              -r,   --rebuild <package name>
+                    (rebuild a binary package from an installed package)
+              -w,   --wizard
+                    (ask some questions, create a source package, build it and install it)
+              -l,   --list [<package name>|<binary package file>]
+                    (list installed packages, files from an installed package or files from a binary package)
+              -f,   --find-file <file>
+                    (find the package(s) containing "file")
+              -bbi, --batch-build-install <package list>
+                    (build and install a set of packages from "package list")
+              -bu,  --batch-uninstall <package list>
+                    (uninstall a set of packages from "package list")
+                    --upgrade-bd-from-0.1-to-0.2
+                    --upgrade-bd-from-0.2-to-0.3
+                    (migrate data from a previous package database format)
+              -c,   --config-dir <dir>
+                    (specify an alternate configuration directory)
+              -t,   --trip-path <path>
+                    (specify an alternate path to this trip shell script)
+              -v,   --verbose-level <0|1|2|3>
+                    (verbosity where 0=quiet, 1=errors, 2=errors/warnings, 3=errors/warnings/infos)
+              -h,   --help
+                    (print this short help, see also http://www.pierrox.net/trip/ for more)

The essential switches are -b, -i, -u : build, install and uninstall a package.

trip -b : will build a binary package from a source package, while making use of unionfs to monitor created/modified/deleted files during the "make install" step. The argument is the path (absolute or relative) to the source package. The result (unless compilation or installation failure) is a trip binary package containing created files, and eventually modified files. The binary package is created into the current directory. (see "Trip package and database format" below).
trip -i : will install all files from a binary package (previously built using trip -b) into the running system. Trip will refuse to overwrite a file if it already exists, unless you specify --no-conflict. After a sucessfull install an entry in the trip package database is added.
trip -u : remove files previously installed by trip, and also remove the database entry. Directories belonging to a package are removed only if non empty.

Additionally trip provides some other usefull switches :

trip -r rebuild a binary package from the description found in the package database and files present on the system. Such a reconstructed package may not be exactly like the one used originally (in case some files have been modified since the install), but it is a handy way to backup a package before uninstallation. It will be possible to reinstall it later with trip as a normal package if needed.
trip -w will help the user to create a source package, build it and finally install it. This is a step by step process where questions like package name, version, source file location, etc. are asked interactively to the user.
trip -l with no argument gives the list of already installed packages, sorted by ascending installation date. Trip -l will give the list of files belonging to an already installed package (package name can be either "foo" or "foo-1.2.3"). Trip -l will do the same for trip binary package. In these both cases the output will be affected by the verbosity level. At level 3 the output will be the one of "tar --list --verbose". At any other verbosity level trip will give the list of files, a file by line.
trip -f will tell the user which package(s) contains the named file.
trip -bbi build and install a set of packages. The argument is the name of a file containing the list of packages to build and install. Each line of the file is a path to a source package, the same way it should be used for "trip -b". The switch "--no-conflict" can be used in order to ignore potential conflicts during install. It is not recommended however to use this switch unless you know that these conflicts can be safely discarded.

Finally the behaviour of trip can be impacted by the use of these switches :

-c allows the user to specify an alternate configuration directory. This can be usefull for test purpose or for managing different set of installation.
-v will set the verbosity level. Default is 0 (quiet). Currently all messages from trip are thrown on stdout. The output of build and install steps during a package build is not concerned by -v.
-t is used to specify the path of the trip schell script. The path of trip is needed when entering chroot. It should be specified in the trip configuration file but may also be overriden by this command line option.

Trip internals

The idea of trip is to use two filesystems in a union filesystem. See http://www.unionfs.org/. One filesystem is the target filesystem on which the user wants to do "make install", and is used read-only, the second is an empty filesystem merged with the first, and used read-write.

Let's take an example fstab :


/dev/hda1      /            ext3   defaults                           1    1
/dev/hdc1      /mnt/pkg     ext3   defaults                           0    0
unionfs        /mnt/union   unionfs dirs=/mnt/pkg:/=ro,noauto         0    0

The result of this is :

if we read a file in /mnt/union, it will be read from /, if it doesn't exists in /mnt/pkg
if we create a file in /mnt/union it will be really created in /mnt/pkg
if we modify a file in /mnt/union belonging to / it will be copied and modified in /mnt/pkg
if we delete a file in /mnt/union belonging to /, a special file will be created in /mnt/pkg (whiteout mode)
the overall behaviour of the program executing in /mnt/union is as if there were only one normal filesystem.

Hence a chroot in /mnt/union gives these benefits :

in no case the / filesystem will be altered
created, deleted and modified files are all stored in /mnt/pkg
all libraries and tools are available in their real prefix, running ./configure --prefix=/usr is safe, a standard make install is enough.

This principle being expressed, the work of trip is to manage the execution of build/install steps of a source package, and put all created/modified files in a tarball, so that they can be copied later into the real filesystem, if everything goes well.

Trip installation

Installing trip involves two steps.

Configure the unionfs mount point in /etc/fstab

first create mount points :
```
mkdir -p /mnt/pkg /mnt/union
      
```
then edit your fstab in order to add this two lines :
```
/dev/hdc1      /mnt/pkg     ext3   defaults                           0    0
unionfs        /mnt/union   unionfs dirs=/mnt/pkg:/=ro,noauto         0    0
  
```
Modify /dev/hdc1 according to your system. The option "dirs=/mnt/pkg:/=ro" means that /mnt/pkg and / are layered, and as / is specified as read-only (ro), all file modifications will take place into /mnt/pkg.
Finally test your setup :
```
mount /mnt/union && ls -l /mnt/union
  
```
(you should see the content of / and /mnt/pkg merged)

WARNING : before to build a binary package, trip removes all files in /mnt/pkg.

WARNING : Trip wont work as expected if your system is based on several filesystems, for example / and /usr. In this case /usr will not be visible inside /mnt/union and so a lot of things will probably not work. Work is in progress to drop this limitation.

Download and install trip (using trip itself of course !)


TRIP=trip-0.3
wget -O - http://www.pierrox.net/trip/$TRIP.tar.gz | tar xvz
./$TRIP/src/trip -v 3 -c $PWD/$TRIP/src/conf -t $PWD/$TRIP/src/trip -b $TRIP
./$TRIP/src/trip -v 3 -c $PWD/$TRIP/src/conf -i $TRIP-1.noarch.tar.gz

This will dowload trip, construct an installable trip package from the trip source package, and finally install the package. You can have a look at the list of installed files using trip -l trip. You may have to edit $TRIP/src/conf/conf if you named the mount points other than /mnt/pkg or /mnt/union.

Using trip to package LFS from the beginning

Trip as been designed from the ground to build a packaged LFS, starting at the creation of directories till the last package Vim. LFS-6.2 is used as an example, but the process may be applied to other versions of LFS, even CLFS (tested on ARM).

Prerequisites

Follow the LFS book till the chapter "6.1 Introduction". At this stage the $LFS/tools directory is ready.

Temporary trip installation and configuration

When used to build a LFS system, trip requires a special configuration and directory layout. As trip must be reachable from $LFS the same way it would be from /, it will be put in $LFS/trip and a symlink will be created into / (the same way it has previously be done for $LFS/tools). Once the new LFS system is up and ready, trip can be installed and used as explain in the previous paragraph "Trip installation". For now use these commands to install trip in a temporary way :


TRIP=trip-0.3
wget -O - http://www.pierrox.net/trip/$TRIP.tar.gz | tar xvz -C $LFS
mv $LFS/$TRIP $LFS/trip
ln -s $LFS/trip /trip
mkdir /trip/tmp /trip/pkg

This will download trip and copy files into a temporary location into $LFS. The trip shell script is now located at /trip/src/trip, and the configuration directory is now located at /trip/src/conf.

Finally edit /trip/src/conf/conf and use these values, where path are adjusted to your $LFS :


# TRIP configuration file
# TRIP is a TRIvial Packager
# copyright 2006-2007 Pierre Hebert 
# http://www.pierrox.net/trip/


# the real root filesystem, in which resides the operating system
TRIP_FS_ROOT=/mnt/lfs

# the filesystem on which are temporarily created the files during the install phase
TRIP_FS_PKG=/mnt/pkg

# the union filesystem mergin $TRIP_FS_ROOT(ro) and $TRIP_FS_PKG(rw)
TRIP_FS_UNION=/mnt/union

# where to install binary packages
TRIP_INSTALL_DIR=/mnt/lfs

# installed package db location
TRIP_DB=/mnt/lfs/var/lib/trip

# the mode tell if we use /tools or / as build environment (hosted = build from /tools, normal=/)
TRIP_MODE=hosted

# the location of the trip shell script
TRIP_PATH=/trip/src/trip

# root location of temporary files
TRIP_TMPDIR=/trip/tmp

Download trip source packages for LFS


cd /trip/pkg
wget -O - http://www.pierrox.net/trip/source-packages-lfs-6.2.tar.gz | tar xvz

Prepare virtual filesystem and some essential link to the shell


mkdir -p $LFS/{dev,proc,sys,bin,tmp}
mknod -m 600 $LFS/dev/console c 5 1
mknod -m 666 $LFS/dev/null c 1 3
ln -s /tools/bin/bash $LFS/bin/bash
ln -s bash $LFS/bin/sh

Use trip to build and install packages

Running trip

When using trip during the LFS system building, trip can be executed this way :


/trip/src/trip -c /trip/src/conf -b foo-1.2.3

Alternatively some environments variables can be adjusted to simplify the command line :


PATH=$PATH:/trip/src
export TRIP_CONFIG_DIR=/trip/src/conf
trip -b foo-1.2.3

(We will use this second form)

Build the first package

Now that your host system is ready to build the new LFS system using trip, it is time to start the easy job. Issue the following command to construct the very first package :


cd /trip/pkg
trip -b lfs-base-6.2

It will create a package named "lfs-base-6.2-1.noarch.tar.gz" in the current directory, containing the standard directory structure. Install it with :


trip -i lfs-base-6.2-1.noarch.tar.gz

Build and install linux-libc-headers-2.6.12.0 and man-pages-2.34


trip -b linux-libc-headers-2.6.12.0 && trip -i linux-libc-headers-2.6.10-1.i386.tar.gz
trip -b man-pages-2.34 && trip -i man-pages-2.34-1.noarch.tar.gz

(if you don't build for i386 architecture edit the linux-libc-headers-2.6.12.0 package)

Create some essential symlinks


    ln -s /tools/bin/{cat,grep,pwd,stty} $LFS/bin
    ln -s /tools/bin/perl $LFS/usr/bin
    ln -s /tools/lib/libgcc_s.so{,.1} $LFS/lib

Build and install glibc-2.3.6


    trip -b glibc-2.3.6 && trip -i glibc-2.3.6-1.i686.tar.gz

(replace i686 with your architecture)

Re-adjusting the toolchain


    chroot "$LFS" /tools/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$ ' \
    PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin /tools/bin/bash --login +h
    mv /tools/bin/{ld,ld-old}
    mv /tools/$(gcc -dumpmachine)/bin/{ld,ld-old}
    mv /tools/bin/{ld-new,ld}
    ln -s /tools/bin/ld /tools/$(gcc -dumpmachine)/bin/ld
    gcc -dumpspecs | \
    perl -p -e 's@/tools/lib/ld-linux.so.2@/lib/ld-linux.so.2@g;' \
         -e 's@\*startfile_prefix_spec:\n@$_/usr/lib/ @g;' > \
          `dirname $(gcc --print-libgcc-file-name)`/specs
    exit

Build and install all other packages

We will use the "--batch-build-install" option to automate the process :


cat << EOF > pkg_list
binutils-2.16.1
gcc-4.0.3
db-4.4.20
coreutils-5.96
iana-etc-2.10
m4-1.4.4
bison-2.2
ncurses-5.5
procps-3.2.6
sed-4.1.5
libtool-1.5.22
perl-5.8.8
readline-5.1
zlib-shared-1.2.3
zlib-static-1.2.3
autoconf-2.59
automake-1.9.6
bash-3.1
bzip2-1.0.3
diffutils-2.8.1
e2fsprogs-1.39
file-4.17
findutils-4.2.27
flex-2.5.33
grub-0.97
gawk-3.1.5
gettext-0.14.5
grep-2.5.1a
groff-1.18.1.1
gzip-1.3.5
inetutils-1.4.2
iproute2-2.6.16-060323
kbd-1.12
less-394
make-3.80
man-db-2.4.3
mktemp-1.5
module-init-tools-3.2.2
patch-2.5.4
psmisc-22.2
shadow-4.0.15
sysklogd-1.4.1
sysvinit-2.86
tar-1.15.1
texinfo-4.8
udev-096
util-linux-2.12r
vim-7.0
lfs-bootscripts-6.2
lfs-configuration-6.2
EOF

trip --no-conflict --batch-build-install pkg_list

Note that we use the "--no-conflict" option because the installation of some packages overwrite manually created symlinks, like cat,grep,pwd,stty.

Final steps

The "shadow" package requires some manual configuration :


chroot "$LFS" /tools/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$ ' \
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin /tools/bin/bash --login +h
pwconv
grpconv
rm /etc/{passwd-,group-,shadow-,gshadow-}
passwd root

You also need to build your kernel. It is possible to package linux, but as you will probably want to choose your own configuration depending on your hardware, it is probably easier to keep it under a well known location like /usr/src/linux. Follow the LFS instructions about it.

The last two packages lfs-bootscripts-6.2 and lfs-configuration-6.2 contains some default configurations that you will certainly want to change. So have a look at the content of these packages in order to adapt them to your needs. At this point come back to the LFS book at "7.5. Configuring the setclock Script" for the last steps. You will in particular need to :

edit /boot/grum/menu.lst
edit /etc/fstab
run grub

Trip package and database format

Source package

The source package describes how to build and install a software. Its name should be composed on the <name>-<version> schema, for example "glibc-2.3.6". Inside the package directory are located two subdirectories : "src" and "support".

src : contains required source files, eventually some patches or default configuration files. More generally it contains every input materials needed to build the package.
support : contains package meta data, instructions on how to build and install the source package, and eventually some additionnal scripts :
- identification : this is a list of variables describing the package
  - TRIP_NAME : for example glibc
  - TRIP_VERSION : for example 2.3.6
  - TRIP_RELEASE : used to distinguish between two different packages of the same version
  - TRIP_URL : where to find the package
  - TRIP_DESCRIPTION : short comment about the package
  As the file "identification" is meant to be sourced by bash, you should take care of syntax, especially quotes and escape some characters.
- build.sh : this optionnal shell script should be executable. It contains instructions on how to build the package (something like tar xzf $TRIP_NAME-$TRIP_VERSION.tar.gz && cd $TRIP_NAME-$TRIP_VERSION && ./configure && make)
- install.sh : this file is mandatory and should be executable. It contains instructions on how to install the package (something like make install)
- {pre,post}_{un,}install.sh : these four files are optionnal and should be executable. If present, they will be incorporated into the binary package, and executed during installation of the binary package. They can be used to start a daemon after installation, or stop it before removal of the package, for example.

The build.sh and install.sh scripts are executed in a chrooted environment inside the /mnt/union filesystem. If these two scripts complete sucessfully, trip will build a binary archive with all created/modified files, the identification file and {pre,post}_{un,}install.sh scripts if available (see Binary package below).

When building a binary package from a source package, trip sets some environment variables that can be used in build.sh and install.sh :

TMP_BUILD_DIR : temporary location, mainly used to extract and compile a package. Files created under this directory won't be stored in the binary package.
SRC_DIR : points to the src subdirectory of the currently builded source package.
TRIP_* : meta-data found in the identification file.

Binary package

A binary package is a gzipped tar archive containinig the files to install, meta-data and eventually {pre,post}_{un,}install.sh scripts.

Files are stored in an uncompressed archive named "files.tar". Other files are stored in a directory named "support". Running "tar tvf lfs-base-6.2-1.noarch.tar.gz" will show :


drwxr-xr-x root/root         0 2007-07-30 15:18:24 lfs-base-6.2-1.noarch/
drwxr-xr-x root/root         0 2007-07-30 15:18:24 lfs-base-6.2-1.noarch/support/
-rw-r--r-- root/root       162 2007-07-30 15:18:24 lfs-base-6.2-1.noarch/support/identification
-rw-r--r-- root/root     51200 2007-07-30 15:18:24 lfs-base-6.2-1.noarch/files.tar

Database of installed packages

The location of this database is specified by the TRIP_DB variable in the configuration file. Its default value is /var/lib/trip.

The list of installed packages is stored in $TRIP_DB/list (one package per line), sorted by installation date ascending.

Each installed package has a subdirectory of its name in $TRIP_DB. The subdirectory contains the following files :

files : contains the list of installed files, a line per file
files-detail : contains the output of tar --list --verbose. This file may be used later to compare date/permissions of actually installed files with their expected state
identification : contains meta data about the package
{pre,post}_{un,}install.sh : eventually scripts run before/after install and uninstall.

Known limitations

As we merge two filesystem there is a problem if your linux installation is based on several filesystem (for example / and /usr), because /usr won't be visible from /mnt/union. A workaround is scheduled for trip-0.4.
trip works only on GNU/Linux, with bash. It has few prerequisites though, such as common tools like tar, coreutils, grep, etc. and unionfs.

Where to find trip and packages

There are two ways to access trip versions and packages : HTTP and Subversion. Please use HTTP, except if you plan to do regular updates.

Trip

svn "stable" : http://svn.pierrox.net/trip/versions/<version> (with <version> like "trip-0.3")
svn devel : http://svn.pierrox.net/trip/versions/devel
http : current version is trip-0.3.

Packages

svn : http://svn.pierrox.net/trip/lfs-6.2. Please use svn ls before to do a checkout or export, as there is actually more than 500 packages
http : lfs-6.2 binary packages for i686
http : lfs-6.2 source packages (linux libc-headers for i386), may not be in sync with svn
http : clfs-1.0.0 binary packages for armv4tl
http : clfs-1.0.0 source packages (linux libc-headers for arm), may not be in sync with svn

IMPORTANT : when building or installing binary packages, you should follow a specified order, as trip currently does not handle dependencies :

order for lfs-6.2 : order
order for clfs-1.0.0 : order

Conclusion

ACKNOWLEDGEMENTS:
- http://www.linuxfromscratch.org/ (and BLFS too)
- http://www.unionfs.org/
- http://www.linuxfromscratch.org/hints/
CHANGELOG:
- [2007-07-31] - Third revision, major rewrite and new trip version 0.3.
- [2006-11-19] - Second revision, corrected some minor things.
- [2006-05-13] - Initial hint.