Introduction

This page gives hints on how to build a Cross Linux From Scratch GNU/Linux operating system, for an ARM machine, using trip as package manager. The aim is not to produce a ready to use system, but to build a complete, self-contained and packaged development environment.

The key point of this guide is to bootstrap an ARM machine with a minimal build system, and then to continue building higher level softwares from the ARM machine, with a native compiler. Cross compilation is only used at the initial stage in order to produce a minimal running system. This makes thing considerably more powerfull and safer than a complete cross compiled root filesystem.

We are using QEMU as an ARM emulator. It provides a complete system emulation, with hard disk, ethernet, video, keyboard and mouse.

As a package manager we use Trip. It is a lightweight tool, originally targeted at LFS but usable on most of Linux distros. Its aim is to install packages in a 100% safe manner, without being a pain when building and installing a lot of packages. Trip needs unionfs and some basic tools like bash, tar, find, grep, gzip.

We suppose we are using a x86 computer to produce a first ARM self contained system, but it could be done from any other architecture, provided that it runs a Linux 2.6 kernel. It could be done from a previous ARM system, in which case it would not even be a Cross Linux From Scratch but only a Linux From Scratch.

The process of building a complete operating system involves many components. Many things can be done differently, you are free (and encouraged) to try alternate ways. QEMU is used in our case to simulate an ARM machine, but you can use a real hardware if you have one faster. QEMU make things easier however, since everything can be embedded on a single computer, and since changes/backup of virtual hard disks and RAM upgrade can be done quickly.

This guide has been written with the Eten G500 in sight. It contains specific steps for this machine, especially patching QEMU and the Linux kernel in order to produce a versatile ARM system based on a ARM920t processor, instead of a 926. The 920t patch should be considered as a hack. It could be done for other processor too.

Suggestions and feedback are welcome.

Audience

You are supposed to have background knowledge on the following topics : Linux, system softwares, compilation, cross compilation, possibly ARM architecture.

Overview

We use QEMU as a machine emulator. The virtual machine is started using a NFS root and a minimal environment (a shell). The QEMU virtual machine communicates with the host using TCP/IP. Once the virtual machine is started, we begin to build and install some packages needed to build final softwares, then we build all others packages using Trip. The different steps are :

  • on the host computer build a minimal cross-compiled system
  • on the host build the tools needed to run the ARM virtual machine
  • start the virtual machine and from the minimal system, start building packages
  • finally configure the newly built system

Prerequisites

You must have read the CLFS (or LFS) book at least once before to continue.

Hardware

You will need 3 or 4Gb free on your hard drive. A fast CPU is recommended, especially if you run QEMU. To have an idea of emulation speed, lets consider the approximate execution time of the command dd if=/dev/zero bs=1M count=10 | gzip > /dev/null :

  • native ARM920t@400MHz (Eten G500 based on s3c2440) : 4.8 seconds
  • QEMU on a 1.1GHz Pentium-M : 7.8 seconds
  • QEMU on a Intel Core Duo 6400 : 2.8 seconds (faster than native processor, looks interesting)
  • x86 native code on a 1.1GHz Pentium-M : 0.4 second

64Mb of RAM is not enough, use 128Mb or add some swap space.

Software

The following list of required softwares is not exhaustive. It shows the major components involved in the build process.

Of course you can deviate from the versions used here. In this case I cannot certify it will work of course. I cannot even certify that using the very same versions will lead to the same result, as many components take part in the process of building softwares.

Other

Internet, brain, books, time, possibly coffee.

Preparing the build environment

Preparations

We choose /home/clfs-trip as root for our work. Create some directories :

export CLFS_TRIP=/home/clfs-trip mkdir -p $CLFS_TRIP/{src,build,dist,root,pkg,vm}

Where the purpose of these directories are :

  • src : store the source packages and patches
  • build : temporary build directory
  • dist : locally installed softwares
  • root : location of CLFS files, the final root filesystem, the NFS root of the virtual machine
  • pkg : temporary location used to "fake install" packages (see trip)
  • vm : files used by the ARM virtual machine

Gather the required source files :

cd $CLFS_TRIP/src wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.3.tar.bz2 wget ftp://ftp.fsl.cs.sunysb.edu/pub/unionfs/unionfs-1.4.tar.gz wget http://belnet.dl.sourceforge.net/sourceforge/e2fsprogs/e2fsprogs-1.39.tar.gz wget http://fabrice.bellard.free.fr/qemu/qemu-0.8.2.tar.gz wget http://www.pierrox.net/G500/20070109/config-2.6.18.3-versatile_pb-920t wget http://www.pierrox.net/G500/20070109/linux-2.6.18.3-versatile_pb-920t.patch wget http://www.pierrox.net/G500/20070109/qemu-0.8.2-versatile_pb-920t.patch wget http://www.pierrox.net/G500/20070109/trip-0.2-clfs-1.0.0.tar.gz

Furthermore download the CLFS book and needed packages and patches as described in CLFS chapter 3.2 and 3.4. In case some external dependencies are unreachable, they are mirrored at http://www.pierrox.net/G500/20070109/.

Build CLFS /cross-tools and /tools

Follow the CLFS-1.0.0 book from chapter 1 to end of 6. There is no ARM specific guide but following MIPS (and replacing mips with arm) is ok. Take care when following the book not to include MIPS specific parts (patches and asm headers for example).
At the end of chapter 6, /tools and /cross-tools are ready to build the basic system software. I choosed as CLFS_TARGET arm-9tdmi-linux-gnu.

Important : make sure these variables are set :

CLFS_TRIP=/home/clfs-trip CLFS=$CLFS_TRIP/root CLFS_HOST="$(echo $MACHTYPE | sed "s/$(echo $MACHTYPE | cut -d- -f2)/cross/")" CLFS_TARGET=arm-9tdmi-linux-gnu PATH=/cross-tools/bin:$PATH export CLFS_TRIP CLFS CLFS_HOST CLFS_TARGET PATH

Create some essentiel directories, device nodes and symlinks

mkdir -p $CLFS/{dev,proc,sys,bin,tmp} mkdir -p $CLFS/usr/lib mkdir $CLFS/dev/{pts,shm} mknod -m 600 $CLFS/dev/console c 5 1 mknod -m 666 $CLFS/dev/null c 1 3 ln -sv /tools/bin/{bash,cat,grep,pwd,stty} $CLFS/bin ln -sv /tools/lib/libgcc_s.so{,.1} $CLFS/usr/lib ln -sv /tools/lib/libstd* $CLFS/usr/lib ln -sv bash $CLFS/bin/sh mkdir -p $CLFS/mnt/{pkg,union}

Install Trip

Untar the trip-0.2-clfs-1.0.0.tar.gz, it contains both trip-0.2 and the clfs-1.0.0 associated trip packages.

cd $CLFS tar xf $CLFS_TRIP/src/trip-0.2-clfs-1.0.0.tar.gz mv trip-0.2-clfs-1.0.0 trip

Create a temporary fstab for trip pkg and unionfs filesystems

mkdir $CLFS/etc echo << EOF > $CLFS/etc/fstab 192.168.101.2:/home/clfs-trip/pkg /mnt/pkg nfs defaults,nolock 0 0 unionfs /mnt/union unionfs dirs=/mnt/pkg:/=ro 0 0 EOF

Build an ARM Linux kernel

This kernel will be used for the QEMU virtual machine. The provided linux config file mainly configures the kernel with ARM versatile PB machine type and support for NFS root.

cd $CLFS_TRIP/build tar xjf $CLFS_TRIP/src/linux-2.6.18.3.tar.bz2 cd linux-2.6.18.3 cp $CLFS_TRIP/src/config-2.6.18.3-versatile_pb-920t .config patch -p1 < $CLFS_TRIP/src/linux-2.6.18.3-versatile_pb-920t.patch make ARCH=arm CROSS_COMPILE=arm-9tdmi-linux-gnu- make ARCH=arm CROSS_COMPILE=arm-9tdmi-linux-gnu- INSTALL_MOD_PATH=/tools modules_install cp arch/arm/boot/zImage $CLFS_TRIP/vm/kernel-2.6.18.3

The patch applied on the kernel tree allows to choose a 920t processor for the versatile pb machine.

Build the unionfs module

Unionfs is required for Trip.

cd $TRIP_CLFS/build tar xzf $CLFS_TRIP/src/unionfs-1.4.tar.gz cd unionfs-1.4 make LINUXSRC=$CLFS_TRIP/build/linux-2.6.18.3 \ CC=arm-9tdmi-linux-gnu-gcc \ ARCH=arm \ CROSS_COMPILE=arm-9tdmi-linux-gnu- \ unionfs.ko make LINUXSRC=$CLFS_TRIP/build/linux-2.6.18.3 \ CC=arm-9tdmi-linux-gnu-gcc \ ARCH=arm \ CROSS_COMPILE=arm-9tdmi-linux-gnu- \ MODPREFIX=$CLFS/tools \ MODDIR=lib/modules/2.6.18.3 \ install-mod

Build QEMU

cd $CLFS_TRIP/build tar xf $CLFS_TRIP/src/qemu-0.8.2.tar.gz cd qemu-0.8.2 patch -p1 < $TRIP_CLFS/src/qemu-0.8.2-versatile_pb-920t.patch ./configure --prefix=/usr/local/qemu \ --cc=/usr/local/gcc-3.4.6/bin/gcc \ --host-cc=/usr/local/gcc-3.4.6/bin/gcc \ --target-list=arm-user,arm-softmmu make install

Some explanations :

  • --prefix=/usr/local/qemu : choose your prefix as you like, but since it is a patched version, avoid /usr
  • --cc=/usr/local/gcc-3.4.6/bin/gcc : qemu has problems with gcc4, you may have to install an older gcc version, use this option according to your own build environment
  • --target-list=arm-user,arm-softmmu : this selects only the ARM emulation

Running the virtual machine

The virtual machine is ready, it is now time to start it. Once it has been started, some more packages will be built in /tools, then final packages will be built using trip.

Configuring and starting QEMU

QEMU has a lot of options, so putting the command line in a script is probably a good idea.

cd $CLFS_TRIP/vm cat << EOF > qemu.sh /usr/local/qemu/bin/qemu-system-arm \ -M versatilepb_920 \ -kernel kernel-2.6.18.3 \ -nographic \ -append "root=/dev/nfs rw nfsroot=192.168.101.2:/home/clfs-trip/root \ ip=192.168.101.3::192.168.101.2:255.255.255.0:bipbip:eth0:off \ console=ttyAMA0 \ init=/tools/bin/bash" \ -net nic -net tap,script=./qemu-ifup.sh EOF chmod +x qemu.sh

Some explanations :

  • -M versatilepb_920 : select the type of emulated arm system, in our case it is our special version of versatile pb based on 920t
  • -kernel kernel-2.6.18.3 : being able to boot directly the kernel avoid us to install a boot loader and a more complex system
  • -nographic : disables the video output. For the moment it will not be used but will be reactivated later
  • -append "..." : this selects the command line passed to the kernel (see bellow)
  • -net nic -net tap,script=./qemu-ifup.sh : this select the virtual network interface mode

The kernel command line is composed of these parameters :

  • root=/dev/nfs rw nfsroot=192.168.101.2:/home/clfs-trip/root : tells the kernel to use the prepared root filesystem as its root
  • ip=192.168.101.3::192.168.101.2:255.255.255.0:bipbip:eth0:off : configure the ethernet interface with a fixed ipv4 adress
  • console=ttyAMA0 : the first UART is used as the QEMU console, so that typing in the terminal will send characters to the VM console

The qemu-ifup.sh script configures the tap interface :

cat << EOF > qemu-ifup.sh #!/bin/sh ifconfig tap0 192.168.101.2 EOF chmod +x qemu-ifup.sh

So we use 192.168.101.2 for the host and 192.168.101.3 for the guest. Instead of a tap interface you may prefer to use bridging.

Booting this way drops the user in a really restricted environment. However this is enough for what we have to do and it avoids to install more packages.

./qemu.sh

From that point all steps take place in the virtual machine console

Build some additionnal packages

As our "init" is just a bash, we need to dome some work manually when the virtual machine is started.

export PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin:/trip/trip/bin

Set the date manually because this virtual machine has no RTC clock.

date -s "<enter the current date !>"

Build additionnal tools that will be used later :

util-linux-2.12r

{literal} cd /trip/tmp tar xf /trip/clfs-1.0.0/src/util-linux-2.12r/src/util-linux-2.12r.tar.bz2 cd util-linux-2.12r patch -p1 -i ../util-linux-2.12r-gcc4_fixes-1.patch cp -v configure{,.orig} sed -e 's@/usr/include@/tools/include@g' configure.orig > configure ./configure make ARCH="" CPU="" -C lib make ARCH="" CPU="" -C mount mount umount make ARCH="" CPU="" -C text-utils more cp -v mount/{,u}mount text-utils/more /tools/bin

module-init_tools-3.2.2

insmod is needed to load the unionfs module

cd /trip/tmp tar xf /trip/clfs-1.0.0/src/module-init-tools-3.2.2/src/module-init-tools-3.2.2.tar.bz2 cd module-init-tools-3.2.2 ./configure --prefix=/tools make make install

perl-5.8.8

Perl is needed later, during gcc build for example

cd /trip/tmp tar xf /trip/clfs-1.0.0/src/perl-5.8.8/src/perl-5.8.8.tar.bz2 cd perl-5.8.8/ ./configure.gnu --prefix=/tools -Dstatic_ext='Data/Dumper IO Fcntl POSIX' -Dcc="gcc" make perl utilities cp -v perl pod/pod2man /tools/bin install -dv /tools/lib/perl5/5.8.8 cp -Rv lib/* /tools/lib/perl5/5.8.8 ln -sv /tools/bin/perl /usr/bin

Build packages using trip

Now everything is setup to build final packages. The following command will load the unionfs module and mount the pkg NFS share, as required for trip :

insmod /tools/lib/modules/2.6.18.3/kernel/fs/unionfs/unionfs.ko mount /mnt/pkg

Each package should be built using : trip -b /trip/clfs-1.0.0/src/<package>. Once the package has been built a binary package is created and can be installed with trip -i <binary package>. For example : trip -b /trip/clfs-1.0.0/glibc-2.4 && trip -i glibc-2.4-1.armv4t.tar.gz.

Build the packages in this order :

  • clfs-base-1.0.0
  • linux-headers-2.6.17.13-09092006
  • man-pages-2.34
  • glibc-2.4

Adjusting the toolchain : this operation does not fit in any package, so do it once. In case you would like to come back to the previous step, in order to re-use the /tools directory for another build, just remove the file $(dirname $(gcc --print-libgcc-file-name))/specs.

gcc -dumpspecs | \ perl -p -e 's@/tools/lib/ld@/lib/ld@g;' \ -e 's@\*startfile_prefix_spec:\n@$_/usr/lib/ @g;' > \ $(dirname $(gcc --print-libgcc-file-name))/specs

Continue with these packages :

  • binutils-2.17
  • gcc-4.1.1 : you will get normal conflicts on previously created symlinks, install with trip --no-conflict -i gcc-4.1.1-1.armv4tl.tar.gz
  • coreutils-5.96 (conflicts with previously created symlinks, use --no-conflict)
  • iana-etc-2.10
  • m4-1.4.4
  • bison-2.3
  • ncurses-5.5
  • procps-3.2.6
  • sed-4.1.5
  • libtool-1.5.22
  • flex-2.5.33
  • iproute2-2.6.16-060323
  • perl-5.8.8 (conflicts with previously created symlinks, use --no-conflict)
  • readline-5.1
  • zlib-1.2.3
  • autoconf-2.59
  • automake-1.9.6
  • bash-3.1 (conflicts with previously created symlinks, use --no-conflict)
  • bzip2-1.0.3
  • diffutils-2.8.7
  • e2fsprogs-1.39
  • file-4.17
  • findutils-4.2.27
  • gawk-3.1.5
  • gettext-0.14.5
  • grep-2.5.1a (conflicts with previously created symlinks, use --no-conflict)
  • groff-1.19.2
  • inetutils-1.4.2
  • gzip-1.3.5
  • kbd-1.12
  • less-394
  • make-3.81
  • man-1.6d
  • mktemp-1.5
  • module-init-tools-3.2.2
  • patch-2.5.9
  • psmisc-22.2
  • shadow-4.0.16
  • sysklogd-1.4.1
  • sysvinit-2.86
  • tar-1.15.1
  • texinfo-4.8
  • udev-096
  • util-linux-2.12r
  • vim-7.0
  • clfs-bootscripts-1.0.0
  • udev-rules-1.0.3
  • clfs-configuration-1.0.0

Post build configuration

Update the root password since it has not yet a correct value

passwd root

Note : at this stage it seems the qemu keyboard handling makes passwd fail. If you encounter the same problem, use passwd -d root and choose another password later.

Create the final fstab :

echo << EOF > $CLFS/etc/fstab 192.168.101.2:/home/clfs-trip/root / nfs defaults 0 0 proc /proc proc defaults 0 0 sysfs /sys sysfs defaults 0 0 devpts /dev/pts devpts gid=4,mode=620 0 0 shm /dev/shm tmpfs defaults 0 0 192.168.101.2:/home/clfs-trip/pkg /mnt/pkg nfs defaults,nolock 0 0 unionfs /mnt/union unionfs dirs=/mnt/pkg:/=ro 0 0 EOF

Create a .bashrc and .bash_profile for the root user (examples) :

cat << EOF > /root/.bashrc #!/bin/bash alias cp='cp -i' alias rm='rm -i' alias mv='mv -i' alias ll='ls -l --color' [ "\s-\v\$ " = "\s-\v\$ " ] && PS1="[\u@\h \W]\$ " EOF cat << EOF > /root/.bash_profile #!/bin/bash LC_ALL=POSIX PKG_CONFIG_PATH=/usr/lib/pkgconfig PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/trip/trip/bin export LC_ALL PKG_CONFIG_PATH PATH if [ -f ~/.bashrc ] then source ~/.bashrc fi EOF

Indicates the known shells (will be used later by dropbear for example) :

cat << EOF > /etc/shells /bin/bash EOF

At last but not least, modify inittab so that a getty is started on our console (ttyAMA0, first virtual UART), so that we can login after the next reboot.

sed 's/tty1 9600/ttyAMA0 115200/' -i /etc/inittab

Final steps

Now stop the virtual machine

sync exit

I don't know the best way to unmount /, sync may not be enough. Exiting the shell will produce a kernel panic, since there is no more running processes. You will then need to kill qemu.

Return to the host build environment, we will install kernel modules into their proper location. Note that neither modules nor kernel are packaged. Also doing this way will create wrong build and src symlinks in $CLFS/lib/modules/2.6.18.3/ since the Linux kernel tree is not located in $CLFS, it is not a problem anyway.

cd $CLFS_TRIP/build/linux-2.6.18.3 make ARCH=arm CROSS_COMPILE=arm-9tdmi-linux-gnu- INSTALL_MOD_PATH=$CLFS modules_install

Install the unionfs module :

cd $CLFS_TRIP/build/unionfs-1.4 make LINUXSRC=$CLFS_TRIP/build/linux-2.6.18.3 \ CC=arm-9tdmi-linux-gnu-gcc \ ARCH=arm \ CROSS_COMPILE=arm-9tdmi-linux-gnu- \ MODPREFIX=$CLFS \ MODDIR=lib/modules/2.6.18.3 \ install-mod

Update the qemu script : the kernel command line don't need anymore an init parameter. Instead of a bash, this is init from sysvinit-2.86 that we want to launch :

cd $CLFS_TRIP/vm cat << EOF > qemu.sh /usr/local/qemu/bin/qemu-system-arm \ -M versatilepb_920 \ -kernel kernel-2.6.18.3 \ -nographic \ -append "root=/dev/nfs rw nfsroot=192.168.101.2:/home/clfs-trip/root \ ip=192.168.101.3::192.168.101.2:255.255.255.0:bipbip:eth0:off \ console=ttyAMA0" \ -net nic -net tap,script=./qemu-ifup.sh EOF chmod +x qemu.sh

The build directories /tools and /cross-tools are not needed anymore. You can remove them, or keep /tools for another build. In this last case remember to delete the spec file created in order to adjust the toolchain. The /cross-tools directory contains the cross toolchain so it is probably a good idea to keep it somewhere, except if you plan to use another toolchain.

Finally restart QEMU one more time

./qemu.sh

You should obtain a first error on Mounting remaining file systems... (harmless) and a second on Setting system clock.... This is a normal error since the virtual machine has no RTC. The last lines should show something like that :

VFS: Mounted root (nfs filesystem). Freeing init memory: 104K INIT: version 2.86 booting Mounting kernel-based file systems: /proc /sys [ OK ] Creating /dev in tmpfs... [ OK ] Copying static entries... [ OK ] Setting Permissons on /dev/shm... [ OK ] Starting udevd... [ OK ] Performing Coldplugging... [ OK ] Mounting root file system in read-only mode... [ OK ] Checking file systems... [ OK ] Remounting root file system in read-write mode... [ OK ] Recording existing mounts in /etc/mtab... [ OK ] Mounting remaining file systems... [ FAIL ] Activating all swap files/partitions... [ OK ] Cleaning file systems: /tmp /var/lock /var/run [ OK ] Setting system clock... [ FAIL ] Bringing up the loopback interface... [ OK ] Setting hostname to clfs-1.0.0... [ OK ] INIT: Entering runlevel: 3 Starting system log daemon... [ OK ] Starting kernel log daemon... [ OK ] clfs-1.0.0 login:

Login as root, and set the date :

date -s "<enter the current date !>"

Build modules dependencies :

depmod -a

Update the trip configuration file. Packages will now be built using the real environment instead of temporary tools located in /tools :

sed 's/TRIP_MODE=hosted/TRIP_MODE=normal/' -i /trip/trip/conf/conf

Going further with Trip

The trip clfs-1.0.0 contains other packages than those needed to build the basic system. At the time of writting it contains required packages to run Xorg, matchbox and GPE. To get a copy of the current repository use SVN :

svn co http://svn.pierrox.net/trip/clfs-1.0.0

For convenience I suggest to build dropbear in order to get a remote access to the virtual machine :

trip -b /trip/clfs-1.0.0/src/dropbear-0.48.1 trip -i dropbear-0.48.1-1.armv4tl.tar.gz

The rest is up to you, « your distro, your rules ». Good luck.

Related links