T2-project

Page updated December 5, 2009

Introduction

The T2 project is a set of Bash scripts to compile a complete Linux distro from source packages. T2 was used to compile the base binary packages for Puppy 2.10, 4.0, and more recently, Quirky (December 2009).

The T2 build system actually can build its own T2 Linux distro, however I post-process the raw binary packages compiled by T2 and use them to build a Puppy Linux or Quirky or Puppy-derivative distro. The tool for doing this post-processing is Woof (see end of page).

In November 2009 I did a T2 compile, based roughly on the T2 "8.0rc" release. This web page introduces T2, shows how to use T2, and uses my recent build as the example. If you wanted to, you could then (after compiling all the packages in T2) use Woof to build a Puppy-based distro.

I must emphasise though, T2 is only of interest to a very few developers. As I have already run T2, I have uploaded all the binary packages (509 of them!) and Woof can use those without having to go through compiling packages from source in T2.

Note, I am using my latest compile with T2 as the basis of Quirky, my new Linux distro.

Getting started with T2

Recompiling everything from source is not easy. Considerable Linux experience is required. A special build environment is required, especially if we want to upgrade the gcc compiler and glibc C library, or to compile for another CPU. Also, source packages may need to have patches applied, or have some kind of pre- or post- script executed. It's incredibly complicated, and for this reason we have teamed up with the T2 project, lead by Rene Rebe. T2 was a fork from ROCK Linux, but since then has developed its own special flavour and is very actively developed and maintained with recent packages.

Learn more about T2 here: http://t2-project.org

When I first started to learn how to use T2, I found the online manual to be incomplete in places, but I posted questions to the mail-list and Rene is extremely helpful. For those who come after me it will be easier, as I have a preconfigured T2-build-package, ready to go to compile packages for Puppy.

Step 1: Getting everything ready

Puppy/Quirky has a devx_xxx.sfs file, that turns Puppy into a compiler environment, where 'xxx' is the version number for example '431' for Puppy version 4.3.1. You must be using Puppy v4.00 or later.

You download the 'devx' file to /mnt/home (the same place as the 'pupsave' personal storage file, select it in the BootManager, then reboot. It's as simple as that. If you need further explanation, see the C/C++/Vala/Genie Compiling page.

If you have a full hard drive installation, there are more steps to install the 'devx' file, see the Hard Puppy web page.

Step 2: Install T2 core-build-package

The T2 build system can be downloaded from the T2 site by SVN checkout, however I strongly recommend that you use the one that I used, as it is heavily modified, especially with a view to creating packages that are as small as possible, in keeping with the Puppy philosophy. You can get it from here:

http://distro.ibiblio.org/pub/linux/distributions/quirky/t2/

The tarball is 120MB. The reason it is so big is I have included several source packages, some patched, some that might be hard to locate. Most of the rest should be on the T2 repositories and will be automatically downloaded when you run the appropriate T2 script.

You need to download it to a Linux partition (ext2, ext3 or reiserfs) with at least 10GB free space, preferably 15-20GB if you want to compile more than the basic packages needed for Puppy/Quirky.

When the T2-pkg is expanded, you will see these directories:

t2-8.0rc-tpup
architecture	build	config	download	misc	package	scripts	source	target
		puppy5b						puppy5

The 'target' directory is where you will find the package-selection and special configurations for Puppy. I have created a directory named 'puppy5' which is used for building the packages suitable for Puppy. If you poke around in here, the main file of interest is 'pkgsel' which is a list of all the packages that will be compiled.

If we have specified that we want 'abiword' in Puppy, what version, where do we download it from, and so on? Look in the 'package' directory and you will find detailed specs for each package. Basically, you will find files with extensions .desc, .conf and .cache. You should get the basic idea of what these files do by viewing them in a text editor. For example, if I look at package/gnome2/abiword/abiword.desc, I see that the version is 2.8.1, and the original download URL is also given. However, T2 will first look at its own online repositories before downloading from the original URL.

The 'download' directory is where source packages get downloaded to. If you are online, T2 will do this automatically, but beware, even if you only download the selection of packages required by Puppy, it is going to be about 1.5GB. You can also download them manually if you want, but T2 does a checksum so is the preferred way of downloading them.

The 'config' directory is where you can create one or more build "profiles". I have already created one called 'puppy5b', and you will see a directory of that name. I created this profile by running this script:
# ./scripts/Config -cfg puppy5b
When I ran the Config script, I chose the 'puppy5' target, plus various other customisations.

If you want to make any changes to the puppy5b profile, you can run this yourself:

# ./scripts/Config -cfg puppy5b

You get a text-mode "kconfig" menu system that offers all kinds of choices, like what CPU to build for and what packages to compile with dietlibc. You can run the above just to see what the choices are, and exit without making any changes.

If you want to study what changes I made to the T2 build system, perhaps with a view to applying the same to a later checkout from SVN, read these notes:

http://distro.ibiblio.org/pub/linux/distributions/quirky/t2/t2-8.0rc-tpup-readme.txt

Step 3: Download source packages

To get going and compile the packages for Puppy, it is very simple. You have to be online, so that T2 can download the required packages. Type this:

# ./scripts/Download -cfg puppy5b -required

Then wait while about 1.2GB of packages will download. Note, '-required' means that only the packages required for the 'puppy5b' profile will be downloaded (not the whole lot, which is over 2400 packages!).

A warning: I changed some of the .desc files inside the 'package' directory. For example, I changed package/www/seamonkey/seamonkey.desc to a later version. These are the changed lines:

[V] 2.0
[D] 1889686604 seamonkey-2.0.source.tar.bz2 
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/releases/2.0/

I then downloaded seamonkey-2.0.source.tar.bz2 directly (manually) into download/mirror/s/. Notice that number '1889686604' -- T2 uses the 'cksum' program for doing checksums, but it is done on the uncompressed file. So, to do it yourself: '# bunzip2 seamonkey-2.0.source.tar.bz2' then '# cksum seamonkey-2.0.source.tar' and use the first number.
Notice the URL -- this is the author's fallback download site, if the package is not available on the T2 repositories.

However, correcting the cksum is optional. If you manually download a later package directly into the 'download' folder, T2 will not check its cksum. All you really need to update in the '.desc' file is the '[V]' entry.

If T2 is not able to download the package, or you didn't do it manually, the Download script will hiccup at that point and output some bold red text. You might like to run this a second time just to get a list of all the failures:

# ./scripts/Download -cfg puppy5b -required

T2 does fall back to trying to obtain the packages from the original author's download sites, but they can change, meaning that you should look in the .desc files and correct any wrong URLs.

Note that I also attempt to host source packages at my own site. So, if the T2 Download script can't find it, look here and grab it manually:

http://bkhome.org/sources/alphabetical/

Step 4: Start compiling!

The 'build' directory is where all output is created. A complete root filesystem is built in here, an emulated environment, complete with its own kernel and CPU. The binary packages will also be created in here as .tar.bz2 files, and these are what we want for our Puppy.

After downloading all required source packages, the entire process of building the binary (compiled) packages is done with just this one line:

# ./scripts/Build-Target -cfg puppy5b

After about one day it should be done!

I'm not kidding, it takes ages, and needs heaps of space. A 2GHz CPU and at least 512MB of RAM is required if you want it to complete in "only" 1 day.
You can still simultaneously use the PC for Internet access, but response is going to be sluggish, and don't expect to do anything else heavy-duty.

Step 5: Compile failures

When T2 compiles a package, the source tarball is expanded and T2 creates a symlink in the 't2-8.0rc-tpup' directory to it. Thus, click on the link and you will be taken into the build environment for the package. Normally T2 will remove the package build environment, but if the build fails it is left there. So, later on you can go in and try to find out what went wrong.

Also, T2 maintains a build log in build/puppy5-*-x86-i486/var/adm/logs/. For example, if Seamonkey has failed to compile, there is this file: build/puppy5-*-x86-i486/var/adm/logs/5-seamonkey.err.
This file is the most helpful for finding the reason for failure.

Lots of failures result in very large package build-environments being left behind, quickly reducing free space in the partition. Plus there are all sorts of other temporary files. You can clean everything up (which does not delete the successfully compiled binaries) by doing this:

# ./scripts/Cleanup

If you do want to delete all the compiled packages and start from scratch, do this:

# ./scripts/Cleanup -build -cache

Step 6: Fixing failures

A build failure may be due to a missing prerequisite, or even a prerequisite compiling after the failed package. Or, configure options may be wrong.

Each package has its own .conf file, where you can control the build process. For example, Seamonkey has package/www/seamonkey/seamonkey.conf. This can be edited, though Rene mentioned that this is not desirable as it affects upgradability of the T2-build-pkg -- by customising these files we are making it difficult to sync with the main T2 build package when they have new releases. To get around this problem, an "over-arch" .conf file can be created in the target/puppy5 directory -- see the online T2 Handbook for details on how to do this.

When you think that you have fixed the failure, maybe cleanup then run the Build-Target script again:

# ./scripts/Cleanup
# ./scripts/Build-Target -cfg puppy5b

Step 6B: Fixing 8.0rc-tpup

There are some hacks that you may have to do manually to get the '8.0rc-tpup' packages to compile. Well, at time of writing I have got it down to just one:

librsvg installation, stage 5, fails. To fix this, open the file build/..../var/adm/flists/gtk+ in a text editor and rename the text string "gdk-pixbuf.loaders" to anything else, for example "gdk-pixbuf.loaders-HIDE". Then do this:
```
# ./scripts/Cleanup -cache
# ./scripts/Build-Target -cfg piglet001
```
After librsvg has succesfully installed, edit the gtk+ file and change that text string back to what it was.
Currently not compiling Mplayer, so don't know if still need this:
mplayer compile, stage 5, fails. To fix this, copy the file libavutil/intreadwrite.h out of the mplayer source package into build/..../usr/include/ffmpeg/
# ./scripts/Cleanup -cache
# ./scripts/Build-Target -cfg piglet001
This is a problem with Mplayer using the external ffmpeg package.

Step 7: Adding more packages

Open the file target/puppy/pkgsel and you will see some packages commented out. You can uncomment these or add more, then run the Built-Target script again. You can also add any more packages that you want. That is, do this:

# ./scripts/Build-Target -cfg puppy5b

Puppy will just continue on compiling and adding these new packages to the build system.

Woof: the "Puppy builder"

If you go through the above steps, you will have 500+ binary (.tar.bz2) packages. The question now is, how to turn those into a "Puppy-like" distro? That is, a nice sub-100MB live-CD with all the features and speed that Puppy is famous for?

This is where Woof comes in:

Introduction to Woof

In a nutshell, what you can do is download Woof and run the special '0pre' script. This script will import the binary packages from T2, and you can then use them to build a Puppy-like live-CD.

Alternatively, and this will interest most people I think, I have uploaded all of the the T2 binary packages that I compiled, to my web site. I have pre-configured Woof to know where these are, and Woof will download and use them.

When running the 'woof_gui' script in Woof, which is a GUI interface, in the first tab you can select 'tpup' or 'quirky' profiles, which are preconfigured to know where I have the t2 binary packages online.

...from then on, it is pretty straightforward to build your own custom Puppy-like distro.

Further notes

There is a little hicupp with T2, the gcc compiler is slightly broken. After you have used Woof to build your distro, boot it and with the 'devx' SFS loaded you have to recompile the gcc package, as documented in my blog:

http://bkhome.org/blog/?viewDetailed=01236

However, in the case of the T2 build that I have recently done, I have already done this and replaced the T2 'gcc-4.3.4.tar.bz2' package with my own. This is where I have them (and Woof knows about this location):

http://bkhome.org/binaries/t2/8.0rc/