Building U-Boot and Linux from scratch for the BeagleBone, and booting
======================================================================

* Author: Philippe Proulx [`pzhilippe.proulx@savoirzfairelinux.com` (remove `z`s)]
* Date:   Mon Jul 22 14:03:30 EDT 2013


Introduction
------------

Here's a really "simple" guide on how to build U-Boot and the Linux
kernel from scratch for the BeagleBone SoC. This is mostly a reminder
of the steps for myself and shows what I recently learned from lots of
different sources and by trial and error.

There are faster/automated ways to do all this. Buildroot has a
BeagleBone target, although as of this date, it builds Linux 3.2. Also,
the Ångström distribution, the "sort of" official BeagleBone
distribution, can be used, but it's old too. This document explains the
raw steps to build everything from scratch for didactic purposes.


Download links
--------------

As of this date, you can get the needed files from here:

* **U-Boot**: <ftp://ftp.denx.de/pub/u-boot/u-boot-latest.tar.bz2>
* **Kernel**: `git clone git://github.com/beagleboard/kernel.git`
* **Custom power firmware**: <http://arago-project.org/git/projects/?p=am33x-cm3.git\;a=blob_plain\;f=bin/am335x-pm-firmware.bin\;hb=HEAD>
* **Latest Busybox ARMv7 binary**: <http://www.busybox.net/downloads/binaries/latest/busybox-armv7l>
* **ARM cross-compiling toolchain**: there are lots of sources
for this; just make sure you get something with a Linux target,
like `arm-linux-eabi`. I used the Buildroot downloaded tools (without actually using Buildroot for the rest of the process).

We cannot use the vanilla kernel as is since TI made lots of patches
that need to be applied which are not mainlined. This is why we're
using the BeagleBoard GitHub repository (contains valid patches and
scripts for BeagleBoard and BeagleBone kernels).


Building U-Boot
---------------

The first step is to build the bootloader, U-Boot. Two important files
will come out of this process: `MLO` and `u-boot.img`. `MLO` is what
the U-Boot community calls the SPL (Secondary Program Loader) and
contains executable code for the second boot stage. Here are the stages
as a reminder (for an MMC boot):

1. the on-board ROM initializes a few cores and searches for
a file called `MLO` on the first FAT partition of the first
MMC card (the one that has a socket on the BeagleBone), then
loads it in memory and executes it
2. the SPL (`MLO`) initializes a few other things and searches
on the same partition for `u-boot.img`, which is the third
boot stage (the actual complete U-Boot program), loads it
in memory and executes it
3. U-Boot starts and after 1-2 seconds, you can get a prompt

`MLO` is compiled by U-Boot too and the vanilla U-Boot knows how to
produce this SPL (it has a board target for this).

Download U-Boot and extract it:

    $ wget ftp://ftp.denx.de/pub/u-boot/u-boot-latest.tar.bz2
	$ tar -xjvf u-boot-latest.tar.bz2

All the available boards are listed in `boards.cfg`. In there you
will find `am335x_evm`, `evm` meaning _evaluation module_.
`am335x_evm` is a common name for the SoC used by
the BeagleBone boards (exact SoC is an AM3359). Although it's not
obvious, this configuration row points to the include file
`include/configs/am335x_evm.h`, defines `SERIAL1` and also `CONS_INDEX`
to 1.

This file, `am335x_evm.h`, contains all the specific configurations
U-Boot needs in order to build `MLO` and `u-boot.img` files that the
SoC will understand. Amongst them is the `CONFIG_EXTRA_ENV_SETTINGS`
definition, which embed environment variables into the U-Boot output
image file. The less we put there, the more manual will be the boot
process, and that's what we want here since we're doing everything from
scratch to understand.

In `include/configs/am335x_evm.h`, remove the following default
environment variables (`CONFIG_EXTRA_ENV_SETTINGS`):

* all the `*args` variables (including `bootargs`)
* all the `*boot` variables
* `loadaddr`
* `fdtaddr`
* `rdaddr`
* `bootfile`
* `fdtfile`
* `loadbootenv`
* `importbootenv`
* `loadramdisk`
* `loaduimagefat`
* `loaduimage`
* `findfdt`

Also comment all the `CONFIG_BOOTCOMMAND` definition block. Save
the file.

This way, we have something minimalist and we're happy. To make sure
you're using your own version of U-Boot, you may also change the prompt
(`CONFIG_SYS_PROMPT`) for something you'll recognize.

Leave the rest as is, although there's still stuff we don't need. This
will at least provide us with a clean U-Boot environment, without
predefined stuff we don't want/understand.

We're now ready to build U-Boot's main configuration:

    $ make ARCH=arm CROSS_COMPILE=/path/to/your/toolchain/arm-something-something- am335x_evm_config

This basically creates `include/config.h` which includes relevant files
for the `am335x_evm` configuration. You will also see the additional
definitions in `boards.cfg` are added at the top of `include/config.h`:

	#define CONFIG_SERIAL1 1
	#define CONFIG_CONS_INDEX 1

We can now build U-Boot:

	$ make ARCH=arm CROSS_COMPILE=/path/to/your/toolchain/arm-something-something- -j4

Wait a few seconds and you will get `MLO` and `u-boot.img` in U-Boot's
root directory. So far so good.

In order to build the Linux kernel as an `uImage`, which is a `zImage`
with the U-Boot legacy image header, we need an installed tool called
`mkimage` which a kernel Makefile will call.

This can be built and installed now:

    $ make tools
    # install tools/mkimage /usr/local/bin
      
Just make sure you can execute it:

	$ mkimage -h

You might need to restart your shell in order to rescan the `$PATH`
binaries if it says something like `command not found`.


Building the Linux kernel
-------------------------

Let's now build Linux.

First step is to clone the aforementioned GitHub repo:

    $ git clone git://github.com/beagleboard/kernel.git

and checkout whatever version tag you want. Unfortunately, because of
the patches that need to be applied, you won't get a bleeding edge
kernel here.

For example, let us checkout the 3.8 kernel:

    $ git checkout origin/3.8 -b 3.8

Then, apply the patches (long step because it actually clones Torvald's
kernel, checksout the correct version and then applies lots of patches
to it, which are in the `patches` directory):

    $ ./patch.sh

After this, you shall see a `kernel` directory which contains the
patched Linux source. Some steps are still required before actually
building it.

    $ cd kernel

The kernel will need a precompiled power management firmware which for
some reason is maintained by the Arago Project (which belongs to TI).

Put it in the `firmware` directory:

    $ wget http://arago-project.org/git/projects/?p=am33x-cm3.git\;a=blob_plain\;f=bin/am335x-pm-firmware.bin\;hb=HEAD -O firmware/am335x-pm-firmware.bin

TI puts default `.config` files into `configs` (from the repo's root), so
use BeagleBone's default `.config` like so:

    $ cp ../configs/beaglebone arch/arm/configs/beaglebone_defconfig

and create the configuration (will create `.config`):

    $ make ARCH=arm CROSS_COMPILE=/path/to/your/toolchain/arm-something-something- beaglebone_defconfig

We may now compile the kernel:

    $ make ARCH=arm CROSS_COMPILE=/path/to/your/toolchain/arm-something-something- -j4 uImage dtbs

Let me explain what important files in `arch/arm/boot` those two targets
will create while this is compiling:

* `zImage`: this is a self-extracting compressed
	  kernel. You execute this file, and it has an decompression
	  algorithm to extract the rest of it, which is the actual
	  kernel to execute.
* `uImage`: this is `zImage` with a 64-byte
	  U-Boot header so that U-Boot knows a few things when asked to
	  boot this file (the previously installed `mkimage` tool is
	  used to create this of the `zImage`).
* `dts/am335x-bone.dtb`: the compiled device tree "blob" which
	  the kernel must have in order to initialize its drivers and
	  SoC-specific routines.

At the end of the process, you will see:

	Image arch/arm/boot/uImage is ready

and will be happy about it. `mkimage` will also report the load address
and entry point of the created image, both of which should be 0x80008000
(external DRAM starts at 0x80000000 on the AM335x SoC).

Usually, when booting (we will do this later), you need to load `uImage`
and `dts/am335x-bone.dtb` at different memory locations, then ask
U-Boot to boot by providing those two addresses. But there's another
way to give the DTB to Linux. In `.config`, you will see

	CONFIG_ARM_APPENDED_DTB=y

which tells the kernel to look at the end of its `zImage` for a valid
DTB if you don't provide it with a memory address when booting. So you
only need to concatenate `dts/am335x-bone.dtb` to `zImage` and then use
`mkimage` to create a `uImage` with appended DTB from this.

This is exactly what the `uImage-dtb.am335x-bone` target does:

    $ make ARCH=arm CROSS_COMPILE=/path/to/your/toolchain/arm-something-something- uImage-dtb.am335x-bone

Now you also have `arch/arm/boot/uImage-dtb.am335x-bone`. Its size
should be the size of `zImage` + the size of `dts/am335x-bone.dtb` + 64.


Creating a root filesystem
--------------------------

The kernel will be able to start booting with the above files, but
won't be able to finish because it will complain it cannot find any root
filesystem. We need a very basic one.

Let us create a root minimalist filesystem from scratch.

Create a new directory for the root filesystem:

	$ mkdir rootfs
	$ cd rootfs

As root, create the basic directory structure:

	# mkdir bin dev proc sys

Download the latest Busybox build for ARMv7 and make it executable:

	# wget http://www.busybox.net/downloads/binaries/latest/busybox-armv7l -O bin/busybox
	# chmod +x bin/busybox
	
Create Busybox init and sh links:

	# ln -s busybox bin/init
	# ln -s busybox bin/sh

That's really all we need for Busybox: an init and a shell (in fact,
`/bin/sh` could be the init itself). We'll execute everything else
by calling `busybox` directly once in that shell (think minimalist).

Create a few `/dev` important nodes:

	# mknod dev/console c 5 1
	# mknod dev/null c 1 3

So here we have it:

	# tree
	.
	├── bin
	│   ├── busybox
	│   ├── init -> busybox
	│   └── sh -> busybox
	├── dev
	│   ├── console
	│   └── null
	├── proc
	└── sys


Bringing it all together: creating the SD card
----------------------------------------------

Let's review what we have:

* U-Boot's `MLO`
* U-Boot's `u-boot.img`
* Linux's `uImage-dtb.am335x-bone`
* our root filesystem

We need to put all this on a properly formatted SD card. Until the end
of this document, my SD card device is `/dev/sdb`. Yours might be
different: be careful here.

Delete all existing SD card partitions. Use fdisk with the `d`
command to delete everything and write it with `w`. You might need to
unplug/replug your SD card for the changes to be seen by the kernel.

Create a FAT partition and an ext4 partition:

    # fdisk /dev/sdb <<EOF
    n
    p
    1
    
    +16M
    t
    4
    a
    n
    p
    2
    
    
    t
    2
    83
    w
    EOF

Again, unplug/replug the device.

Format both partitions:

	# mkfs.vfat -F16 -v /dev/sdb1
	# mkfs.ext4 /dev/sdb2

Mount both partitions:

	# mkdir /mnt/bbone_sd1
	# mkdir /mnt/bbone_sd2
	# mount /dev/sdb1 /mnt/bbone_sd1
	# mount /dev/sdb2 /mnt/bbone_sd2

Now, don't ask me why, but `MLO` needs to be in the very first sectors
of the partition. Copying it first ensures this. Everything else
can be copied in any order.

Copy files to the FAT partition:

	# cd /mnt/bbone_sd1
	# cp /path/to/MLO .
	# cp /path/to/u-boot.img .
	# cp /path/to/uImage-dtb.am335x-bone .
	# touch uEnv.txt

The empty file `uEnv.txt` is read by U-Boot and can be used to
dynamically set initial environment variables. We're creating an empty
one just to avoid U-Boot complaining it cannot it.

Copy the root filesystem to the ext4 partition:

	# cd /mnt/bbone_sd2
	# cp -a /path/to/rootfs/* .

Unmount both partitions:

	# cd /mnt
	# umount bbone_sd1
	# umount bbone_sd2
	
Ready to boot!


Booting
-------

Plug the USB BeagleBone cable into your host machine and plug the
external power into the BeagleBone. On your host machine, look at a
TTY named `/dev/ttyUSB0` or such. Use screen to connect to it at the
correct baud rate (always 115200 bauds):

	# screen /dev/ttyUSB0 115200,cs8

Unplug the SD card from your host machine and plug it into the
BeagleBone. Press the BeagleBone reset button near the RJ-45 socket.

After a few seconds, you should get a U-Boot prompt.

Now, we need to do 3 important things:

1. set the kernel boot parameters
2. copy the kernel image (`uImage-dtb.am335x-bone`) in memory
3. boot that memory location

This is really easy using U-Boot.

At the U-Boot prompt, type:

	# setenv bootargs "earlyprintk console=ttyO0,115200n8 root=/dev/mmcblk0p2 rootwait ro rootfstype=ext4 init=/bin/init"
	
You might find it strange that we specify `/dev/mmcblk0p2` as the root
filesystem. What is `/dev/mmcblk0p2` if there's no root filesystem
yet? It turns out this is just a convention and it's not actually
pointing to a specific file. The kernel understands a lot of device
prefixes like this (`sd`, `hd`, `fd`, etc.). `mmcblk0p2` is a synonym
for the second partition of the first MMC device (i.e. our ext4
partition). Just a note: instead of this arbitrary synonym, we could 
write the complete (still arbitrary) device number.
From [Documentation/devices.txt](https://www.kernel.org/doc/Documentation/devices.txt),
we can see that the second partition of the first MMC block device is major number
179 and minor number 2 (which means device number 0xb32), so the `root`
parameter could be `root=0xb32` as well here.

`earlyprintk` is a parameter to tell the kernel to print to the console
as soon as possible (even before the UART driver is initialized!). This
is usually possible because the SoC specific code knows how to talk to
its own UART (mostly seen on embedded devices).

`ro` mounts the root filesystem as read-only (as opposed to `rw`).

Now, the kernel image must be loaded at its load address (which is
0x80008000). However, if we copy the image contents to this memory
location, U-Boot will get an exception from the ARM CPU when trying to
boot because of unaligned memory accesses. This is because `zImage`
must be aligned on 0x80008000, not `uImage` (the encapsulated `zImage`
doesn't know about its `uImage` shell). So the `uImage` needs to be copied 64
bytes _before_ 0x80008000 (64 bytes being the U-Boot header size):
at 0x80007fc0.

Here's how to do it with U-Boot:

	# fatload mmc 0:1 0x80007fc0 uImage-dtb.am335x-bone

which means: load file `/uImage-dtb.am335x-bone` from the FAT filesystem
of the first partition of the first MMC device at memory location
0x80007fc0.

Now we can boot:

	# bootm 0x80007fc0
	
Linux should start right away. After a few seconds, the kernel will
execute its init (we specified `/bin/init`, which points to Busybox) and
Busybox's init will ask:

	Please press Enter to activate this console.
	
Do so.

You are now commanding the BeagleBone as root!

We could also have loaded `uImage` (with no appended DTB) and the DTB
at separate memory locations and still use the `bootm` command this
way:

	# fatload mmc 0:1 0x80007fc0 uImage
	# fatload mmc 0:1 0x81000000 am335x-bone.dtb
	# bootm 0x80007fc0 - 0x81000000

The `bootm` command's second argument, here `-`, is the address of an
initramfs, but we don't have any; `bootm` reads `-` as "no argument".

`bootz` can also be used to boot a `zImage` directly, although this is
not recommended because `uImage` adds some protection around it (CRC
check, etc.):

	# fatload mmc 0:1 0x80008000 zImage
	# fatload mmc 0:1 0x81000000 am335x-bone.dtb
	# bootz 0x80008000 - 0x81000000

As you already noticed, we're loading `zImage` at 0x80008000, and not
0x80007fc0, since the U-Boot header is not present in `zImage`.

Please note that U-Boot must be built with `bootz` support to use it (I
believe it's done by default since I may use it and I didn't modify the
default build features).


Using Linux
-----------

Of course this system from scratch is not really useful. For instance,
no command will be found since nothing is in `/bin` but `init` and
`sh`. If you want to execute a command, precede it with `busybox `:

	# busybox ls
	bin dev lost+found proc sys
u
Still, if you list `/proc` or `/sys`, you will find empty directories.
They are mount points, but nothing is mounted. So procfs can be mounted,
for example:

	# busybox mount -t proc proc /proc
	# busybox cat /proc/cmdline 
	earlyprintk console=ttyO0,115200n8 root=/dev/mmcblk0p2 rootwait ro rootfstype=ext4 init=/bin/init

Of course, the usual way of doing this is to use `mount -a` which reads
`/etc/fstab` and mounts everything at once, but we don't have such a
file.


Conclusion
----------

So this is the very basis of an embedded Linux system creation. From
there, you have something working and can experiment with more useful
configurations:

* automating the booting process (this can be done using
  `uEnv.txt`)
* compiling kernel modules and installing them in the root
  filesystem (kernel targets `modules` and `install_modules`)
* getting the kernel image using TFTP in U-Boot (U-Boot knows
  TFTP and even DHCP)
* getting the root filesystem using NFS (Linux must be
  configured accordingly to understand NFS and be able to boot
  a remote filesystem)
* installing a complete official distribution, like
  Arch Linux ARM