Cross-compiling for PS3 Linux

Now that the PS3 is out and multiple Linux-based distributions are available which can be installed using Open Platform [playstation.com] it's time to start developing on some publically available hardware!

Although the PPU and SPU compilers can be installed and used on the PS3 directly, I find it much more familiar and convinient to cross-compile from my desktop and just ship the resulting executables over to the target (PS3).

In this article, I will detail the basic steps I used to get started building on a host PC and running on the PS3.
Install Linux
I have sucessfully compiled and run using both Yellow Dog Linux [terrasoftsolutions.com] and Fedora Core [redhat.com].

This article assumes that Linux is already installed on the PS3. It's very easy to install and the process is already documented quite well.

Carl Bender over at PS3PC.net has written a very good guide on Installing Fedora 5 Linux on Your PS3 [linuxps3.net]

See also: Installation Guide for Yellow Dog Linux [terrasoftsolutions.com]
See also: Installation Guide for Fedora Core 5
See also: Linux on the Playstation 3 Wiki [pslinux.org]
See also: Installing Gentoo on the PS3 [daniel.jp]

NOTE: For the sake of this article, Yellow Dog Linux 5 (32 bit version for PS3) will be assumed. A 32 bit host PowerPC Fedora Core 5 installation will also be assumed (Although 64 bit and x64 versions of the libraries are available for other types of hosts.)
cat /proc/cpuinfo (For the Target PS3)
processor : 0
cpu : Cell Broadband Engine, altivec supported
clock : 3192.000000MHz
revision : 5.1 (pvr 0070 0501)

processor : 1
cpu : Cell Broadband Engine, altivec supported
clock : 3192.000000MHz
revision : 5.1 (pvr 0070 0501)

timebase : 79800000
machine : PS3PF

cat /proc/interrupts (For the Target PS3)
 CPU0 CPU1
10: 19437 0 PS3PF irq controller Edge ehci_hcd:usb1
11: 20767742 0 PS3PF irq controller Edge ehci_hcd:usb2
16: 0 0 PS3PF irq controller Edge ohci_hcd:usb3
17: 0 0 PS3PF irq controller Edge ohci_hcd:usb4
128: 0 574866 PS3PF irq controller Edge IPI0 (call function)
129: 0 3024105 PS3PF irq controller Edge IPI1 (reschedule)
130: 0 0 PS3PF irq controller Edge IPI2 (unused)
131: 0 0 PS3PF irq controller Edge IPI3 (debugger break)
132: 555759 0 PS3PF irq controller Edge IPI0 (call function)
133: 2998857 0 PS3PF irq controller Edge IPI1 (reschedule)
134: 0 0 PS3PF irq controller Edge IPI2 (unused)
135: 0 0 PS3PF irq controller Edge IPI3 (debugger break)
136: 0 0 PS3PF irq controller Edge Virtual UART
137: 0 0 PS3PF irq controller Edge spe00.0
138: 1 0 PS3PF irq controller Edge spe00.1
139: 7 0 PS3PF irq controller Edge spe00.2
140: 0 0 PS3PF irq controller Edge spe01.0
141: 2 0 PS3PF irq controller Edge spe01.1
142: 6 0 PS3PF irq controller Edge spe01.2
143: 0 0 PS3PF irq controller Edge spe02.0
144: 2 0 PS3PF irq controller Edge spe02.1
145: 6 0 PS3PF irq controller Edge spe02.2
146: 0 0 PS3PF irq controller Edge spe03.0
147: 2 0 PS3PF irq controller Edge spe03.1
148: 13 0 PS3PF irq controller Edge spe03.2
149: 0 0 PS3PF irq controller Edge spe04.0
150: 2 0 PS3PF irq controller Edge spe04.1
151: 13 0 PS3PF irq controller Edge spe04.2
152: 0 0 PS3PF irq controller Edge spe05.0
153: 1 0 PS3PF irq controller Edge spe05.1
154: 9 0 PS3PF irq controller Edge spe05.2
155: 27210328 0 PS3PF irq controller Edge ps3fb vsync
156: 1809885 0 PS3PF irq controller Edge PS3PF stor
157: 387328 0 PS3PF irq controller Edge PS3PF stor
158: 65 0 PS3PF irq controller Edge PS3PF stor
159: 1509 0 PS3PF irq controller Edge snd_ps3pf
160: 0 78885 PS3PF irq controller Edge gbec connection
BAD: 0
Install elfspe2 and libspe2 on PS3
elfspe2 allows SPU executables to be run standalone from the commandline (aka spulets)
libspe2 is a PPU library for launching and communicating with SPU executables.

1. Copy the following files to the PS3. These files can be found on the PS3 Linux Add-On Packages CD in the spu directory.
  • libspe2-2.0.0-be0644.3.20061107.1.ps3pf.ppc.rpm
  • elfspe2-2.0.0-be0644.3.20061107.1.ps3pf.ppc.rpm
2. As root, rpm -ivh *.rpm

Install toochain on host PC
I am using Fedora Core 5 installed on a PowerPC Mac Mini as my host machine for PS3 development. Working from a PowerPC platform is extremely convinient. However, all of the following libraries are also either available as i686 packages or can be recompiled for the i686 platform if you prefer that.

cat /proc/cpuinfo (For the Host PC)
processor : 0
cpu : 7447A, altivec supported
clock : 1249.999995MHz
revision : 0.2 (pvr 8003 0102)
bogomips : 83.20
timebase : 41620997
machine : PowerMac10,1
motherboard : PowerMac10,1 MacRISC3 Power Macintosh
detected as : 287 (Mac mini)
pmac flags : 00000010
L2 cache : 512K unified
pmac-generation : NewWorld
1. Copy the following files to the host PC. These files can be found at Barcelona Supercomputing Center, Linux on Cell [bsc.es] under Programming Models -> Linux on Cell -> Cell BE Components -> GNU Toolchain.
  • ppu-binutils-3.2-4.ppc.rpm
  • ppu-gcc-3.2-4.ppc.rpm
  • ppu-gcc-c++-3.2-4.ppc.rpm
  • ppu-toolchain-3.2-4.src.rpm
  • ppu-toolchain-debuginfo-3.2-4.ppc.rpm
  • spu-binutils-3.2-6.ppc.rpm
  • spu-gcc-3.2-6.ppc.rpm
  • spu-gcc-c++-3.2-6.ppc.rpm
  • spu-newlib-1.14.0.200610300000-1.ps3pf.ppc.rpm
  • spu-toolchain-3.2-6.src.rpm
  • spu-toolchain-debuginfo-3.2-6.ppc.rpm

2. As root, rpm -ivh *.rpm
Install libspe2 on host PC
1. Copy the following files to the host PC. These files can be found on the PS3 Linux Add-On Packages CD in the spu directory.
  • libspe2-2.0.0-be0644.3.20061107.1.ps3pf.ppc.rpm
  • libspe2-devel-2.0.0-be0644.3.20061107.1.ps3pf.ppc.rpm
2. As root, rpm -ivh *.rpm

Building Hello World (for libspe2)
1. On the host PC, compile the example:

ppu-gcc -m32 ppu_hello.c -lspe2 -o ppu_hello
spu-gcc spu_hello.c -o spu_hello

NOTE: If the 64 bit support headers and libraries are installed on the host the -m32 can be omitted from the PPU compilation step.

2. Copy the two executables to the PS3.
3. To execute spu_hello using libspe2, just run ./ppu_hello
4. To execute spu_hello using elfspe2, just run ./spu_hello directly.

Hello World source (for libspe2)
ppu_hello.c
  0#include <stdlib.h>
1#include <libspe2.h>
2
3int
4main()
5{
6 unsigned int createflags = 0;
7 unsigned int runflags = 0;
8 unsigned int entry = SPE_DEFAULT_ENTRY;
9 void* argp = NULL;
10 void* envp = NULL;
11
12 spe_program_handle_t* program = spe_image_open("spu_hello");
13 spe_context_ptr_t spe = spe_context_create(createflags, NULL);
14 spe_stop_info_t stop_info;
15
16 spe_program_load(spe, program);
17 spe_context_run(spe, &entry, runflags, argp, envp, &stop_info);
18 spe_image_close(program);
19 spe_context_destroy(spe);
20
21 return (0);
22}
spu_hello.c
 0#include <stdio.h>
1
2int
3main( unsigned long spuid )
4{
5 printf("Hello, World! (From SPU:%d)\n",spuid);
6 return (0);
7}
Using the IBM SDK
The IBM SDK uses libspe not libspe2, so in order to build the IBM libraries and samples, libspe must be installed.

What is the difference between libspe and libspe2? Will both continue to be used?

libspe2 is a re-design of libspe. The folks at IBM have strongly implied that libspe is on its way out and we should expect a future revision of the SDK to be refactored for libspe2.

Roland (RSei) gave an excellent description of reasoning behind the design of libspe2 in IBM's Cell Broadband Engine Architecture forum:
"There have been a number of requirements and issues with libspe1 that led to the design of a new major version with a different API. I'll try to explain a few major aspects just briefly:

1. libspe is supposed to be the "low-level API" to use SPE resources. We think that the "SPE context" introduced in libspe2 is the better low-level construct than the "SPE thread" (as defined in libspe1), which already suggests a particular programming model and view. By using "SPE contexts", it is, e.g., possible to have other models like (synchronous) function offload to SPEs more easily without introducing the complexity and overhead of threading into an application. Another example is the possibility to exchange the code on an SPE, but leaving the data in place, which allows for easy and efficient "chaining" of processing steps und PPE control. In the thread model, this would have to rely on SPE programs using overlays. By the way, it is very easy to have the libspe1 thread model as a special case implemented on top of libspe2 and we have actually done this exercise internally.

2. Many people asked for a more complete "SPE thread library" (similar to what you usually have, e.g., in pthread). By removing the special concept of an "SPE thread" (in the libspe1 sense), we are actually addressing this requirement. When using libspe2, the programmer relies on the thread package of choice and just uses SPEs in these threads. All thread-specific aspects of the application are standard - so you have full functionality.
3. There were many complaints about the event API in libspe1 - from usability to efficiency. We think, we found a good solution in libspe2.

4. We feel that the "SPE groups" in libspe1 were tieing together rather orthogonal concepts like scheduling and event handling. So we gave up this construct. You may have noticed that we introduced "SPE gang contexts" and you have probably already guessed that we are working on gang scheduling to leverage this - but "gangs" are purely a scheduling construct and do *not* replace the previous groups.

5. You are right that binding threads to specific, physical SPEs has been part of the libspe1 API, although it had never been implemented. There are many discussions about this feature. At this point, we don't have a conclusive answer how we want to support "affinity" of threads to physical SPE resources. We simply felt we are not ready yet to define the API and stick to it in the future."
1. Copy the following files to the host PC. These files can be found at Barcelona Supercomputing Cente, Linux on Cell [bsc.es] under Programming Models -> Linux on Cell -> Cell BE Components -> GNU Toolchain.
  • libspe-1.1.0-1.ppc.rpm
  • libspe-debuginfo-1.1.0-1.ppc.rpm
  • libspe-devel-1.1.0-1.ppc.rpm
2. As root, rpm -ivh *.rpm

3. Copy the libspe libraries from the Host PC at /usr/lib/libspe.so.* to /usr/lib/ on the PS3.
4. Copy the following file onto the host PC. This file can be found at IBM alphaWorks' IBM Cell Broadband Engine Software Development Kit download page. You will need to agree to the licenses in order to download the file.
  • cell-sdk-lib-samples-1.1-10.noarch.rpm
5. As root, rpm -ivh cell-sdk-lib-samples-1.1-10.noarch.rpm. The source files should now be installed in /opt/IBM/cell-sdk-1.1.
6. Only minor modifications are needed to cross-compile the SDK.
  • cd /opt/IBM/cell-sdk-1.1
  • Open make.footer
  • Search for (starting at line 84 in my copy):
    ########################################################################
    # Common GNU Defines (Host, PPU32, PPU64, SPU)
    ########################################################################
  • Delete the following section (starting at line 91 in my copy):
    ifeq "$(HOST_PROCESSOR)" "ppc64"
    SCE_ROOT =
    SCE_SYSROOT =
    SCE_PPU_BINDIR = /usr/bin
    SCE_SPU_BINDIR = /usr/bin
    PPU_TOOL_PREFIX =
    PPU32_TOOL_PREFIX =
    else
    # SCE_VERSION is defined in environment or in make.env
    SCE_ROOT = /opt/sce/$(SCE_VERSION)
    SCE_SYSROOT = $(SCE_ROOT)/ppu/sysroot
    SCE_PPU_BINDIR = $(SCE_ROOT)/ppu/bin
    SCE_SPU_BINDIR = $(SCE_ROOT)/spu/bin
    PPU_TOOL_PREFIX = $(PPU_PREFIX)
    PPU32_TOOL_PREFIX = $(PPU32_PREFIX)
    endif
  • Insert the following section at the same location:
     SCE_ROOT =
    SCE_SYSROOT =
    SCE_PPU_BINDIR = /usr/bin
    SCE_SPU_BINDIR = /usr/bin
  • If 64 bit support is not installed, search for (line 150 in my copy):
    #********************
    # 64-bit PPU Targets
    #********************
  • If 64 bit support is not installed, delete the following lines:
    PPU64_TARGETS := $(strip $(PROGRAM_ppu64) \
    $(PROGRAMS_ppu64) \
    $(LIBRARY_ppu64) \
    $(SHARED_LIBRARY_ppu64))

    ifdef PPU64_TARGETS
    TARGET_PROCESSOR := ppu64
    endif
  • Save the changes
7. If GLUT is not installed on the host PC, install it (for Fedora-based hosts) with yum install freeglut-devel
8. The SDK and samples should now build without errors: cd src; make (Although quite a few warnings will be generated - there is a bit of non-standard compliant code in the SDK which should be fixed.)
9. Copy the following files from the host PC to the target PS3's /usr/lib directory.
  • /opt/IBM/cell-sdk-1.1/src/lib/matrix/ppu_shared/libmatrix.so
  • /opt/IBM/cell-sdk-1.1/src/lib/image/ppu_shared/libimage.so
  • /opt/IBM/cell-sdk-1.1/src/lib/vector/ppu_shared/libvector.so
  • /opt/IBM/cell-sdk-1.1/src/lib/surface/ppu_shared/libsurface.so
  • /opt/IBM/cell-sdk-1.1/src/lib/noise/ppu_shared/libnoise.so
  • /opt/IBM/cell-sdk-1.1/src/lib/fft/ppu_shared/libfft.so
  • /opt/IBM/cell-sdk-1.1/src/lib/gmath/ppu_shared/libgmath.so
  • /opt/IBM/cell-sdk-1.1/src/lib/math/ppu_shared/libmath.so
  • /opt/IBM/cell-sdk-1.1/src/lib/misc/ppu_shared/libmisc.so
  • /opt/IBM/cell-sdk-1.1/src/lib/audio_resample/ppu_shared/libaudio_resample.so
10. Now anything built with the IBM SDK should be able to run on the PS3.
Access the PS3 Over VNC
I have two Playstation 3 units and only one HDMI input on my HD TV and that one is going to be used for game playing, not developing. So the PS3 I use for development is head-less. The vast majority of the time I can accomplish everything I need by a simple secure shell to the PS3. But occasionally I want to use the machine as though I were local, and that is what VNC is for.

How to setup VNC on the PS3 (for Yellow Dog Linux):
1. Secure shell from the host to the PS3 with X11 using: ssh -X [PS3_IP_ADDRESS]
2. On the PS3, launch the firewall security settings application using: system-config-securitylevel. At this point you will need to enter the root password for the PS3.
3. Click on "Other ports", then "+ Add" and add port 5901 (TCP). This will allow the VNC connection through the firewall running on the PS3. Go ahead and close the application.
4. On the PS3, run the VNC server using: vncserver. If this is the first time you've run the server, you will need to provide a password that will be used to access the machine.
5. On the host PC, start the VNC client using: vncviewer [PS3_IP_ADDRESS]:[DISPLAY_NUMBER]. The display number was printed when the server was started. It defaults to 1 (ONE).
6. After you enter the password, you should now see the PS3 window manager running with an open shell by default.
7. In order to kill the VNC server use: vncserver -kill :[DISPLAY_NUMBER]
8. In order to use the default Yellow Dog window manager (Enlightenment), uncomment the following lines in ~/.vnc/xstartup on the PS3 and restart the server.
unset SESSION_MANAGER
exec /etc/X11/xinit/xinitrc

The only real practical difference between using the PS3 over VNC and using it locally will be if you are writing graphics to the framebuffer. These effects will only display over the locally connected display.

Upgrade libspe and libspe2
The official release of libspe and libspe2 that were available at launch have some minor issues that were patched recently. Both libraries are being actively developed and there will always be new patches available for brave developers. There is a cumulative version available through December 6.

To build and install the latest version:

1. Download the following files from [Cbe-oss-dev] libspe and libspe2 december release to the Host PC.
  • libspe-1.2.0.tar.gz
  • libspe2-2.0.1.tar.gz
The files will probably need to be renamed locally after download.
2. Untar the two files with:
  • tar xzvf libspe-1.2.0.tar.gz
  • tar xzvf libspe2-2.0.1.tar.gz
3. In the libspe2-2.0.1 directory, open the make.defines file, and change the equivalent section to be:
ifeq "$(CROSS_COMPILE)" "1"
SYSROOT ?= sysroot
prefix ?= /usr
CROSS ?= ppu-
EXTRA_CFLAGS = -m32 -mabi=altivec
else
4. Save the file, then build the patches for speevent using:
patch -p1 < initevent.diff
patch -p1 < event-public.diff
patch -p1 < make_speevent_thread_safe.diff
5. Build the library using: make; make install
6. Copy all the files (recursively) in the libspe2-2.0.1/sysroot/usr/ directory to the /usr/ directory on the PS3 and the Host PC.
7. In the libspe-1.2.0 directory, open the Makefile file, and change the equivalent section to be:
ifeq "$(CROSS_COMPILE)" "1"
SYSROOT ?= sysroot
prefix ?= /usr
CROSS ?= ppu-
EXTRA_CFLAGS = -m32 -mabi=altivec
else
8. Save the file, then build the library using: make; make install
9. Copy all the files (recursively) in the libspe-1.2.0/sysroot/usr/ directory to the /usr/ directory on the PS3 and the Host PC.

Congratulations, libspe-1.2.0 and libspe2-2.0.1 are now installed on the PS3 and will be used by the any applications which are dynamically linked to either of those libraries.

Special thanks to Dirk Herrendoerfer for both making the release available and for answering my questions on the build procedures.