Upgrading a 32-bit Linux system to 64-bit, in-place

Alex Franchuk

2015-10-09 23:34

For about 5 years now my primary computer has been a Linux system based off of LFS. When I first compiled it, I was hesitant to go 64-bit, because LFS on it's own doesn't provide any directions for making a multilib system, and I knew I would need 32-bit libraries.

So, I made the decision to just go 32-bit. It made sense since I only had 4GB of RAM at the time anyway. I have seen the need for more over the years (running Firefox with multiple pages loaded and running a RAM-heavy game would sometimes bring the computer to a grinding halt), but it was still manageable. But I knew that I would probably eventually upgrade to 64-bit, especially when I could get a multilib setup. I put it off for a while, but two weeks ago I decided to bite the bullet and do it.

The way I figured, it should be relatively simple. If I had a 64-bit kernel with IA32 emulation support, just swapping in the new kernel should let the system boot up as normal, running a 64-bit kernel but with all the old 32-bit binaries and libraries being used. Ideally, it would Just Work™. Naturally, after getting the system up and running, I would want to get some standard 64-bit libraries in place to get multilib applications working as well.

I searched online for any information about how to go about it, but I really couldn't find much. It turns out everyone uses distributions ☺. So I've decided to document my process for that one other person out there who is in the same boat as me. Or just for those who are curious about it...

tl;dr There were some bumps along the way, but upgrading a 32-bit system to 64-bit in-place basically works as one would expect it to.

The basic process is as follows:

Make a 64-bit toolchain for cross-compilation
Cross-compile the kernel (and modules, including graphics drivers!)
Load the system with the new kernel
Build and install a 64-bit compile toolchain and libraries (gotta have that 64-bit C library!)
Build and install 64-bit and 32-bit libraries and applications (I figure it's a good idea to keep the version of your multilib libraries the same...)
Breathe a sigh of relief

Below are some loose instructions on how you might follow the steps I took, and some more details about my experience with this process.

Make a 64-bit toolchain for cross-compilation

This one is pretty standard, and there are lots of examples and tutorials online.

You need to get GNU binutils and compile them so that they will operate on 64-bit binaries. That's usually the painless part. Then you need to get GCC and Glibc, and do a bit of a dance to compile them, since each depends on the other! Glibc needs to be compiled using the compilers that we generate from GCC, and naturally the GCC compilers need a reference C library to use! Also, for Glibc you'll need the header files of whatever kernel you plan to use.

So, you first have to build only the gcc compilers (without libgcc, which is what needs the C library).

$ make all-gcc
$ make install-gcc

And then you can build Glibc headers and some libraries (but not all of them) using the new compilers. After that, you go back and make libgcc and install it, and finally fully make Glibc and install it. Whew. For more details check out this link.

Cross-compile the kernel

To compile Glibc, you should have already had your kernel headers around. Building the kernel is simple. You'll want to run

$ make ARCH=x86_64 menuconfig

which will ensure that the .config file will have the appropriate fields for a 64-bit kernel. Then, set the proper toolchain prefix (based on whatever you did for binutils and gcc/g++) and make sure that the new kernel will be able to support 32-bit binaries with IA32 emulation enabled:

General setup --->
    (your-prefix) Cross-compiler tool prefix

Executable file formats / Emulations --->
    [*] IA32 Emulation
    <*> IA32 a.out support
    [*] x32 ABI for 64-bit mode

The x32 ABI is optional, but it doesn't hurt to have it in case you do decide to compile x32 binaries.

Build the kernel in the 'usual' way, taking care to include ARCH=x86_64 in the make commands. The generated compressed kernel image will actually be at arch/x86/boot/bzImage (or a name based on whatever compression you used for the image), but arch/x86_64/boot/bzImage will be a symlink pointing to that as well.

Now just install the kernel and add an entry in your bootloader to use the new kernel image. You should be able to now boot into a 64-bit system! Albeit, with pieces missing.

Load the system with the kernel

As I said at the end of the previous section, you ought to be able to boot into a 64-bit system now. Some init scripts may not immediately work, and if you had external kernel modules, you'll have to rebuild those since they need to be 64-bit now. But, you should be able to get in, at least to a terminal. The first time I booted, I had forgotten to enable IA32 emulation, so this quickly led to the kernel complaining that it couldn't launch init (a 32-bit binary at the time). But that was simple to fix.

If you're lucky, you will be able to get your X display server running immediately. I was not so lucky. The difficulty arose because I have an Nvidia graphics card, so I had to use an external kernel module. However, the normal Nvidia driver installer assumes that your environment and kernel will be the same architecture as your X server! That is a reasonable assumption to make, but in this case I needed to compile a 64-bit module for the graphics card, but a 32-bit X server module. I solved this by getting the 32-bit Nvidia release, which had a nice 32-bit X server module sitting pre-compiled for me to copy. Another quick solution, just to get a graphical environment (if you want it) is to change the X server driver to a generic one that doesn't need a kernel module, such as VESA.

I've since recompiled my X server to be 64-bit, with 32-bit libraries installed for multilib compatibility.

Build and install a 64-bit compile toolchain and libraries

At this point, once you're in a 64-bit world, you'll want a toolchain which is adjusted for the paths of the new system. I decided to put all 64-bit binaries in /lib64 and /usr/lib64, allowing the 32-bit binaries to remain in /lib and /usr/lib. Then, just build and install GNU binutils, Glibc, and GCC (for starters) with the lib directory set appropriately.

The helpful guidance and tips from William Feely were invaluable in getting all the pieces working, as he's taken the LFS guide and has been making adjustments so that someone can generate a multilib LFS system. His BLFS guide and wiki were helpful as well, although at the time of this writing the BLFS guide was mostly just like the normal BLFS. The wiki was documenting the changes he was going to make to the BLFS guide.

Only a little while into my process of compiling native libraries did I realize that something was definitely wrong with my g++ cross-compiler. I'm not sure what went wrong there, but after doing some careful compilation, I was able to compile my new Glibc, gcc, and g++, and then use the new ones (which were correct as far as make test is concerned) to generate those libraries that were affected. The symptoms were, needless to say, some segfaults and all kinds of weird things, normally involving (from what I could tell) pthreads. I'm sure there were some other issues too. For instance, the defective g++ cross-compiler was unable to generate exceptions! Well, it would generate them, but the c++ runtime would always experience a SIGABRT when encountering them, so that wasn't good at all. I mean, having an entire, IMPORTANT feature of the c++ language not work at all would definitely generate some buggy libraries!

On another note, ALWAYS run make test or make check, where appropriate, for the basic binaries and libraries of the system. It's much better to find out early that your system has a buggy binary or library rather than after it is integrated and used by other parts of the system. This might not be as important for more specific binaries or libraries that you install, but it is essential for the libraries that will be used very often, or have critical functionality.

Build and install 64-bit and 32-bit libraries and applications

Like I said in the summary, it's a good idea to install for both 64-bit and 32-bit, since having your multilib libraries at the same version and API and all that (rather than just keeping whatever version you previously had on the 32-bit system) is a good idea. So just go through the fun process of installing the basic libraries and applications you'll be using. So far, I've installed the X server (and libraries), GTK2, Python2 and Python3, fluxbox, and a handful of other libraries and binaries which I felt should be 64-bit.

I have kept a lot of the 32-bit binaries I had (because the whole point of this way of upgrading was to take advantage of what I already had), but I'll upgrade them to be 64-bit binaries on an as-needed basis. So far though, everything is pretty stable!

Config.site is your friend

My recommendation for this stage is to set up some useful config.site files. I placed them at /etc/config.site and /etc/config-32.site, and set CONFIG_SITE=/etc/config.site to be exported in my shell startup scripts. Then, compiling the 64-bit version of any autoconf package can be done with:

$ ./configure

and compiling the 32-bit version is as simple as:

$ env CONFIG_SITE='/etc/config-32.site' ./configure

For reference, my config.site looks something like this:

test "$prefix" = NONE && prefix='/usr'
test "$libdir" = '${exec_prefix}/lib' && libdir='${exec_prefix}/lib64'
test "$libdir" = '${prefix}/lib' && libdir='${prefix}/lib64'

if test "$prefix" = '/usr';
then
    test "$sysconfdir" = '${prefix}/etc' && sysconfdir='/etc'
    test "$sharedstatedir" = '${prefix}/com' && sharedstatedir='/var'
    test "$localstatedir" = '${prefix}/var' && localstatedir='/var'
fi

if test "$prefix" = '/usr/local';
then
    test "$sysconfdir" = '${prefix}/etc' && sysconfdir='/etc'
    test "$sharedstatedir" = '${prefix}/com' && sharedstatedir='/var'
    test "$localstatedir" = '${prefix}/var' && localstatedir='/var'
fi

export PKG_CONFIG_LIBDIR="/usr/lib64/pkgconfig"

My config-32.site is similar, except it doesn't change the libdir at all, and defines

CC="gcc -m32"
CXX="g++ -m32"
export PKG_CONFIG_LIBDIR="/usr/lib/pkgconfig"

at the end instead. It probably should set the autoconf build argument too, but so far I've had success with just setting gcc and g++ to compile for 32-bit for almost every package. I think there were one or two where I had to specify --build=i386-pc-linux-gnu, but I can't remember exactly which ones.

The PKG_CONFIG_LIBDIR definitions are necessary so packages that use pkg-config to see what you have installed (which is MANY of them) don't think you have a 64-bit library installed when you really only have the 32-bit version installed, and visa versa. It also ensures that the package will link against the correct version. It took me a little while (and a few botched builds) to realize this issue.

Python bites

All right, cute titles aside, Python really is annoying for multilib systems, because they hard-code the lib directory! The autoconf libdir variable is used for installation, but no matter what that is, the lib directory Python will look in for modules will always be {prefix}/lib/python{version}/..., and this isn't defined in a single place for you to change so that, for instance, our 64-bit python will look in {prefix}/lib64/.... However, after a bit of reading through the files, I was able to figure out a set of changes which, so far, has worked well for me. Note that this is necessary for both Python2 and Python3, and I didn't make a patch file for the reason that it would (probably) only work for the single version(s) of Python that I installed.

You should make changes to the following files after running configure. Also, the Python configure script doesn't properly propagate the configure variables loaded from config.site, so you should specify the prefix, libdir, sysconfdir, and other flags manually to configure. You should change all of these files so that occurrences of 'lib' are instead 'lib64', when building for 64-bit.

Makefile so SCRIPTDIR is correct (you COULD use a shared SCRIPTDIR, but I prefer to keep it separate, since installed modules may have compiled, architecture-specific parts)
Modules/getpath.c
Modules/Setup (if applicable, for SSL and zlib)
Lib/distutils/command/install.py
Lib/distutils/sysconfig.py
Lib/platform.py (optional)
Lib/site.py
Lib/sysconfig.py

After making these changes and installing, you should also link things appropriately to the multiarch wrapper as described on the BLFS wiki for Python2 from William Freely.

Breathe a sigh of relief

You did it! It's a bit of a trek, but I thought it was a good experience, and I learned new details about how gcc sets its default paths, among other things. I had to understand that thoroughly when fixing my broken g++ cross-compiler and creating the new native compilation toolchain. But after it is all said and done, my system is working quite well, and now I can even add more RAM without enabling PAE! So that's always cool.