Building new Linux distribution.. where to go?

Printable View

03-05-2010
LiteHacker

Building new Linux distribution.. where to go?

Hey guys. I am looking forward to building a Linux distribution that is made extremely custom to my ideas of how an OS should run. (I have some previous history of writing up Operating Systems in assembly language in both 16 and 32 bit.)

I have gained a bit of knowledge around how Linux operates and used Arch Linux for some time to familiar myself with some of the controls and directories in Linux. I tried using LFS (Linux from Scratch), and found that what it accomplishes is essentially a very slow way of doing the same as Arch Linux's package manager.

However I don't understand how to do a few specific things, which I think are central to building a new distribution:

1. What is the very most base required for Linux to run? With both LFS and Arch Linux, if you want to install the kernel, you have to install a whole bunch of other things, like bash, init, readline, kernel-headers, etc. as dependencies.
From what I understand however, the Linux kernel can run without most of these packages.
For example, I may want to have Linux boot up to turn on only one program (lets say a mysql server) and that's it. No bash or shell of any kind, no man pages, no utility commands like "rm" or "cd"
So, what is the very base minimum requirement for a Linux kernel? How do I isolate it from everything else? In theory if I have only the very base, it should take up less than a megabyte. (ie, I should be able to fit it on a floppy disk.)

2. How do I choose exactly what type of features I want from a Linux kernel? Slackware seems to give these general "hughsmp", "generic", "hugh", etc. versions of Linux. How do I choose any options available for Linux? ie being able to read ext2 or fat, or both.. Being able to execute ELF or a.out, or both.
To be able to include some features in Linux, for example handling TCP connections.. I might want to remove these if I am not using a network.

3. This might be a bit more advanced, but I didn't see how to do this anywhere: How do I detect which devices are connected to the computer? For example, if I want to detect what type of network card you have so that I can install the appropriate driver, what do I do? Is it just scanning "lspci" and "lsusb"? Or is there a way to do it without those programs?
03-05-2010
reed9

I think there is simply too much information to give a lengthy reply on a forum. (Not to say that I have that information, but in general.)

You should look at the Linux Kernel in a Nutshell book:
Linux Kernel in a Nutshell

You might want to look at some very small projects like ttylinux and TinyCore, to see how they have been able to whittle themselves down.

You'll might want to look at alternative compilers to GCC, like TCC, and/or a smaller C library to glibc, like uClibc

Coreutils can be replaced with the much smaller Busybox. But you need some way of interacting with the kernel, which means basic tools and a shell at the least.
03-05-2010
GNU-Fan

Quote:

Originally Posted by LiteHacker

1. What is the very most base required for Linux to run?

The kernel itself. It should have compiled in everything it needs to access the files on drive, like an IDE/SATA/USB driver for the motherboard and the driver of the particular filesystem (e.g. ext2).

Most probably you would also want to install a bootloader, like GRUB. Though not absolutely necessary, it provides the needed work after BIOS post in order to load&start the kernel.

Lastly, the application (i.e. sash) you would like to have run, of course.

Quote:

Originally Posted by LiteHacker

With both LFS and Arch Linux, if you want to install the kernel, you have to install a whole bunch of other things, like bash, init, readline, kernel-headers, etc. as dependencies.
From what I understand however, the Linux kernel can run without most of these packages.

Right. See above.

Quote:

Originally Posted by LiteHacker

For example, I may want to have Linux boot up to turn on only one program (lets say a mysql server) and that's it. No bash or shell of any kind, no man pages, no utility commands like "rm" or "cd"
So, what is the very base minimum requirement for a Linux kernel? How do I isolate it from everything else? In theory if I have only the very base, it should take up less than a megabyte. (ie, I should be able to fit it on a floppy disk.)

MySQL is a very heavy application to begin with. It has very much dependencies, which you would have to provide. Theoretically, all these libraries can be compiled in statically. You would end with one very big binary, a few 100MB in size.

Try to start with something small, like busybox or sash.

Quote:

Originally Posted by LiteHacker

2. How do I choose exactly what type of features I want from a Linux kernel? Slackware seems to give these general "hughsmp", "generic", "hugh", etc. versions of Linux. How do I choose any options available for Linux? ie being able to read ext2 or fat, or both.. Being able to execute ELF or a.out, or both.
To be able to include some features in Linux, for example handling TCP connections.. I might want to remove these if I am not using a network.

These options you do select when you configure your new kernel. (make menuconfig)
You choose whether you want to have support compiled in the kernel, compiled as loadable module, or not at all.

Quote:

Originally Posted by LiteHacker

3. This might be a bit more advanced, but I didn't see how to do this anywhere: How do I detect which devices are connected to the computer? For example, if I want to detect what type of network card you have so that I can install the appropriate driver, what do I do? Is it just scanning "lspci" and "lsusb"? Or is there a way to do it without those programs?

You can type lsmod and see what modules are being loaded by the kernel.
Often one device needs several modules.

It seems you are not yet familiar with the concept of static vs. dynamic libraries. I recommend you read about that first.

Then you should visit and read
Linux Kernel in a Nutshell
where you learn about how to build you own upstream kernel.
03-06-2010
alf55

Quote:

Originally Posted by LiteHacker

Hey guys. I am looking forward to building a Linux distribution that is made extremely custom to my ideas of how an OS should run. (I have some previous history of writing up Operating Systems in assembly language in both 16 and 32 bit.)

I have gained a bit of knowledge around how Linux operates and used Arch Linux for some time to familiar myself with some of the controls and directories in Linux. I tried using LFS (Linux from Scratch), and found that what it accomplishes is essentially a very slow way of doing the same as Arch Linux's package manager.

However I don't understand how to do a few specific things, which I think are central to building a new distribution:

1. What is the very most base required for Linux to run? With both LFS and Arch Linux, if you want to install the kernel, you have to install a whole bunch of other things, like bash, init, readline, kernel-headers, etc. as dependencies.
From what I understand however, the Linux kernel can run without most of these packages.

The package "init" is needed as this what the kernel calls to initialize the hardware and make the computer useful. This process does not end until the machine is shut down and it starts the next layer programs and is an ancestor of every process that is run on that machine.

The package "bash" is needed as the "init" process uses bash scripts to get the machine up and running.

The kernel-headers are needed if you intend to allow a kernel module to be build on you machine (3D-video, vmware, VirtualBox, etc) which need to be compiled for the running kernel (every time that the kernel is updated to a different version, these kernel modules need to be rebuild) which means that the C compiler, linker, etc also.

Quote:

For example, I may want to have Linux boot up to turn on only one program (lets say a mysql server) and that's it. No bash or shell of any kind, no man pages, no utility commands like "rm" or "cd"

The kernel does not have the code to do the utility command like "rm" or "cd" and the server you have decided to run may not be able to do its job without those utilities.

Quote:

So, what is the very base minimum requirement for a Linux kernel? How do I isolate it from everything else? In theory if I have only the very base, it should take up less than a megabyte. (ie, I should be able to fit it on a floppy disk.)

It will never completly fit on a floppy. There used to be (pre-livecd) distros (which some still exist) "Tom's Boot/Root" which needed two disks (one held the kernel and the other had the filesystem), "Damn Small Linux (DSL)" known to exist even today fits on a floppy (check it out and see how little you have when it is stripped down).

Quote:

2. How do I choose exactly what type of features I want from a Linux kernel?

You need the kernel source, the software development system (C, linker, etc). The first step to building a kernel is to select the features that are to be built (into the kernel or built into the kernel as a kernel module). This is done by one of these methods:
1) make xconfig
2) make gconfig
3) make menuconfig
4) make config
or
5) make oldconfig
Note:
#1 requires QT (present if you installed KDE development)
#2 requires GTK (present if you installed Gnome development)
#3 Perfered to use if #1 or #2 is not avaliable
#4 The most basic (not user friendly method)
#5 is used only if you have a an existing kernel configured and built and you have patched the kernel sources to a new version of the kernel (not to add a feature that is not yet in the kernel sources).

The first many times thought this plan on a lot of time viewing the "help" and maybe visiting web sites to know what the option is providing when included.

After the configuration, you need to
1) make bzImage
2) make modules
3) make modules_install
4) copy the build kernel (#1) into the "/boot" directory without coping over a working kernel.
5) make a "initfs/initrd" if needed and put it into "/boot"
6) add the new choice for the new kernel to your boot (GRUB or LILO) configuration file (If it is LILO remember to re-run "lilo".)

Quote:

Slackware seems to give these general "hughsmp", "generic", "hugh", etc. versions of Linux. How do I choose any options available for Linux? ie being able to read ext2 or fat, or both.. Being able to execute ELF or a.out, or both.

Linux will NOT run with a root filesystem of "FAT", "VFAT", or "NTFS" as they do not allow a UNIX link (hard or soft) and they are used in lots of places.

The ELF is needed for modern executables (built since the late 90s) and "a.out" is the old (pre-C++) format and is only needed for an executable that was built a long time ago.

Quote:

To be able to include some features in Linux, for example handling TCP connections.. I might want to remove these if I am not using a network.

Networking can not reasonably be removed as it is needed for every server.

Quote:

3. This might be a bit more advanced, but I didn't see how to do this anywhere: How do I detect which devices are connected to the computer? For example, if I want to detect what type of network card you have so that I can install the appropriate driver, what do I do? Is it just scanning "lspci" and "lsusb"? Or is there a way to do it without those programs?

Hardware detection is the job of the "haldeamon".

From what I see you have a lot of reading and research, and learn to build and run your own kernels before you will have any change of doing what you like.

NEVER EVER copy a self built kernel over the only working kernel as your built kernel might not function (It has nothing to do with how many times you have built a custom kernel, a build may fail). If you are using LILO rather than GRUB, remember to re-run lilo after every thing is in place after building a kernel.

I hope to see many posting as you move thought this path of learning...
03-06-2010
simon

while i encourage you to continue with this interesting project, one way or another, i agree with alf55 that "you have a lot of reading and research" to do, and i think that learning "to build and run your own kernels" is just one small part of what you'll need to become familiar with "before you will have any chance of doing what you like". in fact i think these are understatements: your application (a mysql server, for example) will probably expect a lot more than just the linux kernel in its environment, so if you're wanting to replace basic stuff like init and the shell, you will probably need to do an enormous amount of custom coding to get the application to work with your unique non-gnu operating system.

as you'll have noticed if you understood the various stages of building a lfs system, the parts of a gnu/linux system are often interdependent, and you can't simply remove a core os component and expect everything else to work without it. generally if you remove something you'll need to either replace it with something else, or modify other components to work without the part you've removed. this means some serious hacking: great if you're keen and are more interested in learning than the end results, but if you simply want a trimmed-down system for a server, it would be MUCH easier (and more in keeping with unix principles) to pick up someone else's project and tweak that to your liking. to build your own from the ground up could take months or even years of hard work, much of which could be duplicating the work already done on existing trimmed-down distributions.
03-06-2010
mitis2010

i think you already have some great advice and starting points, but if you really want to slim down here are two more ideas:

1. uClinux (read: you see Linux) is a project intended to bring linux to embedded resource scarce systems. With some work you should be able to make it run on your desired machine (if it's not already able to :wink:)

And if you're using a PC (or a system that has a BIOS :P)
2. coreboot - a linux meant to replace (and provide better functionality than) proprietary BIOS firmware. It is able to incorporate GRUB and other "payloads", offloading the task from your main processor.

Alternatively you could see how the coreboot guys and gals fit a linux in the BIOS firmware and still had enough room for "payloads"

hope this helps (or at least gives an interesting read :wink:)
best of luck on your project
03-07-2010
onederer

So you like the basics and the hard way?

Have you tried Gentoo? There are levels that you can play with. It depends if you want to start from the basic compilation, or some of it compiled, or finally the last level. The choice is up to you. It is a running Linux. Updates are continuous. No need to wait for the next upgrade version. Gentoo is always up-to-date.