trinque

Find here the genesis vpatch¹ for my Python implementation of the "Online Codes" erasure code. See the following for an explanation of OC's applicability to decentralized swarm downloads. I consider this algorithim implemented atop Pest as new message types a prime candidate for rapid, decentralized, uncoordinated download of large files or broadcast data-streams (audio, video, etc).

This implementation provides the core algorithm and relevant tests. Since this work I've shifted attention to reimplementing in Common Lisp, as I found my attempts to parallelize the Python version were making the code less clear. The algorithm itself can easily scale horizontally. That said, someone more familiar (or more patient) with Python's multiprocessing module may unlock at least another order of magnitude of performance out of this implementation.

1. This supersedes the previous WIP version. [↩]

In Martin Heidegger's What is Called Thinking, Heidegger points to Nietzsche's "the wasteland grows" as the descriptor of our time. By this he meant that our structure of meaning is disintegrating, and with it our ability to self-model. We are awash in and are the last-men, reciting the words of imagined voices of authorization as we erode.

Why have we gone mad? Heidegger via Nietzsche names the death of God, the all-superlative, all-integrator, the intergenerational foundation of western meaning. Nietzsche writes "we have killed him", though it would be better to say that Nietzsche marked the turning past-which God could no longer be constructed. As Jaynes describes in The Origin of Consciousness, this is not a unique historical event. The pantheon grew quiet in many cultures when the languages of their worshipers grew to sufficient complexity and capacity for self-reference, replaced by the dipole of mono-god and atomic worshiper, and the novel "internal narrative". Now, evidence for either end of the dipole is wanting; the self and superlative other drift apart and darken.

Nietzsche proposed a way out. His Zarathustra preaches a message of self-deification, of becoming the superman. It would be an error to imagine this meant that one must become like the old gods, the god-kings, those of divine right and infallibility. We've been down that road together, haven't we, and we know how it ends. Nietzsche was not proposing we return to the past; indeed, he proposed a turning *away* from precedent. What thwarts our exiting through Nietzsche's door? To him, it is our desire for revenge. "Behold, the wasteland grows. Woe to him who hides wastelands within."

The Nietzschean ubermensch is he who exists as a relationship between himself and the ideal being. It is a focal superlative, ever ahead. Nietzsche's concept of the eternal recurrence wraps the ubermensch such that past-present-future are connected. While the ancients considered cyclical time fact,¹ there is utility in asking oneself to weigh choices as though they must be lived an infinite number of times. Whether cyclical or an arrow, one's actions stand eternally in time, their generations of consequences unending. The man's identity is the process of history understood as the endless project to define.

Psychological operating systems are not chosen.  They are imposed by culture, environment, and happenstance of experience. Read Heidegger, read Nietzsche, and see if their combined genius does not impose itself upon you as it has me, drawing a line of Promethean fire to the superlative future, the eternal project.

1. And we'd be arrogant to dismiss this with incomplete systems of physical law in our hands. [↩]

Find here a WIP implementation of the "Online Codes" erasure code in Python. There's much room for optimization, both in the selection of encoding parameters and use of e.g. "multiprocessing" for parallelism.¹ This encoding scheme can yield rapid, uncoordinated swarm downloads, reliable one-way communication over lossy channels. In the context of Pest, it may provide a necessary component of a future P2P file-sharing scheme.

1. Both encoding and decoding should easily parallelize, ignoring hurdles imposed by Python itself. [↩]

Before we continue our enumeration of components, let's discuss how a Linux system boots on an ATX-compatible computer. We'll focus on BIOS/MBR for now.¹

First, the power button is pressed, which causes the PSU to turn on. Meanwhile, the motherboard begins sending "reset" to the CPU. The PSU, having established voltages within acceptable ranges, sets voltage high on the "PWR_OK" pin. This causes the motherboard's chipset to release the CPU from reset. The CPU then executes the BIOS from system ROM.² The BIOS performs a power-on self test, and barring any terminal errors continues the boot process. This may involve executing the BIOS of video cards, hard drives, etc.³ After several lifetimes of initialization, which we'll elide for now, the computer is on and it's time to load an operating system from an attached hard drive.

The BIOS initiates this process via interrupt 0x19h, the handler of which makes successive attempts to load a master boot record at memory address 0x7C00 and execute.⁴ The first sector of a bootable hard drive⁵ contains contains the MBR⁶. This contains executable machine code,⁷ a unique-ish drive identifier,⁸ two null bytes,⁹ the partition table which reserves 16 bytes per partition,¹⁰ and the terminating two-byte "signature" of 0xAA55.¹¹ Having identified a valid MBR, execution jumps to its memory address. Given the significant storage space restrictions in the MBR, the so-called 1st-stage boot-loader does little else than to locate, load, and execute a larger 2nd-stage boot-loader located somewhere else on the drive. The second-stage boot-loader in turn loads and executes an operating system kernel, in this case the Linux kernel.

The initialization process of the Linux kernel will require a series of articles all its own. For now, we'll move on. When kernel initialization is complete, the final stage of the boot process involves handing control over to a user-space program called init. This program controls the rest of the boot process, and is also responsible for graceful shutdown and reboot. There are customary features and behaviors for an init program, but all that's strictly necessary is that it never exit, for an exit from init will result in a kernel panic. Here we'll need to make another decision, so here we'll resume our analysis of options.

Busybox

This software is a damned treat. It was written with embedded systems and initramfs in mind, providing a thorough and useful set of user-land software. The size constraints of embedded systems have kept it from ballooning uncontrollably as Linux utilities more commonly used on desktop systems have. For not much more than the cost of bash¹² you get an init, device node management, networking utilities, archivers, text editors... indeed, all the shell commands you might expect.¹³ If there's a compelling case why Busybox is an insufficient base system, I'd love to know why.

busybox-1.31.1 cloc .
    3726 text files.
    3114 unique files.
    1727 files ignored.

github.com/AlDanial/cloc v 1.70  T=13.75 s (145.6 files/s, 24038.0 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C                               675          27959          60252         180543
C/C++ Header                   1115           1415           2721          31012
Bourne Shell                    179           1556           1468           8658
HTML                             10           1853             32           7324
C++                               1            166             62           1197
make                              6            307            451            911
yacc                              1             93             20            570
Perl                              3            100            185            337
lex                               1             40             12            303
NAnt script                       1             84              0            260
Bourne Again Shell                5             51             94            234
Python                            1             12             13            110
Qt Project                        1              0              0             33
awk                               1              2              8             30
bc                                1             10              0             25
diff                              1              1              8             15
--------------------------------------------------------------------------------
SUM:                           2002          33649          65326         231562
--------------------------------------------------------------------------------

Not-busybox

Assuming we reject busybox, we'll still need to assemble much of what it provides. We'll make some conservative selections going forward, and see where we end up.

sysvinit

Here's a simple standalone init program. Aside init, several other redundant programs are provided, including one for killing everything but init,¹⁴ and another which provides a rudimentary logger.¹⁵

sysvinit-2.95 cloc .
      71 text files.
      66 unique files.
      38 files ignored.

github.com/AlDanial/cloc v 1.70  T=0.30 s (108.8 files/s, 40593.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                               21           1262           2218           8004
C/C++ Header                     9             51            205            258
make                             2             55             25            202
Bourne Shell                     1              3              3             28
-------------------------------------------------------------------------------
SUM:                            33           1371           2451           8492
-------------------------------------------------------------------------------

coreutils

This is the GNU incarnation of the standard UNIX utilities. It's heavier than all of busybox, and doesn't even give you the shell to run the 76k lines of shell script.

coreutils-8.31 cloc .
    2878 text files.
    2858 unique files.
     484 files ignored.

github.com/AlDanial/cloc v 1.70  T=11.84 s (202.2 files/s, 40526.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                              870          29351          36181         151337
Bourne Shell                   637          24014          20200          75874
C/C++ Header                   373           7551          12925          40036
m4                             423           2459           2498          36820
make                            16           3502           3068           7920
Perl                            70           1819           2409           7671
TeX                              1            811           3695           7163
yacc                             1            279            309           1840
Python                           1             12              9             48
sed                              2              0              0             16
-------------------------------------------------------------------------------
SUM:                          2394          69798          81294         328725
-------------------------------------------------------------------------------

bash

So here's the shell then, too. We're up to 536k lines by now, and we're still missing quite a lot.

bash-5.0 cloc .
    1251 text files.
    1219 unique files.
     755 files ignored.

github.com/AlDanial/cloc v 1.70  T=5.80 s (85.6 files/s, 46369.2 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C                                      258          20944          20498         107295
HTML                                     3           3854             38          26338
Bourne Shell                            36           3333           3388          20114
Windows Module Definition               44           2581             11          15096
C/C++ Header                           111           2821           3506           7617
TeX                                      1            821           3462           6762
yacc                                     2            824            968           5398
m4                                       4            478            439           4742
Perl                                     2            535            834           4229
Bourne Again Shell                      27            235            346            994
make                                     3             48             36            110
Assembly                                 2             11             20             48
awk                                      1              8             15             24
sed                                      2              0              0             16
---------------------------------------------------------------------------------------
SUM:                                   496          36493          33561         198783
---------------------------------------------------------------------------------------

net-tools

We'll need these to manipulate our network interfaces if we pass on busybox. 549,362 and counting.

net-tools-1.60 cloc .
     152 text files.
     131 unique files.
      69 files ignored.

github.com/AlDanial/cloc v 1.70  T=0.26 s (314.7 files/s, 65320.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                               67           1767           1571          12322
C/C++ Header                    11            133            116            680
make                             4             87            122            257
Bourne Shell                     1             14             56            103
-------------------------------------------------------------------------------
SUM:                            83           2001           1865          13362
-------------------------------------------------------------------------------

gzip

Busybox comes with its own implementation of several archive formats implemented in about 14k lines. Meanwhile, just a standalone GNU gzip is about four times fatter. 609,918 lines of code to know, own, and understand and we don't even have a text editor yet.

gzip-1.3.14 cloc .
     362 text files.
     352 unique files.
      70 files ignored.

github.com/AlDanial/cloc v 1.70  T=0.70 s (419.6 files/s, 123880.3 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
Bourne Shell                            19           4519           3206          21230
C                                       86           2697           4765          15743
m4                                     106            666            742          10168
C/C++ Header                            69           1530           2666           5905
TeX                                      1            731           2941           5619
make                                     6            628            493           1677
Assembly                                 1             21             38            179
DOS Batch                                1              0              0             18
Perl                                     1              1              2             12
Windows Module Definition                2              0              0              5
---------------------------------------------------------------------------------------
SUM:                                   292          10793          14853          60556

eudev

Our machine needs to make new device nodes for when new hardware is connected.¹⁶ Let's select one of the cheaper device management daemons from Gentoo. 638,677 lines.

eudev cloc .
     205 text files.
     203 unique files.
      60 files ignored.

github.com/AlDanial/cloc v 1.70  T=0.39 s (371.4 files/s, 101693.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                               71           4844           3961          22185
C/C++ Header                    43            690            853           2124
Perl                             2            108             36           1753
XML                              4            120             16           1542
make                            16            121              8            490
m4                               2             74             10            380
Python                           2             38             46            225
Bourne Shell                     3             10              6             25
Markdown                         1              8              0             21
YAML                             1              0              0             14
-------------------------------------------------------------------------------
SUM:                           145           6013           4936          28759
-------------------------------------------------------------------------------

---

We're not nearly done selecting the components that would fill the rest of the gaps in our non-busybox system, and we're already at nearly three times the line count. If there are not significant gains to be had from the additional features found in the heftier implementations of these utilities, it would be difficult to justify their presence in a system which much be understood by its maintainers, instead of merely excreted by them. As we proceed, we must remain cognizant of how much complexity we've chosen to accumulate beneath us. Until next time.

1. Note that GPT partitioning does not require use of UEFI. This was a point of confusion on my part, and perhaps yours! [↩]
2. This is mapped to the memory address 0xFFFFFFF0 on x86. [↩]
3. Yes, your computer is made of computers, which are made of... aw hell. [↩]
4. If anyone knows what hysterical raisins came up with this address, let me know. [↩]
5. Go on, take a look at your own. dd if=/dev/sda of=mbr.bin bs=512 count=1; hexdump mbr.bin [↩]
6. 512 bytes due to historic hard drive sector size. These days sector sizes of 4k are common. [↩]
7. 0x000-0x1b7 [↩]
8. 0x1b8-0x1bb [↩]
9. 0x1bc-0x1bd [↩]
10. 0x1be-0x1fd [↩]
11. Note that your hex editor will display this the other direction on a little-endian architecture. [↩]
12. !!!!!!!!!!! [↩]
13. Not to mention, a full-featured shell that will happily pretend to be bash for you, if you like. [↩]
14. Surely this can be accomplished with a simple shell loop and `kill`? [↩]
15. Aside which you probably intend to install a syslog daemon, so why have two? [↩]
16. A USB drive or peripheral, perhaps. [↩]

We began with definitions last time. Let's add a few more before we press on.

Complexity

The English sources at my disposal underwhelm, so let's instead take a stab at the Latin beneath. Plecto means "I weave" or "I twist."¹ Com- is the familiar prefix denoting "with". Therefore we may not sin too much calling complexity the measure of how weaved-with an item is, either in itself or with its surroundings.² In software, there are two kinds of complexity worth mentioning, accidental and essential.³ Essential complexity is that which cannot be reduced without changing your algorithm. Accidental complexity is an accumulation of human error, of engineering tradeoffs and other failings.⁴ Given the depth-first-search toposort, the essential complexity is linear with the number of vertices and edges, O(V + E). Notice that nowhere do you find terms for whether you used a recursive or iterative solution, whether your recursive solution was implemented on a system with tail-recursion optimization, or whether your program's process was suspended while the operating system received an interrupt. In those invisible terms lie the accidental complexity, and they comprise most of the hulking mass of modern computing.

Ownership

The old language had an entire declension for ownership.⁵ To own something is for that thing to be an extension of yourself, for your causes to flow through it. It means the object is ready to your hand. What does it mean when you say "my country"? Does the country move when you move? What do you mean by "my computer"? Does your light shine out the other side of it, amplified, or does it confound you, complicate your actions, and bury you in ever-increasing cost?

---

We're a few lost lifetimes away from a functional Linux system yet. Let's continue.

Libc

We're building an operating system around the Linux kernel, which exposes an interface of system calls to user-space programs. We could conceivably implement these system calls ourselves, along with the myriad utility functions present in most libc libraries, but we'd be treading ground already trod at great cost. Of the available options, I'll discuss these two.

musl

This thing is slim, performs well, and has good compatibility with POSIX⁶. See here for a comparison with other libc libraries. Moreover, consider this from the musl FAQ:

When will it be finished?

When there's nothing left to remove.

musl-1.1.24 cloc .

    2521 text files.
    2331 unique files.
     247 files ignored.

github.com/AlDanial/cloc v 1.70  T=9.07 s (252.8 files/s, 11741.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             1448           6314           7295          48270
C/C++ Header                   537           4316            273          30931
Assembly                       299            490            539           6485
Bourne Shell                     4            111            160            618
awk                              3             56             74            301
make                             1             68             10            157
sed                              1              0              6              9
-------------------------------------------------------------------------------
SUM:                          2293          11355           8357          86771
-------------------------------------------------------------------------------

After weighing the thing, I'm truly in love.

glibc

Meanwhile, lets strap a loathsome pig down and take a few measurements.

glibc-2.30 cloc .
   16983 text files.
   16253 unique files.
Complex regular subexpression recursion limit (32766) exceeded at /usr/bin/cloc line 7327.
    3020 files ignored.

github.com/AlDanial/cloc v 1.70  T=69.28 s (201.9 files/s, 25976.2 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C                                     8781         114862         172458         683447
C/C++ Header                          2882          38763          64340         282848
Assembly                              1821          39782          93919         235315
Bourne Shell                            82           1991           2558          15567
make                                   203           2498           2476          11080
Python                                  44           1173           2376           9013
TeX                                      1            813           3697           7162
m4                                      50            349            144           3467
awk                                     38            291            422           2054
Windows Module Definition               17            190              0           2024
C++                                     32            258            310           1238
Perl                                     3             87            142            632
Korn Shell                               1             55             68            435
yacc                                     1             56             36            290
Pascal                                   7             82            326            182
JSON                                     2              0              0             91
Expect                                   8              0              0             77
Bourne Again Shell                       1              8             31             77
sed                                     11              8             15             70
---------------------------------------------------------------------------------------
SUM:                                 13985         201266         343318        1255069
---------------------------------------------------------------------------------------

86k vs 1.23m lines. I get physically ill looking at wanton messes like these. The utter contempt for human comprehension is abhorrent.⁷

If somebody else solves the same problem as you sparing over an order of magnitude, you congratulate them and get out of their way.

---

We still don't have that blinking cursor, but we're getting closer. There are still explosions of accidental complexity to avoid in the days ahead. See you tomorrow.

1. And because the language is so damned orderly, also "I punish". [↩]
2. Latin would happily allow you to disambiguate the two if you wanted! [↩]
3. These are much better discussed here http://curtclifton.net/papers/MoseleyMarks06a.pdf, and I encourage you to go give that a read before continuing. [↩]
4. Consider that mathematical truths are discovered, while products of beings-an-engineer are built. [↩]
5. The genitive or "begetting" form. [↩]
6. POSIX, first codified in 1988, is a standard for the APIs and utility programs of UNIX-like operating systems. Adhering to the standard means that a huge amount of software designed for POSIX can be ported to a given operating system. [↩]
7. Yes, we glossed over the plump, moo-eyed smiles of the Linux kernel and gcc because we have no alternatives. That's no excuse. [↩]

Let's chart a course to a usable system while usable systems can still be built, shall we?

Definitions first

By Republican, we do not simply mean that the item shall be used by the Republic. We mean that the item shall be constructed in such a way that it embodies, serves, and furthers Republican ideology in computer operating systems. As there can be no meaning absent a structure of authority, "Republican OS" is predicated on the Republic being the structure of authority within which said item is defined.¹ Thus relying on "upstream"² for anything is sin. Fallen creatures that we are, I do not expect us to eradicate sin in an instant. Yet inaction is not an option. Inaction is death, both of the Republic and ourselves. From our options we cleave both immediate perfection and tolerance of anything which does not serve us. What remains is a ratchet; it begins loose and ever tightens, squeezing out the unnecessary, the confounding, the redundant, and leaves us with something small enough to fit in our hand.³

By operating system, we mean a complete set⁴ of self-hosting software capable of running on computers which can be obtained with reasonable effort. Self-hosting means that the system can be used to edit, build, and collaborate on next iterations of the software. It means that we turn the ratchet ourselves. Understand that the first build of this self-hosting system incurs the bootstrapping problem, and understand that we do not currently have a solution to this problem.⁵

Hardware selection

We must target particular hardware on which to build this system. This means AMD, or ARM64 machines, and to date it isn't clear which.⁶ Whichever hardware we choose, should we be forced to rely on items we cannot ourselves build, alter, and maintain,⁷ we proceed in sin, and leave debts to pay.

Compiler

Building software, we must choose a compiler which is capable of producing binaries for our target architecture. There has been much discussion already on the precise place to break. Absent deep research on the matter, one can only lean on empirical results, and by those, gcc 4.x seems to have been capable of our needs to date. This is a weak recommendation, and given the hulking mass of the pile in question, it might be all we have the lifetimes to achieve. gcc-4.9.4 being the last bugfix release before the next major revision, it seems as good a place as any to pull the handbrake. It's worth pointing out that this version made the following change:

GNAT switched to Ada 2012 instead of Ada 2005 by default.

Let's weigh the thing before we move on.

gcc-4.9.4 cloc .
   76640 text files.
   75753 unique files.
Complex regular subexpression recursion limit (32766) exceeded at /usr/bin/cloc line 7327.
Complex regular subexpression recursion limit (32766) exceeded at /usr/bin/cloc line 7327.
Complex regular subexpression recursion limit (32766) exceeded at /usr/bin/cloc line 7327.
Complex regular subexpression recursion limit (32766) exceeded at /usr/bin/cloc line 7327.
    4671 files ignored.

github.com/AlDanial/cloc v 1.70  T=338.66 s (212.7 files/s, 30282.2 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C                                    23126         444405         462198        2326698
C/C++ Header                         10280         155822         158227         758024
Ada                                   5052         264536         349229         756828
Java                                  6346         169198         646043         682350
C++                                  17407         137657         176833         612276
Bourne Shell                           152          81229          68484         440850
Markdown                               346          40753              0         339466
Go                                    2170          37979          49221         298060
HTML                                   330          33462           5663         148765
Fortran 90                            4071          17275          31443         101132
m4                                     198           8644           2443          77132
Assembly                               544          13218          32088          58482
XML                                     60           5961            563          44147
Windows Module Definition              122           4104             38          30404
make                                   157           4088           1653          25751
Expect                                 282           5596           9909          22452
Objective C                            512           4852           3066          16604
TeX                                      2           1480           6149          11675
Fortran 77                             431           1077           3850          10659
Objective C++                          242           2388           1502           8020
Perl                                    30            903           1349           4686
MSBuild script                           7              1              0           4675
Pascal                                  13            790           3261           3465
Python                                  10            804            736           2855
XSLT                                    20            563            436           2805
awk                                     18            374            589           2393
OCaml                                    3            310            416           2279
Bourne Again Shell                      15            415            654           1865
CSS                                      9            332            143           1428
yacc                                     2            107            119            977
C#                                       9            230            506            879
Tcl/Tk                                   1             72            112            393
lex                                      1             34             30            156
CMake                                    1             27             31            153
NAnt script                              2             17              0            132
JavaScript                               2             20             81            122
Haskell                                 36             15              0            112
Windows Resource File                    2              3              2             67
SAS                                      1             14             22             32
DTD                                      3             28             70             26
Fortran 95                               2             10              8             21
Lisp                                     1              4             12              8
MATLAB                                   1              0              0              5
DOS Batch                                2              0              0              4
XHTML                                    1              6             16              3
---------------------------------------------------------------------------------------
SUM:                                 72022        1438803        2017195        6799316
---------------------------------------------------------------------------------------

The damned thing broke the tool I was using, so this may be an incomplete result! At any rate, hefty...

Bootloader

The system must boot, and therefore we must select a bootloader. Here are the options. If you know of another, write in.

Syslinux

Apparently this thing can boot far more than LiveCDs these days. If this claim holds true, it might be a way of avoiding having multiple bootloaders present in the system (one for install media, and the other for the installed system). Things like being able to display graphics could probably be chopped out without too much effort. It supports EFI, which permits us use of disks larger than 2tb. It's large, but appears to be a somewhat modular collection of bootloaders. It'd be worthwhile to discover how heavy extlinux is by itself.

syslinux-6.03 cloc .
    2905 text files.
    2760 unique files.
     344 files ignored.

github.com/AlDanial/cloc v 1.70  T=14.98 s (171.1 files/s, 41420.3 lines/s)
-----------------------------------------------------------------------------------
Language                         files          blank        comment           code
-----------------------------------------------------------------------------------
C                                 1147          50307          78851         255734
C/C++ Header                       856          22631          46656         108013
HTML                                 8           3974             11          13407
Assembly                            95           1816           4745           9239
D                                  296              0              0           5825
Pascal                              30            607             20           4809
Perl                                40            720            828           3546
make                                65            997           1140           2896
xBase Header                         7            308            844           1212
Python                               1             34              6            267
Bourne Shell                         7             40             17            179
Lua                                  2             20              5            161
Bourne Again Shell                   3             15             29            150
CSS                                  2             18              0            105
XML                                  1              0              0             39
diff                                 1              7             34             32
Windows Resource File                1              2              0             24
-----------------------------------------------------------------------------------
SUM:                              2562          81496         133186         405638
-----------------------------------------------------------------------------------

Lilo

It only supports MBR partition schemes (which incur the aforementioned 2tb limitation). I'm not aware of any way to boot from CDs using lilo, though one may exist, but I've had no problem using it to boot USB sticks. And the thing is damned slim!

lilo-24.2 cloc .
     251 text files.
     240 unique files.
     113 files ignored.

github.com/AlDanial/cloc v 1.70  T=1.29 s (106.7 files/s, 34785.2 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C                                19           1702            558          11821
Assembly                         21           1683           2627           9178
HTML                             34            131             67           5701
TeX                               8            550            394           3238
C/C++ Header                     22            532            787           1379
Perl                              4            262            356           1295
make                             12            195            148            592
Bourne Again Shell                1             87             34            578
Bourne Shell                     15            133            195            473
CSS                               2              9              4            280
--------------------------------------------------------------------------------
SUM:                            138           5284           5170          34535
--------------------------------------------------------------------------------

Elilo

This obviates the 2tb drive limit, and is even smaller than classical lilo! That said, much of the complexity may be hiding in EFI. I hear there's inquiry into EFI ongoing, so I'll await spyked's report for now.

elilo-3.16 cloc .
     119 text files.
     113 unique files.
      25 files ignored.

github.com/AlDanial/cloc v 1.70  T=0.46 s (204.5 files/s, 52649.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                               46           2746           3715          10905
C/C++ Header                    35            792           1643           2969
Assembly                         5             89            407            430
make                             8            111            213            184
-------------------------------------------------------------------------------
SUM:                            94           3738           5978          14488
-------------------------------------------------------------------------------

Grub

There are two paths to consider here, the "legacy" branch and the 2.x branch. The former is quite manageably-sized, but does not support EFI.

grub-0.97 cloc .
     193 text files.
     184 unique files.
      45 files ignored.

github.com/AlDanial/cloc v 1.70  T=1.03 s (144.2 files/s, 84368.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                               65           7117           9156          32871
Bourne Shell                    12           1242           1625           9938
C/C++ Header                    49           1374           2986           5915
TeX                              1            590           2464           4032
Assembly                         9            807           1324           1913
m4                               3            308             89           1700
make                             8             99             54            459
Perl                             1             91             87            339
-------------------------------------------------------------------------------
SUM:                           148          11628          17785          57167
-------------------------------------------------------------------------------

Meanwhile the 2.x branch is unsurprisingly plump, but does support EFI.

grub-2.04 cloc .
    2228 text files.
    2157 unique files.
     377 files ignored.

github.com/AlDanial/cloc v 1.70  T=11.32 s (164.6 files/s, 50901.3 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C                                      931          44555          37610         264712
C/C++ Header                           563          10793          21301          53226
Bourne Shell                            18           6803           4182          32919
make                                    10           2987            469          25152
m4                                     150           1485           1504          20611
TeX                                      2           1512           6521          12598
Assembly                               172           3186           6809          10329
Windows Module Definition                5            651             21           3627
Python                                   3            168            213           1379
lex                                      1             54             34            305
yacc                                     1             44             44            267
awk                                      1              6             16             80
C++                                      1              8             17             80
sed                                      5              3              0             59
Lisp                                     1              3              0             50
---------------------------------------------------------------------------------------
SUM:                                  1864          72258          78741         425394
---------------------------------------------------------------------------------------

Refind

This one's slim, supports EFI, and appears to be the work of a single guy. Should we proceed down the path of EFI, it appears to be worth further investigation.

refind-0.11.4 cloc .
     312 text files.
     307 unique files.
     114 files ignored.

github.com/AlDanial/cloc v 1.70  T=1.71 s (115.7 files/s, 51557.9 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C                                70           6296           8340          36255
C/C++ Header                     84           2769           7447          14805
HTML                             20           2170              2           6581
Bourne Again Shell                5            106            347           1524
Python                            5            148            222            494
make                             10            125            123            313
CSS                               1             13              2             73
Bourne Shell                      2              8             25             29
Perl                              1              1              0              6
--------------------------------------------------------------------------------
SUM:                            198          11636          16508          60080
--------------------------------------------------------------------------------

Kernel

Our chosen hardware will require a kernel which supports it. Regardless of which kernel version we choose⁸ this is likely to be the second-hardest component of the system after hardware fabrication to maintain by Republican effort alone in the near term. The codebase is gargantuan even for older versions. Lets first take a look at 2.6.39.2, the last 2.6 release.

linux-2.6.39.2 cloc .
   36684 text files.
   36131 unique files.
    4074 files ignored.

github.com/AlDanial/cloc v 1.70  T=244.28 s (133.5 files/s, 56149.6 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C                             16087        1501167        1531763        7743098
C/C++ Header                  13589         314737         536239        1632163
Assembly                       1217          39835          92468         204275
XML                             139           3119            948          40974
make                           1390           6004           6374          22654
Perl                             41           2973           2462          13900
Bourne Shell                     61            638           1475           3644
yacc                              5            453            322           2987
Python                           21            594            343           2727
C++                               1            209             57           1521
lex                               5            202            237           1318
TeX                               1            108              3            911
awk                               8             90             79            714
Bourne Again Shell               28             74             55            446
HTML                              2             58              0            378
NAnt script                       1             87              0            356
Pascal                            3             49              0            231
Lisp                              1             63              0            218
Objective C++                     1             55              0            189
ASP                               1             33              0            137
XSLT                              6             13             27             70
sed                               1              0              3             30
vim script                        1              3             12             27
--------------------------------------------------------------------------------
SUM:                          32610        1870564        2172867        9672968
--------------------------------------------------------------------------------

The thing weighs in at over 9 million lines. Start reading...

Since then, the line-count has almost doubled. Here's the latest version as of the publish date of this article.

linux-5.4.6 cloc .
   65658 text files.
   65224 unique files.
   13254 files ignored.

github.com/AlDanial/cloc v 1.70  T=478.71 s (109.5 files/s, 53144.6 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C                                    27621        2731603        2273195       13904947
C/C++ Header                         19699         527802         950568        4273634
Assembly                              1315          46707         101106         227391
JSON                                   272              1              0         159315
Bourne Shell                           557          12529           9357          50235
make                                  2509           9400          10531          41322
Perl                                    56           5593           4073          27924
Python                                 110           4443           4131          24035
YAML                                   188           3039            880          15426
HTML                                     5            665              0           5508
yacc                                     9            692            355           4627
lex                                      8            326            300           2014
C++                                      8            300             82           1873
Bourne Again Shell                      51            354            296           1748
awk                                     10            140            116           1058
NAnt script                              2            147              0            556
Windows Module Definition                2             15              0            109
m4                                       1             15              1             95
CSS                                      1             27             28             72
XSLT                                     5             13             26             61
vim script                               1              3             12             27
Ruby                                     1              4              0             25
INI                                      1              1              0              6
sed                                      1              2              5              5
---------------------------------------------------------------------------------------
SUM:                                 52433        3343821        3355062       18742013
---------------------------------------------------------------------------------------

In either case, absent a multi-billion dollar business alongside, major changes to this artifact are not happening. The open-source "dorks with laptops changing the world" approach never happened, and it's not about to now. Absent control of hardware production, I don't even think a kernel version can be specified. As soon as the chosen hardware becomes unavailable, one will be forced to either attempt backports of drivers,⁹ or give up and move to a newer kernel version.

Thus I propose a controversial alternative. Instead of targeting an older kernel which may not support available hardware, target the latest available and see what may be sawed off without disrupting needed hardware drivers. The kernel is quite modular, and there may be some hope in reducing the effective line count by several multiples by eradicating architectures, drivers, and other optional items that are unneeded.

And now for a breather...

More choices remain before we have even the vague outline of a bootable, self-hosting, self-editing, viable system. I'll be back tomorrow with the next article. Take care!

1. If you read circularity there, read again. [↩]
2. The stallmans, poetterings, *buntus. [↩]
3. Recall the "fits in head". I intentionally reference and correct it. I don't give a flying fuck what fits in the imagination if it cannot be wielded. [↩]
4. Complete means that all components are wholly included in the set, and not merely referenced. [↩]
5. Though glimmers of a way out appear in the darkness. [↩]
6. I will leave this portion unspecified for now, as sadly all we have is a vague notion that a certain vintage AMD chip might be usable. [↩]
7. Binary-blob firmware, drivers, yes... but don't dare absolve yourself of your inability to fabricate your own hardware! [↩]
8. ...and indeed, which kernel. [↩]
9. Which either sums to a backport of significant parts of the newer kernel version. [↩]

Warmth from the father; water from the sky
Who pays for the sunlight? Not I.

Breath from the mother; her hand on your eyes
Who wakes from a daydream? Not I.

For comfort is fleeting
And strength withers fast
How unlucky, while kneeling
His knees turned to glass

His daughters all wept
And his sons, they imbibed
They poured out his ash
On the urn was inscribed

Offspring of Adam, and Fortuna's son
He wiled his days spending twice what he'd won