SDBForth

I created a forth environment for Linux on modern CPUs, with support for dynamic library loading, and made a graphical application as a demo.

Try it out

Download the latest release or check it out from GitLab.

Background

Forth is a computer programming language originally developed by Chuck Moore. It was commonly used in embedded systems, but today is mostly used by hobbyists.

I took an existing implementation of Forth, known as Jonesforth, and modified it to work with Linux running on AMD64/x86-64 CPUs.

I also added dynamic library loading, and created an application using SDL2 to show a greeting message.

Motivation

I've been interested in the implementation of computer programming languages for some time. Forth appealed to me because of it's simple core, yet highly complex behaviour.

For example, unlike most programming languages the Forth interpreter is routinely modified as it is running, making it possible (and actually quite painless) to create self-modifying applications.

It also has a few other tricks, such as compiling subroutines just-in-time, and allow functions to be composed simply by writing their names next to each other (eg. applying function A to the output of function B is just B A).

Of course there are some drawbacks:

  • I found the syntax confusing at first, as to me naturally come does not reverse-poslish notation.
  • Jonesforth cannot link to other software libraries, and wouldn't work properly with them if it could.
  • As it has fallen out of favour for professional use, there are few online resources for Forth compared with eg. Javascript or C# (although I found the resources I came across to generally be of high quality).

Goal

There were 3 goals of this project:

  1. Implement Forth (or at least the critical parts thereof) in assembly language for AMD64/x86-64 CPUs.
  2. Allow this implementation to interface with existing programs and libraries.
  3. Create a new program with it, to demonstrate that it works.

What I did

I downloaded a copy of Jonesforth, which is a small Forth implementation for x86 CPUs, written in assembly language. I translated the instructions into 64-bit variants, and made other necessary adjustments (eg. the mechanism to make system calls is different). This left me with a working implementation for 64-bit CPUs, that could read and write text files by making operating system calls directly, now called SDBForth.

Unfortunately Jonesforth does not play nicely with other software. I had to replace the early stages of the interpreter's start sequence with a main method written in C, to allow the system's C library to work correctly. The C library is required by most other libraries and programs, so it is absolutely essential that it is working properly to be able to link to other, more interesting, libraries.

I also had to change how the memory allocation worked. Jonesforth will usually allocate memory by directly using the brk system call. However, it is not aware of what the memory allocation routines in the C library are doing, and would quickly corrupt the heap if allowed to do so. Therefore it was necessary to remove any dynamic allocation from the core parts of Jonesforth, and use the C library's malloc routine instead.

I now needed a way to load other libraries, and routines from them. I added a pair of words to Forth to allow this: SYM to load the address of a symbol (routine or variable), and CALL to call a routine from a library. These are enough to load other libraries, by using SYM and CALL to call the dlopen routine to load a library, and then using SYM to look-up the required routine in that library (and then later using CALL to call it).

I now had everything required to use the SDL library directly from my Forth environment. I decided that creating a window with a greeting message would be sufficient as an example, so I made wrappers for all the routines this required. Then I wrote a program that I thought would show a greeting message in white text on a red background, but instead showed a red background.

There turned out to be a simple reason for this: I was not passing information about colours or rectangles to SDL correctly. To interoperate with other libraries and call their routines, parameters must be laid out in a particular way in CPU registers and memory. The problem in this case was that I had forgotten that fields in structures sometimes need to combined before calling routines. In this case, the text colour was being passed in a structure containing 4 8-bit values (for red, green, blue, and transparency) which should have been merged into a single register for the call. Because of this mistake, the text colour was being taken as fully-transparent red, which of course was not visible. I made a routine that packed the colour properly before rendering the text, and the program now happily displays a greeting.

What I found out

Forth can be an incredibly compact language. In just under 40KB I've got a full general-purpose programming language, with support for loading external libraries, and making graphical applications.

Forth (well Jonesforth and my implementation, at least) is unforgiving; most mistakes, from typos to logic errors, result in a segfault (invalid memory access) sooner or later.

Debugging Forth programms with standard tools doesn't work very well. Forth has no notion of a call stack (it doesn't need one), which confuses many of the normal tools (eg. gdb, valgrind). This can make tracing problems difficult, as entries in the call stack are usually the first place to search in cases of invalid memory accesses (which as previously mentioned, must bugs are). More than once I spent a lot of time stepping through a program just to discover the source of a crash was a typo.

Future directions

I intend to continue working on my Forth environment, and will try to use it for future projects, at least occasionally.

External Links

Copyright 2024