Software
A curated collection of personal software that I've written that I think is worth sharing.
Check out all of my public source code here: git.sr.ht/~jakintosh
Philosophy
The hard part of writing software is legibility and maintenance, not being clever or using the latest frameworks. I do my best to: 1) write as little code as possible; 2) import as little code as possible; and 3) rely on mature and tested code and protocols when I do rely on it.
I try to stay focused on the problem or experiment at hand, and err on the side of "this will still work or compile if I try to use or modify it in 10 years".
Coalescent Computer
The "coalescent computer" was a research project that I spent the majority of 2023 working on in a part-time capacity, and I submitted it to several open calls for grant funding and conference talks.
Much more can be found on the project's dedicated website, but the programming language and virutal machine are both real, usable, and—in my opinion—interesting, so I include brief introductions to them here.
Co Rust Research Programming Language (view source) (project site)
October 2022 - December 2023An experimental content-addressable assembly language.
Read more
Co is a content-addressable concatenative programming language that compiles to the COINS virtual instruction set architecture. It is an exploration of content-addressable assembly code, and is usable to write arbitrary programs which can be executed with the Cohost VM. This program is a CLI that interfaces with the assembler and symbol library.
Code Example
example.co
% push-one LIT8 1 ; ( macro: push 1 on stack ) : add-one ~push-one ADD8 ; ( routine: add 1 to byte on top of stack ) |0x0000 #start ( set padding to 0, create 'start anchor' ) ~push-one ( push 1 on the stack ) >add-one ( call 'add-one' routine ) &start JPR16 ( jump to the 'start' anchor )
Usage Example
~ $ co assemble example.co example.rom co assemble ...ok ~ $ co library import example.co wrote 3B: .push-one => izFmaP8IstgCyo1_NYI8IrIaT7EjhTOgqRmnpxt1kus wrote 4B: .add-one => QCtK12e_FjH4E-CsLPpQhzhsHSFYi-F_VTrhII-adsc wrote 82B: => it133yXzcjar-Qqk8yXaUiZEjs-4y0OHReCSf-N33mg set _root => it133yXzcjar-Qqk8yXaUiZEjs-4y0OHReCSf-N33mg ~ $ co library list . nametable '.': -> add-one -> push-one ~ $ co library list .add-one byteco '.add-one': LIT8 1 ADD8
Cohost Rust Virtual Machine (view source) (project site)
October 2022 - December 2023A virtual "coalescent" stack machine.
Read more
Cohost is a virtual machine that implements the COINS virtual instruction set architecture, and also implements an extensible virtual device protocol over unix sockets. In its current state, it only implements a "console" device, which allows for rudimentary text-based I/O. The primary function of Cohost is to execute programs written in the Co programming language.
COINS Rust Library (view source) (project site)
October 2022 - December 2023A lightweight bytecode for content-addressable computing.
Read more
COINS is the shared Rust library that defines the Coalescent Instruction Set, and which acts as the "living documentation" for the spec. The bytecode has instructions 1-byte long, and also reserves 32 of the 256 instructions for Co's assembly semantics, meaning that text source code can be losslessly converted back and forth to valid bytecode. There's not room to dig into why that's cool in the context of the whole system here, but feel free to email me :)
Web Publishing
Most of the problems we try to solve with software are human coordination and communication problems. And most of the time, a simple, clear webpage can do all of the coordination and communication work needed to solve that problem. The hard part is being able to sit down and make the problem small enough to be solvable like this.
Anyhow, because of this, I administer a lot of websites. I have, over time, developed a sharp and bespoke set of tools that I use to write, deploy, and maintain them. They are biased towards using old, reliable technologies: plain text, bog-standard unix programs, bash scripts, and a small number of dependency-less custom programs. Below are those tools.
Collect Rust Scripting Language (view source)
March 2023 - CurrentA "little language" for rendering plain-text.
Read more
Collect is a custom text rendering language that I built for writing websites. Essentially, you define a set of rendering objects and text blocks in plain text, and then collect
processes all the files, builds a render tree, and then executes the tree where some objects explicitly write out to a file (or the console). I use collect
(in tandem with sam
) as a static site generator for all of my websites.
Code Example
Some examples of what these different blocks look like.
@string "footer"
"""
footer:
hr:
a:(href="https://creativecommons.org/licenses/by-nc-sa/4.0/") CC BY-NC-SA 4.0
"""
@switch "next-newline"
@has-obj "iter.next" "\n"
@default ""
@scope "page-renderer"
@export "$(document-slug).html" "$(document)"
@end
SAM Rust Markup Language (view source)
April 2023 - CurrentA more comfortable way to write X(HT)ML by hand.
Read more
"Semantic Authoring Markdown (SAM) is a simplified markup for semantic authoring." The specification for the format was outlined by Mark Baker (detailed at the link just above). Its core insight is that "XML was designed to be human readable, but not human writable". It defines an alternative syntax for XML (and thus, HTML) that is much easier to write. I implemented a (slightly modified) parser/renderer in Rust for my own personal use, and use it to write all of my websites' "HTML" by hand.
Code Example
html:
head:
meta:(charset="utf-8")
link:(rel="stylesheet",type="text/css",href="styles.css")
body:
header: My Website
main: About Sam
section: Paragraphs
This is a paragraph. This is a {link}(a|href="http://jaktiano.com/software.html").
This is another paragraph with {strong, emphasized text}(strong)(em).
section: Lists
1. This is an ordered list (item)
* This is an unordered list (item)
footer:
Webpage built with SAM
Build and Deploy Pipeline Bash (view source)
March 2023 - CurrentMy "build and deploy" pipeline is a handful of short, readable bash scripts.
build.sh
The build script—aliased to wbd
—sources an optional script/build.sh
file for custom pre/post hooks, renders the source/
directory to the site/
directory using collect
and sam
, and also copies everything from static/
directly to site/
without modification.
#!/usr/bin/bash
set -e
# load overrides
function pre { :; }
function post { :; }
if [ -f "./scripts/build.sh" ]; then
source ./scripts/build.sh
fi
# run pre hook function
pre
if [ ! -d source ]; then
echo "No 'source/' directory"
exit 1
fi
collect source .sam
sam .sam site --batch
rm -rf .sam
for file in ./static/*; do
[[ -e $file ]] || continue
cp -r $file site/
done
# run post hook function
post
stage.sh
The staging script—aliased to wst
—also allows for optional pre/post hooks, runs build.sh
, uses rsync
to "stage" (copy) the contents of site/
to the local /var/www/$SITE
folder, and then (as an added bonus) generates a sitemap.xml file. I run nginx
locally on my machine to serve these "staging" versions of the site, which is exactly what I use on my webserver(s). This makes local testing as close to the real thing as possible.
#!/usr/bin/bash
set -e
# load overrides
function pre { :; }
function post { :; }
if [ -f "./scripts/stage.sh" ]; then
source ./scripts/stage.sh
fi
# strip PWD for site name
SITE=${PWD##*/}
# run pre hook function
pre
# valid site base has at least one '.'
if [[ $SITE != *"."* ]]; then
echo "Invalid site root: "$SITE""
exit 1
fi
# build
$DOTFILES_PATH/plugins/web/build.sh
# sync to local webserver
rsync -rlpcgoiqP
--delete-during
--exclude '.gitignore'
--exclude 'sitemap.xml'
site/ /var/www/$SITE/
$DOTFILES_PATH/plugins/web/gen-sitemap.sh http://$SITE /var/www/$SITE > /var/www/$SITE/sitemap.xml
# run post hook function
post
deploy.sh
Deploy—aliased to wdp
—is almost exactly the same as stage.sh
... as it should be. The only difference is that it uses slightly different rsync
flags, and expects $WEBUSER
to be set so that it can actually connect to the real webserver via SSH. Likewise, it generates the sitemap.xml
by running the gen script on the actual server, so that change timestamps are accurate.
#!/usr/bin/bash
set -e
# load overrides
function pre { :; }
function post { :; }
if [ -f "./scripts/deploy.sh" ]; then
source ./scripts/deploy.sh
fi
# strip PWD for site name
SITE=${PWD##*/}
PROTO=${1:-"http"}
# run pre hook function
pre
# valid site base has at least one '.'
if [[ $SITE != *"."* ]]; then
echo "Invalid site root: "$SITE""
exit 1
fi
# build the site
$DOTFILES_PATH/plugins/web/build.sh
# sync site to server
rsync -rlpcgovziP
--delete-during
--exclude 'assets/'
--exclude '.gitignore'
--exclude 'sitemap.xml'
site/ $WEBUSER@$SITE:~/www/$SITE/
# generate and upload the sitemap.xml
ssh $WEBUSER@$SITE "bash -s" < $DOTFILES_PATH/plugins/web/gen-sitemap.sh $PROTO://$SITE www/$SITE > site/sitemap.xml
scp site/sitemap.xml $WEBUSER@$SITE:www/$SITE/
#run post hook function
post
gen-sitemap.sh
To be honest, this one is really fun. It find
s in a given directory for .html
files, and echo
s a sitemap.xml
to stdout
based on what it finds in 13 lines of bash
, using only echo
, find
, stat
, and date
.
Combined with rsync
's delta copying that preserves timestamps on files that haven't changed on the deployment side, I get a sitemap.xml
that catalogs an entire site with real lastmod
values, and with absolute minimal infrastructure.
#!/usr/bin/bash
base=${1:?"Base url is required"} || exit 1
root=${2:-.}
echo -e "<?xml version="1.0" encoding="UTF-8"?>"
echo -e "<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">"
for file in `find $root -print | grep .html`; do
[ -e "$file" ] || continue # handle 'empty list' case
page=${file#"$root"} # strip directory root from file
seconds=$(stat --printf='%y' $file) # get timestamp of file
lastmod=$(date -d "$seconds" +"%FT%R:%S%:z") # convert to ISO-8601 date
echo -e "\t<url>\n\t\t<loc>$base$page</loc>\n\t\t<lastmod>$lastmod</lastmod>\n\t</url>"
done
echo -e "</urlset>"
init-local-site.sh
To streamline local development (especially across multiple machines), I also have a quick script to set up an NGINX config. It creates a directory in /var/www/
with the proper permissions, and then spits out a template using the site name and the chosen test $PORT
into /etc/nginx/conf.d/
.
#!/bin/zsh
set -e
DIR="${1:?"Site name required"}" || exit 1
PORT="${2:?"Port number required"}" || exit 1
sudo mkdir -p /var/www/$DIR
sudo chown `whoami` /var/www/$DIR
sudo chgrp `whoami` /var/www/$DIR
sudo mkdir -p /etc/nginx/conf.d/
cat <<EOF | sudo tee /etc/nginx/conf.d/$DIR.conf
server {
listen $PORT;
listen [::]:$PORT;
server_name localhost;
root /var/www/$DIR/;
location / {
try_files \$uri \$uri/ =404;
}
location /var {
deny all;
return 404;
}
location /bin {
deny all;
return 404;
}
}
EOF
Tools and Experiments
The knick-knacks and curiosities of the group.
Hyperstruct Python HTML Library (view source)
July 2024 - CurrentDefine and extract structs from HTML with minimal additional markup.
Read more
As an offshoot of the "multiplicity of data perspectives" work I was exploring with the coalescent computer, I spent a lot of time thinking about and researching semantic data on the web. Microformats, Microdata, RDFa, JSON-LD... there had been so many ways to embed structured data in HTML, and yet all of them felt "separate", and sort of wrong.
Hyperstruct is an experiment in taking the best of these ideas and making it feel more natural as part of HTML. Essentially, you can define "shapes" of structured data using CSS selector queries, and then set anchor points in your HTML using schema links. It ends up being very minimal work on HTML authors, and very extensible by using fundamental web components like hyperlinks and CSS selectors.
Example
example.com/index.html
----------------------
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
</head>
<body>
<h1>My Page</h1>
<main>
<article st-type="http://schema.example.com/journal">
<h2>My Journal Entry</h2>
<time datetime="2024-07-24">July 24th, 2024</time>
<p>
Journal entry body text.
</p>
<p>
More body text.
</p>
</article>
</main>
</body>
</html>
schema.example.com
------------------
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
</head>
<body>
<article class="type" id="journal">
<h2>journal</h2>
<section class="field">
<h3>title</h3>
<span class="content">text</span>,
<span class="count">1</span>,
<span class="optional">optional</span>
<h4>selectors:</h4>
<ol>
<li>*[st-prop="title"]</li>
<li>*[st-type="journal"]>h2</li>
</ol>
</section>
<section class="field">
<h3>date</h3>
<span class="content">datetime</span>,
<span class="optional">optional</span>
<h4>selectors:</h4>
<ol>
<li>*[st-prop="date"]</li>
<li>*[st-type="journal"]>time</li>
</ol>
</section>
<section class="field">
<h3>body</h3>
<span class="content">html</span>
<h4>selectors:</h4>
<ol>
<li>*[st-prop="body"]</li>
<li>*[st-type="journal"]>p</li>
</ol>
</section>
</article>
</body>
</html>
bash prompt
-----------
~
$ python extract.py http://schema.example.com | tee defs.json
{
"journal": {
"name": "journal",
"url": "defs.html#journal",
"fields": [
{
"name": "title",
"content": "text",
"count": 1,
"optional": true,
"selectors": [
"*[st-prop="title"]",
"*[st-type="journal"]>h2"
]
},
{
"name": "date",
"content": "datetime",
"count": "all",
"optional": true,
"selectors": [
"*[st-prop="date"]",
"*[st-type="journal"]>time"
]
},
{
"name": "body",
"content": "html",
"count": "all",
"optional": false,
"selectors": [
"*[st-prop="body"]",
"*[st-type="journal"]>p"
]
}
]
}
}
~
$ python parse.py http://example.com/index.html defs.json
{
"type": "journal",
"properties": {
"title": "My Journal Entry",
"date": "2024-07-24",
"body": [
"<p>\n Journal entry body text.\n</p>",
"<p>\n More body text.\n</p>"
]
}
}
Radeon Kernel Module Patch Bash Script (view source)
November 2023
A script to specifically patch the radeon
linux kernel module on the 2006 MacBook Pro (2,2).
Read more
In the Summer of 2023, I found what would have been my "dream computer" when I was in high school in a junk shop for $25—a 2006 15" MacBook Pro. It had clearly been sat on and abandoned about a decade earlier, but I couldn't resist. After taking it home and testing it with power, it actually worked and I set out to restore it and make it productive again.
To make a long story short, I replaced and cleaned up all the hardware and got a fully modern Arch Linux system up and running, with the sole exception of the discrete GPU. For EFI atrocities I won't get into here, this particular piece of hardware needed some extra help to get the GPU running under Linux; this script/repository is that extra help.
Essentially, I need to patch the radeon
kernel module to bundle the vbios binary and load it manually at boot. This script automates that patching process. If you have the same computer, and are also trying to run (Arch) Linux on it, it might help you.
Also, I do actually use this laptop to do real productive work in 2024.
patch_radeon.sh
Absolutely do not run this script. Read it. Understand every line. Rewrite it yourself so that you know what is going on. Many parts of the kernel patching process are changing all the time. Use this as a guide. Again: Do not copy/paste and run this script.
#!/usr/bin/bash
KERNEL_NAME="${1:?"Kernel name required"}" || exit 1
if [ $KERNEL_NAME == "linux" ]; then
echo "Kernel name: " $KERNEL_NAME
elif [ $KERNEL_NAME == "linux-lts" ]; then
echo "Kernel name: " $KERNEL_NAME
else
echo "Kernel name must be 'linux' or 'linux-lts'"
exit 1
fi
UNAME=`uname -r`
MAJORVER=`echo $UNAME | cut -d '.' -f 1-1`
KERNELVER=`echo $UNAME | cut -d '-' -f 1-1`
EXTRAVER=`echo $UNAME | cut -d '-' -f 2`
LOCALVER=`echo $UNAME | cut -d '-' -f 2-3`
echo "Patching Radeon Module With:"
echo "Kernel Version: $KERNELVER"
echo "Extra Version: $EXTRAVER"
echo "Local Version: $LOCALVER"
# download linux kernel source
echo "Downloading Source for $KERNELVER"
curl https://cdn.kernel.org/pub/linux/kernel/v${MAJORVER}.x/linux-${KERNELVER}.tar.xz -o linux-${KERNELVER}.tar.xz
echo "Extracting source"
tar xf linux-${KERNELVER}.tar.xz
cd linux-${KERNELVER}
# set up build
echo "Setting up Build"
make mrproper
# set up config
echo "Updating Config with $LOCALVER"
zcat /proc/config.gz > .config
sed -i "s/.*CONFIG_LOCALVERSION=.*/CONFIG_LOCALVERSION=\"-${LOCALVER}\"/g" .config
sed -i "s/.*CONFIG_LOCALVERSION_AUTO=.*/# CONFIG_LOCALVERSION is not set/g" .config
make olddefconfig
# copy over the module symbol table
cp /usr/src/${KERNEL_NAME}/Module.symvers ./Module.symvers
# apply patch
echo "Patching Radeon Module"
patch -i ../radeon_bios.patch drivers/gpu/drm/radeon/radeon_bios.c
# prepare
echo "Preparing Build"
make scripts prepare modules_prepare
echo "Running Build with $EXTRAVER"
make EXTRAVERSION=${EXTRAVER} -C . M=drivers/gpu/drm/radeon
# replace old module
echo "Replacing Module"
zstd drivers/gpu/drm/radeon/radeon.ko
sudo rm -f /lib/modules/$UNAME/kernel/drivers/gpu/drm/radeon/radeon.ko.zst
sudo mv drivers/gpu/drm/radeon/radeon.ko.zst /lib/modules/$UNAME/kernel/drivers/gpu/drm/radeon/radeon.ko.zst
# build modules.dep and rebuild ramdisk
echo "Rebuilding Ramdisk"
sudo depmod -a
sudo mkinitcpio -P
# load module
echo "Loading Module"
sudo modprobe radeon modeset=1 backlight=1
# delete source
cd ../
rm -rf linux-${KERNELVER}
rm linux-${KERNELVER}.tar.xz
radeon_bios.patch
radeon_bios.c
changes occasionally, and the most recently updated .patch file will live in the repository linked in the header. Here's what it looks like in 2024.
34a35,36
> #include <linux/firmware.h>
>
139a142,176
> static bool radeon_read_bios_from_firmware(struct radeon_device *rdev)
> {
> const uint8_t __iomem *bios;
> resource_size_t size;
> const struct firmware *fw = NULL;
>
> request_firmware(&fw, "radeon/vbios.bin", rdev->dev);
> if (!fw) {
> DRM_ERROR("No bios\n");
> return false;
> }
> size = fw->size;
> bios = fw->data;
>
> if (!bios) {
> DRM_ERROR("No bios\n");
> return false;
> }
>
> if (size == 0 || bios[0] != 0x55 || bios[1] != 0xaa) {
> DRM_ERROR("wrong sig\n");
> release_firmware(fw);
> return false;
> }
> rdev->bios = kmalloc(size, GFP_KERNEL);
> if (rdev->bios == NULL) {
> DRM_ERROR("alloc fail\n");
> release_firmware(fw);
> return false;
> }
> memcpy(rdev->bios, bios, size);
> release_firmware(fw);
> return true;
> }
>
679a717,719
> if (!r)
> r = radeon_read_bios_from_firmware(rdev);
>
Command Rust Library (view source)
June 2023 - CurrentA lightweight command line argument parser for Rust programs.
Read more
The "gold standard" arg parser in the Rust ecosystem—clap
—has too many dependencies, and adds a ton of bloat to program size. I wrote this to see if I could get all the features I cared about without the bloat, and the answer was "yes". In 420 lines of Rust, I have a single-file arg parsing library that handles subcommands, operands, options (both -s(hort) and --long), and auto-generated --help and --version functionality. It is because of dependency minization efforts like this that the Rust programs I write compile near instantly.
command
is featured prominently in large CLI programs like co and cohost
Usage example
~
$ co --help
Usage: co <subcommand>
About: the assembly language for the coalescent computer
Subcommands:
assemble.....assembles co source code into ROM files
library......manage the symbols and namespaces of your local co symbol library
build........builds a co project
~
$ co library --help
Usage: co library <subcommand>
About: manage the symbols and namespaces of your local co symbol library
Subcommands:
import.......import symbols from a co source file into the symbol library
resolve......resolve the imports in a co source file to hashes in the symbol library
list.........list all names at a given namespace inside the symbol library
clear........clear the given name or namespace in the symbol library
clean........remove all symbols in the library that are not referenced by a name
~
$ co library import
ERROR: Missing operands 'source'
Usage: co library import [<options>] <source>
About: import symbols from a co source file into the symbol library
Options:
-n, --name............the target name in the library ( .example.name )
Operands:
source.................path to the target co source code file
Command Hare Library (view source)
November 2023A lightweight command line argument parser for Hare programs.
Read more
I spent some time learning the Hare programming language, and wanted better arg parsing than it had built in. I ported my Rust library to Hare, and made it quite a bit nicer along the way.
Usage example
~
$ books --help
books (v1) by @jakintosh
balance books
Usage: books [subcommand] | [options...]
Subcommands:
entry create a new entry
account manage accounts
Options:
-h, --help prints this help message
~
$ books entry --help
books-entry (v1) by @jakintosh
create a new entry
Usage: books-entry [options...]
Options:
-h, --help prints this help message