Hey, so for the last month I've been playing around a lot with Rust and writing various backends with it, trying out various architectures and crates and see which combination I enjoy the most. Still really content with Axum and the overall choice of Rust, don't quite know why but writing Rust is just so satisfying, you do a huge change and then just fix the compiler errors / clippy errors one by one and for the most part if it compiles things actually work (unlike other languages/runtimes).

Now, one of the bigger projects I've started is a new kind of git forge called RubHub and since I wanted to start dogfooding it pretty much immediately I had to figure out how to deploy this thing.

First up I just went with Docker, mainly since a Dockerfile makes it rather simple for other people to run you backend software meaning I'd have to write one anyways, might as well use it myself. So after finishing that I just setup a super simple compose file to actually run that thing on some VPS. And while it does work I wanted to support multiple replicas from the beginning, not because it's necessary but because it should be quite simple as long as your state lives somewhere else. And while it did work there were a couple of things I noticed that I quite disliked.

Using Docker Compose in production

So one of the problems that I encountered quite early on was that I had to use network-mode: host because I am using SO_REUSEPORT to have multiple processes listen on the same socket and have the Linux kernel load-balance between them. I could have also done this in the nginx reverse-proxy that does TLS but I didn't like that I'd have to specify in the nginx config every replica, kind of liked the simplicity of just running multiple processes and nginx not having to know how many replicas there are right now.

In general this works quite alright, of course that behaviour only works on Linux which is fine since for now that's the OS I'm using for serving the "production" site.

Apart from that one thing that bothered me is that the implementation in podman doesn't seem to support multiple replicas, additionally from what I can tell compose doesn't really check if a replica went down or restart them which kind of seems to be the main benefit of having them in the first place.

Additionally one thing that irked me was that the Rust backend only used 6-10 MB of memory, but then docker added another 123 MB on top of that, not sure about you but doesn't sit right with me if a "slim" container runtime uses 10x the amount of memory than the actual service.

systemd

So while Docker does work I wanted an alternative that was a bit leaner, funnily enough I landed on systemd. It's already managing the VPS anyways so might as well use it for managing this service as well. So first I just wrote a very simple unit file running a single replica, easy enough and worked quite well, multiple replicas were also quite simple to add by just using a templated service, so now I could just do systemctl enable rubhub@1 --now && systemctl enable rubhub@2 --now and have 2 replicas up and running. Rather nice, another nice benefit of using systemd is that I don't have separate system/application logs, they both get put into journald and then transmitted to my Grafana dashboard. So now I was looking into what other things I might try to make things more resilient, and found out about the whole notify system and of course there was already a Rust crate for using it. This now allows my backend to tell systemd that it started up correctly and is ready (with an additional timeout/check that produces an error in the syslog if it failed to start up in a certain amount of time) as well as a watchdog that requires my backend to regularly notify systemd otherwise it gets restarted and an error gets put into the syslog. This seemed rather nice, especially when using a single-threaded async runtime this meant that if for some reason a task got stuck in an infinite loop (or very very long operation) I could detect this. Rather nice since this actually protects the backend against a lot of mistakes I might make, I mean sure I try my best not to write code that could end up blocking the entire runtime but mistakes do happen and it's nice to know that in that case the application server will be restarted and an error appears in the syslog for me to investigate later.

Additionally I found it quite nice to find out that you could also directly give the capabilities to the service to open a privileged port as a normal user, should harden things some more since otherwise I'd have to run things as root and drop privileges, but much simpler to just run as a restricted user from the beginning.

Conclusion

So in general I have really underestimated systemd, there is a lot of hate on the internet about it bloating up systems and everything but I have to say here I quite liked the various features it provides, from what I know there aren't many init systems doing similar things, generally requiring you to write your own supervisor / service manager if you want more sophisticated monitoring/supervision, so quite nice to know that a single 50 line unit file, and about 30 lines of Rust code are sufficient to get you all of those benefits.

Adiós, べン

Alright, so for the past couple of weeks I've been hard at work migrating a Node backend to Rust. Mainly this was due to some architectural problems with the old implementation and since a lot of parts had to be rewritten anyways I thought I might try out another language. Don't get me wrong, Node is fine, but it's also not a language that's super fun to write. Or to be more exact, the overall ecosystem isn't all that fun, while it is productive and generally gets the work done and can be quite fast/efficient, it has been taken over by people that just want to get things done with the least amount of effort and understanding possible. Probably doesn't help that it's become the language that every young developer starts with.

So, since I've been meaning to rewrite this app for a while now I thought this might be a good chance to try out some other languages/runtimes. First I gave Elixir/Phoenix a try, and while it did feel quite productive it didn't quite click with me, afterwards I gave Bun and Elysia a shot which worked quite well but the combination of Bun segfaulting a little too often for my taste in combination with the overall experience not being that different from node led me to give an old favorite another shot: Rust.

Tech Stack

So first off I'll be describing the overall tech stack since this might not apply to other frameworks, I knew from the start that I needed something async (LOTS of mostly idle websockets), which probably meant tokio. Since axum seemed to be one of the bigger framework I though I'd give it a try since the example did look quite nice, and so far axum seems like a fine choice, don't remember a single instance where it made issues for me, just a solid performer overall!

Apart from that I still needed a way to talk to a database, postgresql to be exact, for this I wanted to give sqlx a try, checking queries during compilation sounded somewhat unusual but I've really grown to like it. While it didn't make me that much more productive it generally just works reliably.

Of course we have a couple of standard Rust crates like anyhow, serde and so on, not going to bother going into this more since they are just solid performers overall (as has been my general experience with the overall Crates ecosystem).

Oh and before I forget, of course mold is used as the linker during development, combining this with bacon which automatically recompiles whenever a file changes is a rather smooth development environment, not quite as fast as Vite or Bun but still fast enough that it doesn't slow me down (~500ms on my ratherrrr average Desktop).

The frontend is being bundled via Bun, I have a script that runs the Bun bundler in watch mode for the various frontend bundles the app requires, Rust then either serves them directly when in development, or uses include_bytes to embed the HTML bundle into the Rust executable (still have to figure out how to add the assets themselves, but so far things work quite well).

The Ecosystem

Rust has quite the extensive ecosystem for doing backend webdev by now, so in general I didn't feel any less productive due to missing libraries (which can be a problem with languages like Scheme for example). Additionally I'm always pleasantly surprised by the overall quality of the crates ecosystem, quite unlike the JS ecosystem where I generally have to try out at least 2-3 libraries before finding one that actually works good enough and is actively maintained.

Productivity

Funnily enough I didn't feel any slower writing Rust, in general it didn't seem to make much of a difference as compared to languages with a GC. I kind of feel as though Rust is greatly suited for doing API backends, since the main point that slows one down in writing Rust is when having to specify complicated lifetimes, which doesn't happen in API backend for the most part. Most state is pushed to the database anyways, making each request independent of each other, because of this one pretty much never has to share state between requests which means no complicated lifetimes! This makes writing Rust not that different from writing JS/TS with the added benefit of the compiler actively nudging one into writing a stateless app server.

Additionally I was really missing proper sum types / enums in TS, there are just so many things that are much easier to express via sum types and pattern matching that I might even say Rust has made me more productive. One thing that has been bothering me a little though is duplicate types, since it's just an API backend the frontend is doing most of the rendering and we need to specify the types somehow, so far I've just written zod schemas myself for the couple of types that are relevant to the frontend, but I've read that there are some crates that can automatically generate types (maybe even schemas?) straight from Rust types. Will give those a try in the future but so far I wanted to keep things somewhat simple, especially when it comes to additional build steps I'm a little hesitant.

Also Rust having a sound type system builtin makes the overall experience so much nicer, TypeScript is sadly only a veneer on top of raw JS and therefore inherits a lot of dynamic behaviour from that, which sometimes is nice but generally something I'd like to avoid for software that needs to be reliable.

Performance

Still haven't deployed the rewritten backend into production yet, so far things seem quite snappy, although performance wasn't an issue with the old backend the memory consumption alone is much better and the CPU load for that process actually going to 0% when idle, unlike Node.js which always seems to do something (might just be Express.js or one of the various packages though).

Final verdict

I'm really suprised at how easy it was to use Rust for this project, would have thought that it'd be a little harder but ended up being rather simple. Honestly now I'm thinking of doing most projects, even the small prototypes directly in Rust since I feel just as productive and the result is much more stable/performant and to top it all off: Rust is more fun to write than JS/TS. Especially TypeScript isn't quite as fun to write as it was a couple of years ago, I think the ecosystem embracing TS that much has led to some issues since now everything needs to be completely typed, which is a commendable goal but does end up with absolutely arcane type definitions that regularly break my LSP and otherwise slowing my machine down to a crawl (especially tRPC while nice, does make my machine feel quite slow).

Now I just need to find a nicer alternative to React for the frontend, too bad that most alternatives don't have mature component libraries since to me that's the main benefit of using a framework. Will give SolidJS a try in another project soon though, since it keeps coming up and the examples I've seen do look quite enticing.

Adiós, べン

In the last week I've given both GuixSD and NixOS a try, installing GuixSD on my desktop computer and NixOS on my laptop. In the beginning, things were alright, but as soon as I wanted to do more than just browse the web on my machines, things started to fall apart.

The problems I encountered were mostly the same, though GuixSD sadly had a few more issues. Pretty soon I came to the conclusion that doing development on these machines is actually harder than on a normal Linux distribution.

The Nonguix Problem

First off, the whole nonguix issue. While I do have a certain amount of respect for the GNU project, the way they handle nonguix and proprietary drivers/firmware just seems so stupid. I think NixOS does it right here—during installation you get a checkbox asking whether you want nonfree firmware/software or not. That's it.

GuixSD could do something similar since they already check whether the machine you're installing it on requires nonfree drivers. Instead of just throwing an error, they should allow you to use linux mainline right there in the installer.

The installation of GuixSD was quite complicated. First I tried to follow some blog post on how to use a nonguix installation image, but that resulted in an unbootable system. Then I tried the CLI approach and gave up after guix pull still wasn't done after 2 hours.

The way I finally got it working was by installing GuixSD with linux-libre (since the desktop I was installing it on only needed nonfree drivers for WiFi and the GPU) and then switching to linux mainline for GPU drivers.

In general, I really dislike the overall vibes I get from the linux-libre fanatics. Why does it matter whether the nonfree firmware is loaded by the driver or shipped on the device itself? From what I understand, most of the supported hardware just ships the proprietary blobs on the hardware itself.

So it sounds like it's less about actually using/supporting free hardware and more about using devices that give one the illusion of freedom. I'd much rather see more effort being spent on developing actual free hardware alternatives and promoting them.

Performance Issues

This is another issue I thought I might accept but just couldn't. Doing a guix pull is so atrociously slow that I can't accept it. We have blazingly fast machines these days, and with the right software they just fly.

For example, on the same machine I could install a basic Arch Linux system in about 5 minutes, then install packages for about 15 minutes, and restore my home directory from a backup on my NAS (which takes about an hour—could speed it up by using Ethernet or cleaning up my backups). But after that's finished, I have a system that works great.

I suppose with Nix/Guix the same should be even simpler since just loading my custom config should result in a fully working system. But arriving there seems to take quite some time. In general this isn't much of an issue for me since most interesting stuff lives in my home directory, and by now it takes me about 5 minutes to adjust the config files in my /etc/.

Documentation Gaps

While Guix does have quite a nice manual, sadly it just wasn't enough. I regularly ran into issues and had to search the web to find somebody who had accomplished this somehow. Most of the time I just had to browse Guix configs from random people on GitHub and figure out what was possible and how by reading the source.

This was also a problem with NixOS, where there was even less documentation, but that was made up for by much more help on the web. There I could generally figure out most things by finding old posts on Reddit or Stack Overflow.

But while doing this, I started questioning the overall architecture. The general idea seems to be to provide an abstraction layer over the upstream software and then build the configuration out of Nix expressions/Scheme code. If that's the case, then each package maintainer is also maintaining a public API, which sounds like a lot of responsibility and should mean lots of high-quality documentation so people can figure out how to use these packages/modules.

Here I was quite disappointed that, at least out of the box, there wasn't a great LSP-like editing setup for these system configuration files. It would be fantastic if I could get autocomplete for all packages with inline documentation on possible values and what they do—like you get for pretty much any modern language. Even with a language like C, you get header files you can read to figure out how to use some library.

Where Are the Promised Benefits?

One feature I was initially quite excited about was per-project dev environments. It sounded like a great idea, especially when combined with direnv. And on the terminal it actually works great—I could just cd into some directory and suddenly have a full C toolchain. Fantastic.

However, the problem is that this doesn't seem to play well with editors. Since I hadn't installed these globally, I had issues actually getting the required header files for the LSP, or the LSP in general. Additionally, getting tree-sitter grammars for Emacs just did not work at all. Emacs could not download the pre-packaged ones or compile them, and installing the grammars globally seemed to do nothing.

VS Code wasn't much better and also struggled with finding LSPs, ending with me just installing everything globally. Maybe this could be worked around by adding the editor to the environment, but I don't want multiple versions of Emacs/VS Code running all the time, and I do switch quite often between projects.

Additionally, I've noticed that for the projects I maintain/contribute to, I don't really need per-project environments since they all work with the latest versions of GCC/Rust/Node/SBCL.

It also seemed like quite often there wasn't any good abstraction at all, so I still needed to configure most system packages separately—only this time in Nix/Scheme instead of by editing a config file. To me this seems like quite a missed opportunity, since it'd be great if I could specify what behavior I'd like and then, depending on the backend/other installed packages, Nix/Guix would figure things out for me.

With a nice editor on top, this could be quite interesting. But the problem is that I already know how to set up all the system packages I care about, so now I have to learn a new syntax to gain what exactly?

The Flakes Confusion

This is something I still haven't quite figured out. A lot of people seem really excited about flakes and use them for pretty much everything. But it's still an experimental feature, and some people advise against using it. To me as a newcomer, this is quite confusing—it seems like there are 2 ways to set up Nix but no clear description of when to use what.

Internet Dependency Issues

While I don't travel quite as much these days, this has bitten me in the past when trying out GuixSD. You need to change a small part of your system config and suddenly it pulls in all these packages over the web, but you only have a cheap eSIM with limited traffic.

This is never a problem with normal distros—you can change your config as much as you want without any internet connectivity whatsoever. You can then plan ahead a little, and when you're at an Airbnb/hotel/cafe with nice WiFi, you update all packages.

Conclusion

I still like the overall idea, but so far I've only seen it slowing me down in my daily work. Of course, this might be less of an issue once you get more familiar with the system, but for that to happen, good documentation helps a lot. Additionally, I need some things to work MUCH better out of the box to consider such a change.

It does seem like a nice way to manage cloud instances, though.

Adiós, べン

GuixSD

GuixSD is a Linux distribution that lets you define your entire operating system configuration using Guile Scheme. As a long-time fan of Lisp, this greatly appeals to me, so I’ve chosen GuixSD for my main desktop machine.

Since it’s a GNU project, the documentation is excellent—I really appreciate how much effort most GNU projects put into their manuals. That said, there’s still quite a steep learning curve. You need to wrap your head around a lot of unfamiliar concepts if you want more than just a very boring desktop experience. It also more or less requires learning Scheme, though for me that’s actually a plus since I haven’t been writing much Scheme lately.

The installation process was… complicated. Officially, only linux-libre is supported. At first I tried installing with a nonguix installation medium, but failed miserably. The TUI installer worked fine, but the resulting system wouldn’t boot. I then tried the CLI approach but gave up after a guix pull spent half an hour crawling to just 3%.

Luckily, my desktop only needs proprietary firmware for the GPU, so I ended up installing standard GuixSD with the linux-libre kernel, then added the nonguix channel to get a standard Linux kernel so I could actually use my GPU. I hadn’t realized AMD GPUs aren’t supported in linux-libre—I always thought AMD was preferable to NVIDIA on Linux. But from what I’ve gathered, only older Intel iGPUs are properly supported in linux-libre.

While that wasn’t too bad, I did forget to set up the nonguix substitute servers. This meant I ended up compiling my own Linux kernel, which took a while but was surprisingly painless. (I really like how these systems can transparently compile from source—fantastic feature!)

Now I’ve set up my usual i3 environment, with IceCat instead of Chromium. It’s been a while since I used Firefox as my daily driver, and I’m not sure whether it’s Guix or the IceCat patches, but it actually feels snappy—I really like it.

I’ve also been playing around with direnv while doing some C coding on Nujel. It’s very convenient to have environments change automatically as you cd around the filesystem. This will be especially nice when experimenting with various languages where I might not have full toolchains installed on every machine I use.

Package availability isn’t as broad as on Nix or Arch, which I expected. But that just makes it more fun to create Guix packages for the small programs I’ve grown attached to over the years.

To my surprise, the Emacs experience wasn’t all that great. It works as expected in general, but using distinct dev environments causes issues since the required LSPs may not be available in the global Emacs environment.

So far, I’ve just added node and clang to my user profile so Emacs can use npm for a TypeScript LSP and clangd for C/C++. There are direnv packages for Emacs too, though I haven’t quite figured out how to use them properly.

Another unsolved problem is tree-sitter. Even after adding the grammars to my user profile, Emacs can’t find the .so files. I’m considering two possible fixes:

Option 1: Create my own Emacs package

Might be worth it, since I spend so much time in Emacs. I could compile the tree-sitter grammars as part of the package and maybe also include the Emacs packages I use.

Option 2: Locate the .so files manually in Elisp

I’d rather not, since it feels hacky. But tree-sitter lets you specify directories for grammars. I could query the system to figure out where Guix stores them and set the path. If multiple directories aren’t supported, I might have to create a directory of symlinks to all the grammars. It’d probably work, but it doesn’t feel clean. Still, it’s a fallback if building my own Emacs package turns out too complicated.

Conclusions

I’m excited to dive deeper into Guix itself. I want to figure out why NixOS updates are so much faster than Guix—surely Guile should be a much faster runtime than Nix? Maybe there’s room for optimization.

NixOS on the horizon

On my laptop, I’m giving NixOS a try. While I’m not a huge fan of the language, the ecosystem feels much bigger, and it helps that they have no issue supporting non-free firmware (pretty much required for most notebooks). I also like how updates are noticeably faster than on Guix.

So far, I haven’t spent much time customizing the system, so I’ll probably write more about my NixOS experience in a later article.

Adiós, べン

Introduction

I've been using Linux and BSD Operating Systems for about 20 years now, starting with Fedora Core 5, switching to Ubuntu after a while, then moving to Arch Linux and mostly staying there while using Gentoo, Alpine, NixOS, GuixSD, FreeBSD and NetBSD on and off.

In general, I've noticed a change in the overall community which seems less enthusiastic these days. Back then a new release was exciting, and podcasts like the Linux Action Show exemplified a certain passion and enthusiasm that somehow got lost over the years. Additionally, it seems that a lot of newer Linux components have mostly angered and fractured the community, with many distributions defining themselves by NOT using certain components (glibc and especially systemd, for example).

Pulseaudio

This was probably the first one that saw widespread hate, with people actively advocating against its adoption. Especially as users got forced into using it over time because various applications dropped support for ALSA, leaving people to either run pulseaudio or find alternatives for those apps. (I do remember getting quite upset at Firefox dropping ALSA support, since I generally much preferred ALSA/JACK and had a lot of issues initially with pulseaudio.)

Unity

Another forced replacement that angered a lot of people (or maybe they were just very loud). I know that I was affected by it and switched from Gnome 2 to ratpoison, which over time led me to Arch Linux. Most of the issues I had with my Ubuntu installation got fixed because I looked things up in the Arch Wiki, which got me wondering why I wasn't using that distro since their wiki seemed to have all the answers. I absolutely loved my setup back then when they still had their own BSD-inspired init system.

Gnome 3

Gnome 3 was also met with a lot of resistance. It did feel like it overly simplified things to a fault, trying their hardest to hop on the tablet bandwagon (the iPad was released just a bit earlier—coincidence? I think not!). It was rather bad in the beginning from what I remember, though in the long run they kind of got their act together and made it work somehow for desktop users. I know a friend of mine was running Gnome 3 for quite a while, and I never really could get used to it. It just felt so dumbed down and generally not like anything that made efficient use of screen real estate. Also, though this is subjective, it never really looked all that pleasing, especially if you compare it to modern KDE versions which do look gorgeous (though I prefer something more minimal, I can respect KDE).

It's also interesting that this caused a fork of the Gnome 2 codebase named MATE—the first time I've seen a fork of an old version gain some sort of traction. But that might just be because Gnome 2 was actually really good and, with the right theme, also looked quite good for its time.

Btrfs

Back in the day this was hyped quite a bit—finally replacing the aging ext4 default with a proper modern CoW filesystem. I did try it out a couple of times and it was the only time I've had data loss due to a filesystem bug. Never again gave it a shot. Also, from what I can tell, it's been in development for quite a while and hasn't gained much traction, with most distros still using ext4. Generally it seems like xfs/zfs have gained more adoption.

systemd

Probably the most controversial change in Linux, with a lot of outrage from the community and many forked distros that just remove systemd and replace it with something else. I was also quite against it in the beginning, having a lot of issues when Arch switched to it. Though over time it has mostly stabilized and works quite well now. Also, having some sort of standard way for defining/running services seems like a good idea.

Wayland

Another replacement for some older *nix technology that's mostly Linux-only and which a lot of people actively avoid. This time it's also quite justified since, for the most part, basic functionality barely works after over a decade of development. I mean, just try to record your screen on Wayland—depending on your compositor it might work or not. In general, the issue here seems to be that working old technology gets replaced by some hyped piece of technology that doesn't really improve things that much, while breaking a lot of edge use-cases which angers power-users who require these features.

Docker

While some people really dislike Docker, it has become pretty much the standard way of packaging software for production. Still, it seems to have split the community a bit, and the missing BSD support leaves a bad aftertaste since this makes using FreeBSD much harder for servers. Before, it didn't really matter that much since to user space programs it seemed similar enough to not cause many issues.

io_uring

Another hyped technology that seems to not really have much of an impact. Although in this case there wasn't much outcry over it, probably because it can be completely avoided if one doesn't want to use it. It does promise some big performance improvements which a lot of people would enjoy. From what I could tell, there isn't really much software that uses it. I would have thought that by now software like nginx, apache, and HAProxy would have incorporated it to gain a nice performance increase. Kind of wonder why that is?

Wireguard

Not sure whether this can be considered a Linux innovation. It seems to be widely adopted by now, with most OSes having support for it—probably just because it works much better than OpenVPN while being much simpler to configure. Adding this mainly to end things on a somewhat optimistic note.

Snap

I don't know anybody who actually likes to use snap packages. It's also quite infuriating that quite often snapd is running on Ubuntu servers that aren't using a single snap package, but it's still running, eating up quite a bit of CPU for something that's not being used at all.

Conclusions

I'm wondering what has been happening. A lot of these technologies were met with resistance and sometimes forced on users, leading to a lot of anger. Also, generally it's not really clear whether things improved at all. The Linux machine I'm using today isn't really all that different from what I've been running a decade ago. At times I do try out newer distros and come away quite dissatisfied, though I am quite impressed with KDE, which is just a gorgeous desktop environment. Additionally, Krita, KiCad and Kdenlive have all come from KDE and are absolutely fantastic pieces of software that work great and provide excellent alternatives to proprietary software.

Another observation is that most of these hyped but seriously flawed projects came from either RedHat or Canonical. Maybe this is just the result of Linux development mostly being done by corporate programmers instead of the free software hackers who did most of the early work.

Adiós, べン

Package management in Python

Python seems like one of the few languages where package management seems even more tiresome than in C/C++, and that is considering that those don't really have a widely used package manager (though I suppose apt/pacman somewhat count as on). So for example let's compare what I need to do to compile/run some application I've found on GitHub, with most languages it's a couple of lines of code to build/run the app and you're good to go.

Rust

For example with Rust, a supposedly super complicated language, I generally only have to run cargo run. Setting up a Rust toolchain is also quite simple with rustup, although you can also install it with your distros package manager and things generally work, I think I've only ever had issues with rusty_v8 on FreeBSD, and I think some minor issues on Windows ARM, apart from that things in generally just work.

Node.js

This time a language that a lot of beginners learn, here it's also quite simple in general, for the most part you just run npm ci and you have all dependencies installed, then look at the Readme or package.json to figure out how to start things (might have to start some docker containers in the background for Databases and such). You might have to use pnpm which works similar enough to npm to not be much of a problem, installation is also quite simple.

C

For a language that doesn't have a package manager it's quite simple, on unix-like systems at least, for the most part you can do ./configure && make and then you get an executable to run, the configure script might complain about missing dependencies but these you can quite easily figure out and install with you systems package manager, not that hard.

Python

Now Python. The first problem is that it apparently doesn’t care about backwards compatibility; it’s almost as bad as Lua—most projects need a very specific Python version to run at all. So how do you do that? Arch Linux for example only ships the newest Python in the standard repos, so you need to set up a venv or something to use that exact version. Then you use pip to install dependencies from some .txt file (really? A text file listing a couple of names?). At least you can pin versions. This inevitably fails because one dependency uses native code and there’s no automatic fallback to compile it if no binary is available. But wait—some projects use Anaconda or Miniconda, others use uv. There’s also pipx, which I don’t even know how it differs from pip. Oh, and isn’t there Poetry too? Why can’t this ecosystem just pick ONE standard for describing a package and its dependencies? Node.js nailed this: every project uses the same package.json.

Distributing applications

This is another serious issue. How do you distribute your Python application? I’ve helped a couple of kids get into game dev with Python/pygame, and as soon as they build something cool they want to share it with friends… but how?

Even if we restrict ourselves to Windows PCs, there’s no simple way to bundle the program and all dependencies into a single .exe. Telling a non-developer friend to install Python, pygame, and everything else is a bit much.

These days I just teach them JavaScript/TypeScript. Works way better! The language itself doesn’t seem harder for them, and the huge win is that you can host the game for free on GitHub. One simple CI pipeline deploys everything to GitHub Pages and they get a shareable link. Sure, writing the pipeline is a little fiddly, but you can usually copy-paste a standard file that works for most games.

Native languages are obviously more complicated, but distributing a Rust executable, for example, isn’t that bad. Cargo is standardized enough that plenty of templates exist one can just use.

Conclusions

I actively argue against using Python for anything, except AI/ML, where you’re basically forced into it because all the essential libraries are Python-only. It’s frustrating, because outside that niche I haven’t found a single redeeming quality. It just feels like a slow, messy language that people assume is “good for beginners” and then teach by default.

Adiós, べン

What’s good about tRPC?

It does provide some benefits if you just want to build some backend API quickly. It is rather simple to add a new endpoint and then use it in a React frontend with full type inference. The whole context system is also quite neat and can make nested routes quite simple to implement.

What’s bad about tRPC?

It obfuscates the actual network requests quite a bit, making it hard to debug what/when requests are actually happening. It also makes accessing the same API from languages other than TS/JS much harder than just implementing a REST API. In general, I don’t see much added value from tRPC over a well-engineered REST API which can be consumed by pretty much any reasonable runtime. It also seems to slow down tsc quite a bit, making the whole dev experience more sluggish.

It also makes simple things like uploading a file much more complicated, sometimes having to resort to workarounds like base64-encoding a file to send it to the tRPC endpoint. All of this is much simpler with standard REST.

Additionally—though this is harder to quantify—it seems to slow down the TS LSP… a lot. I’m working on some tRPC codebase where my M2 Mac requires over a minute after restarting the TS Server to show type information. Compare that to some standard TS full-stack apps I’m working on (with dependencies kept to a minimum) where it restarts in about a second or so.

More generally, I get the feeling that this is just another of these projects that make simple things just a little simpler at the cost of making other things much more complicated, and I’m not sure whether this is a good trade-off—especially considering that AI coding tools work really well at writing simple boilerplate code.

Conclusions

Gave tRPC a try for some smaller projects of mine, and while it did work quite well, it didn’t feel like much of a game-changer since adding another endpoint to something like Express.js isn’t particularly complicated, though using these endpoints in a React frontend is quite easy to do.

Adiós, べン

Back from the back burner

While this project has been on the back burner for a couple of months, we've started working on it again. Now that I've settled in at my new day job and had some time to learn more about how to build a small SaaS startup, I wanted to give Bitmenu another go. This time, instead of trying to do B2C directly using SEO and Google Ads, we'll instead focus on just making that sale by directly going to restaurants (starting with ones where we know the owners) and convincing them of how nice it'd be if one could just order directly from their tables.

Smooth ordering, smoother onboarding

To accomplish this, we've been focusing on getting this ordering feature working super smoothly as well as making the setup as simple as possible. The idea is that it should only take a couple of minutes and a café is all set up to take orders using our system. I've also got some hardware so that I can print out the required QR codes directly from my phone:

Got this super cheap printer, great tool since I can now print QR-Codes while at a potential customer

Five-minute setup checklist

Apart from all this I've also added a bunch of features that are necessary in order to make selling to potential customers much nicer for us, since we'll be busy talking to the customer the tech behind everything should be super smooth. So currently in order to setup table ordering for a restaurant we only need to do a couple of things:

• Create a new restaurant
• Take a picture of the menu, a LLM will then analyze and create a structured menu from that
• Type in all the tables numbers / identifiers into a textarea
• Print out all the QR-Codes
• Let an employee scan a special QR-Code so they receive the orders

That's it, should only take a couple of minutes and we're done. Hopefully this will convince some people to go for a paid plan so that we're finally seeing some revenue, since staying motivated is quite hard with zero revenue.

Hackathon tomorrow, live test on Monday

So tomorrow I'll be meeting up with my cofounder to do some last minute hackathon before we're testing it Monday/Wednesday with our first test customer, hope things will work out and that people will be as excited as I am about it.

Adiós, べン

Why I Built This

So, after ranting a bit about how terrible vibe coding is for anything somewhat complex I've researched a bit and it seems that managing context is one of the main problems for coding agents. Additionally there seems to be some indicators that AI agents do a better job while working on microservies. This led me to try it out myself and so I started refactoring the bitmenu codebase a bit to try and decompose and modularize it better, and while not object/scientific it does seem to improve the quality of the changes claude code makes.

To better understand the codebase, especially as it's undergoing rapid changes due to AI agents, I've built myself a tool in the form of a VS Code extension for quickly visualizing what the current architecture is, focusing on quickly showing which modules are referenced throughout the codebase as well as providing some ballpark figures for complexity (lines of code... not great, I know, but it's probably one of the simpler metrics to calculate).

I've also (well Claude Code to be exact, it's 99% vibecoded) a feature that colors nodes/arrows red/green depending on whether they've been added or removed when compared to the last commit. This should hopefully quickly indicate when an AI agent starts importing like crazy increasing overall system complexity. Since it seems like the ideal are many modules that only depend on a handful of other modules.

Also have to say that Claude Code worked really well here, I was pretty much only doing testing (apart from debugging some minor issues in the beginning) and providing overall guidance on what to do, which features to add (and also which to remove). Though I suppose this just underlines that AI agents are remarkable for small projects, but then quickly grind to a halt as the system gathers a complexity that is too big to fit into the limited context window of today's models.

Now, I've only spent about an hour to build this prototype but I'll be testing it out in the coming days while working on some other projects of mine to try and reduce complexity. Might actually end up publishing it to the repo if it ends up being useful to me, since so far it's just an idea.

Adiós, べン

While I am quite dissatisfied with the experience of coding agents like Claude Code for projects that reach a certain size, they are amazing for quickly spinning up a prototype or building small sites and scripts.

This led me to think about how one might go about structuring things to allow AI agents to still work effectively for bigger projects. One problem is that right now there's no real consensus on how to actually architect a good system. Though it probably helps that a lot of people agree on what makes certain systems bad. This, however, might not transfer to what makes a system architecture good for AI agents, so I'm trying to organize my thoughts in this article.

Where AI Agents Hit Their Limits

A general limitation of most AI agents is the limited context window. This is probably the biggest limitation and a big reason why things work great for small-scale projects but then quickly fall apart—the LLM just can't keep track of everything.

Another issue is that, at least in my experience, properly encapsulating things is essential for the assistant to get things right. I've rarely gotten acceptable results for changes that require non-obvious modifications all across the codebase. Just adding a new function/method or adding a new case works pretty well though.

I've also had pretty bad experiences letting the AI write stateful React components. The component itself is not an issue, and they do an excellent job at that. But figuring out how to handle state, how to pass it through the component hierarchy, and how to build things in a way that the AI can later extend properly has rarely worked out. So far I've mostly had the agent build a prototype and then rewritten all the state-handling code myself later. This still helps a little but not as much as I'd like it to.

Compared to that, I have a couple of projects that don't use React at all but render everything server-side using, for example, EJS. This works surprisingly well because the AI has no problem figuring out that it needs new data in the template, so it makes sure to pass it along from the endpoint. No careful restructuring is necessary, and if one keeps prompting the AI to take a look at the endpoints/templates and see whether there's room for simplification, they do an okay job.

What Humans Struggle With (That AI Doesn't)

Another aspect that might be worth pondering for a bit is what kinds of limitations human coders have, especially compared to LLMs. One thing is that LLMs know pretty much all the common languages and frameworks that are in common use—you'll be hard-pressed to find a human that can write reasonable code in 100+ languages.

Another problem is that humans can only work on a single thing at a time, whereas with agents you're generally only limited by how much you're willing to spend on inference. You could just run 100+ agents in parallel producing thousands of lines of code per second, so the question is how one could enable this.

Humans are also quite slow when it comes to learning new abstractions, while an LLM can easily scan and read a codebase and figure out how to use them quickly enough.

Architectural Strategies for AI Development

One way to circumvent the limited context window is to make sure to use old and stable libraries and use as few custom abstractions as possible. The models have vast amounts of code in their training set, so the knowledge on how to use these libraries is part of the weights. Custom abstractions, even if simple and beautiful, still need to be put into the context window. This puts a limit on how many custom abstractions one can have in a codebase before it overwhelms the LLM, leading to terrible results.

However, these abstractions can still be used as long as any change only requires a limited amount of knowledge about this particular codebase. So let's say we have 10 subsystems working together—for human coders it might be better to build 20 abstractions used in every subsystem. For an LLM, it might be better, though, to make sure that any subsystem only uses a couple of abstractions, even if that means that the total number of abstractions is greater.

In general, I think one needs to be careful that any one change only touches at most 2 subsystems at once, since a global change will fill the context window rather quickly. One should be able to do a global change, though, by using proper API versioning: first introducing the new API, and then changing every subsystem in sequence to use the new API. That way we never have a situation where the LLM needs global awareness.

Additionally, static typing seems absolutely necessary since the feedback loop with the type checker catches a lot of errors and greatly improves the quality of the results. Another thing, though this probably requires direct support by the agent, is proper hiding of implementation details. Let's say the LLM needs to use a certain class—as long as we're only using it, we only need to know about the public methods, and for those we should also get by with the type signature and comment describing the proper usage. The method body or private methods should never matter to an outside consumer. However, so far Claude Code, for example, still happily seems to read the entire module when it only needs to know about a certain class. Maybe tighter LSP integration would be key here since it should easily be able to distill essential information for the LLM to use.

Apart from that, I'm thinking that microservices might be a good pattern for LLM usage, since it enforces proper encapsulation and for the most part all functionality of a service can be described by an OpenAPI spec or something similar. Though modules and classes should serve a similar purpose without all the problems that come from distributed systems.

Next Steps

I'll be experimenting a bit with different architecture patterns and languages to try and figure out which approach produces the best results. This would be good to know for future projects and would be quite valuable insight for my own coding agent work.

Adiós, べン

In the last couple of weeks I've been using Claude Code extensively, both at work and for my own side projects. And while in the beginning the results looked impressive, I'm now quite dissatisfied and convinced that the entire approach is wrong as long as LLMs don't make a huge leap forward in capabilities.

While it is impressive that you can give a short description to Claude Code and have it go at it, adding tasks, finishing them and in general just generating lots of code, in my experience it completely fails to do anything useful as soon as the project you're working on becomes slightly more complicated.

Why Cursor worked better

This was to be expected, however I did not anticipate that as soon as your project spans more than a couple thousand lines things seem to completely fall apart. With Cursor this was much less noticeable because it was integrated into the IDE, putting you in charge and if you thought about the problem, added the necessary context and provided a detailed prompt on how to fix it combined with some good rules defining your overall style/taste would give quite satisfying results.

The CLI problem

Putting Claude Code into a separate terminal pushes one into the direction of only giving high-level directions, if you also couple this with a missing diff view you're pretty much expected to take on a more hands-off role and let the LLM do its thing. In general I have to say that outside of building a super dirty prototype Claude Code has become quite useless to me since any project I've been working on for more than a couple days quickly exceeds that threshold where it just starts messing things up.

It also seems as though this is a sort of gradual decline, kind of like the bigger the codebase gets the less it understands it and just ends up complicating things much further which quickly results in a state where neither LLM nor I understand what is going on anymore. Although at times it's quite surprising how it gets very simple things wrong, for example executing pmpn instead of pnpm, while it quickly realizes the mistake this mistake has me quite concerned. The missing IDE integration also means that it doesn't get immediate linting errors, instead having to manually try and run the type checker or linter (which it doesn't do all the time).

So in general I think that these AI agents need to be tightly coupled to an IDE to allow for quick review/iterations because they mess up quite often. Letting the LLM run loose for 10 minutes, generating thousands of lines of code doesn't seem to work at all apart from whipping up a quick landing page or something. Sometimes it kind of works, but as soon as you seriously start testing the code it generates it becomes super buggy.

What it's actually good at (not much)

Although to be fair, it is quite good at implementing standard functions but even here I had to interject pretty much all the time to fix errors (or prompt it to fix the mistakes). It seems that most people would like for these coding agents to run completely unattended, just give it a ticket and off we go. From what I've seen on multiple projects now this rarely works out at all. At times it also builds quite reasonable UI components as long as they're not too complicated, though I'm not sure whether this is actually helping all that much considering that there are a lot of high quality component libraries and most of the work is just combining these in a sensible manner.

What I actually want

I think tight coupling with an IDE is quite important, it also needs to be super low-effort to get the LLM to take over, like the Cmd/Ctrl + K key in Cursor (would love a voice interface here). Additionally integrating something like Playwright from the beginning and analyzing the project to provide clear instructions on how to handle common tasks as well as how to debug seem essential, somehow Claude Code doesn't seem to adhere too well to these.

The verdict

For now I'll greatly reduce my usage of AI coding tools, Cursor was in a sweet spot for a bit there, but the recent issues have made me a little pessimistic about it. Additionally it was super slow all the time, making VS Code feel almost snappy in comparison. Apart from that I'll keep working on Codewaifu in my spare time and maybe get a usable coding agent out of it.

Adiós, べン

The common i18n pattern (and why I hate it)

It's quite common to use libraries like react-i18next for building applications supporting multiple languages, they mainly work by providing a function that takes a single string argument and returns the string to be used.

I absolutely hate this approach because it's very error prone and I've only seen it work somewhat by adding helper scripts that run in CI to make sure one doesn't forget keys or uses keys that don't exist in the application.

A better, type-safe alternative

It's simple enough to not require any libraries, it's just a couple of lines of typescript that provide a much more powerful interface than libraries using the approach mentioned earlier:

i18n.ts

import { en } from "./en";
import { de } from "./de";

export const lang = document.querySelector("html")?.getAttribute("lang") || "de";
export const getKeys = (language: string) => {
    switch (language.substring(0, 2).toLowerCase()) {
        default:
        case "en":
            return en;
        case "de":
            return de;
    }
};
export const t = getKeys(lang);

en.ts

export const en = {
    cart: {
        showOrder: "Show order",
        hideOrder: "Hide order",
        submit: "Send now",
        total: "Total:",
    },
};
export type TranslationKeys = typeof en;

de.ts

import type { TranslationKeys } from "./en";

export const de: TranslationKeys = {
    cart: {
        showOrder: "Bestellung ansehen",
        hideOrder: "Bestellung verstecken",
        submit: "Jetzt absenden",
        total: "Gesamt:",
    },
};

Now you can use this system as follows in the rest of the application:

import { t } from "./i18n";

console.log(t.cart.showOrder)

Notice how we don't need to pass a string to a function, instead we have a complete object we can just pick the right string from. This has a couple of very important benefits:

• Full autocomplete without any plugins
• Using a key that doesn't exist results in a type error
• Forgetting to translate a key results in a type error as well
• We don't need to add yet another library!

I absolutely love this approach, it's much more maintainable in my experience. You can even use functions for some keys if the translation needs arguments, tsc will notice this and force you to actually call it and provide the right arguments.

Room for improvement

Right now I'm just bundling everything together for all users, this isn't much of an issue because in my projects I only have a couple hundred keys and support only a handful languages.

It's not that complicated though to adapt this scheme to dynamically import the language required. Any LLM coding agent should be able to do that in a couple of seconds.

Adiós, べン

What actually changed?

For the past year I paid $20/month and got 500 fast premium requests to models like Claude. Depending on the model, a single call cost 0.5 or 2 of those requests. The system was transparent: I could see exactly how much runway I had left, and I rarely ran dry. On the odd month I did, I knew the counter reset in a few days and life went on.

Then—overnight—Cursor announced "unlimited" requests. Sounds dreamy, right? The fine print revealed it was really $20 worth of API credits, after which you hit opaque rate-limits. Worse, if you didn't set a spend cap, the system silently flipped to usage-based pricing and people racked up triple-digit bills while they were just… coding. I had a $5 hard limit, so I dodged the bullet, but after a day or two of normal work I was throttled into oblivion.

The whole switch was communicated via one popup I reflexively accepted because I still trusted the product. Big mistake.

Where they screwed up

1. Orwellian marketing

Calling it "unlimited" when it's roughly 10% of the old quota is just plain insulting.

2. Black-box rate limits

When you hit the wall you get an error: Rate limit exceeded. How long until you can continue? Five minutes? Five hours? Five days? Cursor won't say. Compare that to Claude Code: the moment you hit the cap, it tells you "Back in 2 h 14 m" or offers an upgrade. Third-party burn-rate calculators even let you plan your session. That's how professionals treat other professionals.

3. Mystery tiers

Cursor added pricier plans that promise "more usage." How much more? No one fucking knows. I'd happily pay extra for a great tool, but after this stunt I'm not handing over any more money to them without a clear description of what I'll be getting.

4. Radio silence after the backlash

Weeks later there's still no real fix—just refunds for the surprise bills (probably to avoid lawsuits). Usage might be lighter now because people are fleeing; I actually managed a long session a couple of days ago without throttling. That's not a policy change, it's a side effect.

Trying to understand their side

I get it, Cursor probably ran into issues because if you combine Claude 4 with some MCP servers you can make a single request go quite far. I've also managed this mainly by adding the Playwright MCP Server (great tool btw!) and adjusting the system prompt to get Cursor to regularly do 20-30 tool calls with a single request, using up tokens like crazy.

It's understandable that with things changing like this the pricing strategy also needs to change for things to stay fair. Having a simple question resulting in ~50 tokens and a massive refactor producing thousands of tokens cost the same amount of requests would surely lead to problems. If things were communicated like this and with proper transparent rate limits, then I'd be a happy Cursor user still, but this is just a complete trainwreck.

So what now?

Technically Cursor is still great. The UX, the inline diffs, the little touches—they're all still there. But this wasn't a technical failure; it was a business decision executed with staggering stupidity. They torched user trust with a single update.

I've cancelled my subscription and moved to Claude Code. It isn't as charming as Cursor yet, but it's predictable, transparent, and respects my time. There are too many good AI editors out there to keep using one that apparently has no respect for their userbase.

Adiós, べン

New Architecture

This time I ditched everything that's not necessary. It's still a web app, but instead of Electron it's just a CLI app that serves a web frontend to localhost. This simplifies a lot of things because I don't need to deal with executables and any of that other stuff that doesn't really matter. A short overview of the stack:

• tRPC
• Express (with Vite middleware)
• Vite
• Mastra
• React
• shadcn
• Three.js

This is actually a rather nice stack and allows me to move quickly, especially with the help of AI coding tools. Mastra is particularly nice because it abstracts quite nicely over the native LLM APIs and allows you to very easily create agents, and much more importantly, interface with voice LLMs directly.

AI Coding Agent Integration

This pretty much describes the basics but still leaves out the coding agent-specific changes. Here I'm still experimenting quite a bit. In general, just hooking up Claude 4 to an agent that has access to the filesystem and can run CLI commands gets you quite far, but the API pricing is far too much for me. Additionally, I'd like to play around with local models.

To actually get anywhere, I'm abstracting over the AI coding assistant, giving the actual CodeWaifu assistant just access to a CLI coding agent. In the beginning it'll just be Claude Code, but on the side I'm experimenting with using Qwen3 32B, which would be far cheaper and could actually be run locally on some (beefy) machines.

Avatar Creation and Voice Models

In general, I've got to say that VRoid Studio was really nice for creating the avatar. It also exports to VRM, which is a standard format for VTuber avatars and can be easily rendered in Three.js, including animations!

So far the voice models still seem somewhat disappointing. I still have to experiment with the various models and find one that actually allows for somewhat natural conversation with workable tool calling abilities. I've gotten the furthest by just utilizing Whisper for a STT → LLM → TTS pipeline, which while not really natural, did work reasonably well.

Animation and Expression Control

Apart from that, as soon as I get this settled I'll have to experiment with which tools to provide to the avatar, since I'd like the LLM to have some control over the animation and expression of the avatar. Currently I'm thinking of building a finite state machine with the actual animations and interpolation between states being handled by normal code, but the LLM being in charge of which state to change into. Though it might be tricky to actually time it right, because it'd be great if animations could change mid-conversation in a sort of natural fashion. That could make the assistant quite lively.

Mobile Development Use Case

One other thing I'd like to try out and use CodeWaifu for is coding on the go. So far I still don't have a good setup for feeding prompts to Claude Code with just my phone. It would be great if I could just go on a webpage and talk to the avatar, which then prompts Claude Code for me. This is mainly because I'd like to experiment with coding while taking a walk or going to the gym.

Voice-Only Code Review Challenge

The main problem here will probably be that I need to think of a good way for CodeWaifu to explain what's been done using just voice. Because while Claude Code is quite competent, it still tends to make mistakes quite often and for me only really works if I at least casually glance over everything, making sure it doesn't build itself into a corner or make architectural decisions that are terrible.

For this I not only need the executive summary but actually need to be able to understand what happens in a file using voice only. But maybe it's enough to feed an entire file into an LLM and let it give a description of the functionality contained using normal language. Far too often have I heard things like "function, trim, open parenthesis, string, colon, string, close parenthesis, bracket open...", instead of something like "function trim that has a single argument, string of type string".

We'll see. If this fails, using TreeSitter to parse the syntax and then generating a sort of botched English description that an LLM can fix up might also be a workable solution.

Adiós, べン

My Chinese Progress

By now I actually know a couple of Chinese words, and the pure SRS experience has left me wanting more. Especially I've noticed that while I may know certain words, I don't really know how to utilize them. This was something that I originally intended to address via the lessons.

The Problem with the Lessons

These took quite a while to write/rewrite. I did use various LLMs to help with it, but the final editing had to be done by me, and I've rewritten most parts so they sound better as well as changed the examples generated by the LLM. Additionally, over time I've grown tired of the overall architecture, because writing generic lessons seems like the old way of doing things. I'm not a teacher, and so I was writing these lessons as I've learned things myself, so it seems like an approach better suited for someone very familiar with teaching Chinese but with less experience on the technical side.

The New Approach

Now I'm mostly experimenting with utilizing LLMs to generate content on the fly, tailored exactly to the student. Right now I have a new section for reading practice which generates simple stories using just the words the student has already learned. It also includes a question to test comprehension as well as an English translation.

These stories are still somewhat strange, though I suppose that's mainly because of my limited vocabulary. I've started adding a new category with words I'm already familiar with as well as ones I would deem useful when traveling around Taiwan (mainly related to ordering coffee/milk tea), as well as a couple of names so that the characters in the story aren't all called Xiao/Small.

Further Steps

I'll probably implement a listening exercise section pretty soon, because I can mainly use the stories generated for reading and then run them through a TTS engine to play to the student, with the question used for testing comprehension. I'll then have to think of a way to add these to the review section, because it would be quite nice if one generally just has to do the reviews and the system then automatically generates exercises for reading/listening.

I still have to think of a good way to do exercises for writing/speaking. I've done some experiments with building a voice agent on ElevenLabs, but so far this didn't work too well because it generally doubled words (I'm guessing the LLM generates pinyin which their TTS then also emits) and spoke far too quickly. Though maybe I can fix these by getting more familiar with their API, and I do have to say it was kind of fun to talk to an AI for 5 minutes, though it did show me how limited my abilities to express myself in Chinese are.

Adiós, べン

The Multiplayer Challenge

For a while now, I've been trying to get multiplayer working, but nothing quite panned out. Especially frustrating were my attempts to get Claude/Cursor to implement the final steps - I think I tried about 4 times, and it always ended up completely breaking everything. First, I simply asked it to "synchronize these entities over the network please," and nothing worked. Then I tried to be more detailed, even going as far as describing the now-implemented system in great detail (the prompt was 50 lines or so, specifying everything), but it still completely broke everything.

Yesterday I finally decided to implement these basics myself, without any AI help. I removed all non-essential features like the fire system, spells, inventory, and items, focusing solely on getting characters working well over the network with movement/attacks and world manipulation.

The good news is that I really like the current architecture. Most entities don't need to concern themselves with the network at all - they just need to provide/extend a serialize/deserialize method, and that's it. Hopefully Claude can help quite a bit with reimplementing these features now that the hard bits are taken care of.

Design Choices & Trade-offs

Before describing the architecture I've settled on, I'd like to discuss some (known) limitations of the current approach, as there are significant trade-offs to consider.

Trusting the Players

This is a big one, and a choice many games decide against. I might reconsider this decision in a future version, but for now, I'm choosing to trust players. This means I'm willing to accept a certain percentage of cheaters to simplify development and improve the overall experience for non-cheaters.

I don't see this game as particularly competitive - it's more of a fun experience to enjoy with friends on a lazy afternoon, or something to play with other students during classes.

Because of this, I think the risk of cheaters ruining the fun is somewhat low, especially if everybody involved knows each other, since even simple cheating attempts will be quite easy to notice. This makes it more of a social problem than a technical one, especially since the game is free software, making the development of hacked clients very simple (particularly with AI coding tools). It's also quite hard to distinguish between hacked clients and legitimate mods - where do you draw the line? Since I'd absolutely love to see and play modded versions of this game, I think it might be better to focus on making the game as fun as possible and building a community around it.

Additionally, trusting players makes it much easier to deal with high-latency clients, making it more enjoyable to play with friends who are far away.

Current Architecture

Generally, we have a single source of truth for any entity. The difference might be that, depending on the entity, it might not be the server.

We start by establishing a WebSocket connection to the server and then running a JSON-RPC-like protocol on top of it (with one exception: Chunks, which I'll explain later). For that, we have a class that's used by both client and server, allowing us to queue RPC calls which are batched and sent to the other end while wrapping everything in Promises so we can simply do something like:

const playerId = await this.network.setPlayerID();

Which is just some syntactic sugar on top of:

queue.call("getPlayerID", undefined);

This queue method pushes the name and arguments onto an array, adds an ID from a local counter (which is part of the reply and required for looking up the associated Promise in the client). Adding new handlers is also very simple:

queue.registerCallHandler("addLogEntry", (args: unknown) => console.log(args as string));

World Chunks

This system has been working for a while now, and generally it works quite well, although performance isn't all that great. Here, the server is the only authoritative source, but each client can change the world by calling functions on the server that can modify any block they desire. This changes the Chunk data structure on the server, updating the last_updated number (essentially an iteration/frame counter).

This leads to the server pushing updated chunks to the client because the server actually keeps track of each last_updated number for each client. We simply check all the chunks in the vicinity of the player periodically and push the entire chunk whenever something has changed (we might implement delta encoding later, but I'd prefer to optimize this worst-case first).

Additionally, the client notifies the server about chunks it's dropping because they're too far away (we have a sort of garbage collector running periodically on the client, removing distant chunks to keep heap size in check). While writing this, I realize it might be worth experimenting with running the GC whenever we have too many chunks loaded, instead of every 30 seconds or so.

Transferring chunks is also the one thing we don't do via JSON-RPC. Because we'd have to base64 encode the binary data and generate massive strings, we instead just have a different WebSocket message format which encodes the chunk data and the coordinates in a binary format, which should be much more efficient.

All of this should also be gzip encoded transparently by the WebSocket implementation (though I haven't verified this yet).

Entity Synchronization

Here we just use normal JSON-RPC style function calls. Whenever a new Entity (or most likely a subclass) gets created, we allocate a unique ID. These can be shared over the network because when we connect to a server, we reset this counter to playerID << 28, giving us many entities before we have a collision. Additionally, each Entity has an ownerID which is the playerID of the node responsible for doing physics and general world simulation (0 == server).

We now also require a serialize/deserialize method which includes these IDs as well as a keyword indicating the type of Entity. Every node also has a Map where we can get the constructor for any registered Node type. On every frame, we iterate over all Entities and serialize the ones we are responsible for, then send a big batch of these objects to the server. The server then also serializes all Entities that are not owned by that particular client and sends these objects in response.

This way only one node updates each Entity, and since the ownerID is also serialized, we can change owners for existing objects. For example, a player could use a spell creating a Fireball entity - as soon as it's fired and the position/velocity vectors are calculated, we set the ownerID to 0 and force a serialization to the server. Once the server receives this Entity, it looks up the Fireball constructor, creates a new Entity of that type, and then proceeds with updating and broadcasting serialized versions of it to all connected clients.

We could, however, also just let the player remain the owner of that particular Entity, because the server then just acts as a middleman, which works quite well for Characters, for example.

Future Improvements

I'm considering several optimizations and features for future updates:

• Refining the entity ownership transfer system
• Improving network performance (especially for Chunk updates)
• Experiment with delta compression for chunk updates

Closing Thoughts

It's taken me a while to get to this state, but I really enjoy the overall architecture. It should be simple enough for inexperienced coders and AI tools to work with, allowing for easy experimentation with different Entity types. All that's required for a new networked Entity is a serialize/deserialize method, which shouldn't be hard to implement.

Now, I've got a lot of issues and bugs to fix after testing the current version with some friends today. My next steps will focus on making the overall gameplay feel nice and polishing the user experience. Thanks for reading until the end!

Adios, べン

Why MaoMi?

After using Cursor for quite a while, I'm still super impressed with how much it can actually do when prompted properly. Especially in the beginning of a project, it can one-shot a lot of complicated programs. From my experience, most issues start appearing as the codebase grows.

There are certain projects, though, that never really grow above a certain size, mainly GameJam games and little browser games. I was thinking that for these kinds of projects, using an LLM might actually be great and even be a workable solution for non-coders.

This idea came to me because many friends have asked over the years how to get into GameDev. They enjoy playing games and have had ideas for little games they'd like to build but don't know where to start. Setting up Cursor and a local dev environment already seems too much for them. Building a platform where the beginning is super simple might be worthwhile—if the idea ends up being good, they can export everything and proceed with Cursor/Windsurf to make a proper game.

Another limitation of editors like Cursor/Windsurf is that they don't specialize in games, so they can't generate assets for you. It would be great if one could simply say I want a snake game, but instead of a snake it should be a cat eating fish and it generates the code, graphics for the cat/fish, and maybe even sound effects and a cute soundtrack.

Core Features

Here's a short overview of the planned features and the current state, since I've already been working on this for a couple of days.

Game Browser & Distribution Platform

Distribution is quite the problem. I've seen this when working with teens, teaching them how to build games with Python/PyGame. After a while, they want to show their friends the cool games they've been working on... and it's terrible. Making an executable is a lot of work, and then they might need to host it somewhere. It would be much simpler to just send a link to friends and be done with it. A browser game might even work on mobile devices (as long as there are touch controls).

To make this super easy, MaoMi has a Game Browser from the beginning. Every game being created has a little page with the game embedded as well as information about it. This section can be extended with a comment section/upvoting, but this isn't strictly necessary in the MVP phase.

Game Templates

Another issue is that people don't know where to start, which engine to pick, and generally how to proceed from their idea. To make this super simple, I've added a template system. After creating a new game, the user chooses whether it should be 2D or 3D (only 2D is supported for now). Then it uses a template with a working example in Phaser.js / Three.js depending on the choice, allowing them to iteratively change things to get closer to their idea.

By limiting the choices, we can ensure that the starting point is actually good and also ensure that the LLM knows how to develop for this. Additionally, we select technologies that don't need complicated dependencies.

AI-Powered Editor

Unlike Cursor, we're actually hiding most of the code. It is available in a separate tab, but the idea is that users should only use the AI to modify the code. Right now, we have a 2-column setup with the game running in the left column and a chat interface on the right where users can describe what they want to change. This lets them instantly see the results.

Voice Interface

I haven't started on this yet, but I'm quite excited about adding a voice interface, which might completely replace the chat interface. This should make things much easier since most people talk way faster than they can write. Additionally, this frees a lot of screen real estate, allowing a full-screen preview of the game where we're just listening to the player as they play and then modify the game as it's being tested.

Another benefit is that this actually allows it to be used on a phone/tablet, since typing is quite a chore there, but talking to your phone is much simpler and quicker. This is especially important since many people these days don't have a computer/laptop but just a phone.

Asset Generation

This is one of the biggest features for me, and also one where I still have no real idea how well it will work. I'm thinking of trying various diffusion models like Flux/Stable Diffusion with strict presets to hopefully create usable assets.

I might separate some features, for example, generating tilemaps seems like a common enough task that it might be useful as a standalone feature. Or creating a walking animation for a sprite—one just provides a sprite looking down, and the AI then generates a simple 3-frame walk cycle in all 4 directions. This would be a feature I'd love to use (a separate tool can generate the first sprite based on a description).

Sound effects would also be great to generate directly. It might be sensible to build a tool that gets two separate inputs: one describing what the sound effect should sound like, and another with an overall art direction describing the feel to achieve (a punch sound for a Mario game should sound very different from one for a gory Souls-like).

Background music could also be generated automatically given the overall art direction/tone of the game and the specific situation where it's played. At least in my opinion, it's probably more important for things to sound like they all fit together.

Version Management

Not quite as exciting as the other features, but absolutely essential nonetheless. Cursor seems unusable without a Git repo or the ability to rollback, since at times it just makes things worse with every change. Being able to start over from the last known good version is required to avoid giving up entirely.

LLM Avatar / Character

Another idea that many people really like is giving the LLM an avatar and personality. This came about because I always like to change the prompt in Cursor to give the assistant a personality, since the default seems rather dull/cold/corporate. One teen I showed Cursor to changed their assistant to only respond in rhymes, which he absolutely loved and laughed at during many interactions.

I think this can be enhanced by giving the LLM an actual avatar that can move around, show animations, and display stylized emotions. The first (and only in the MVP) character I'm building is a somewhat obese cat that is quite arrogant and smug, especially regarding users since they can't code and need to ask the cat for help. Everyone I've told about this so far thought it might be quite funny, so this will be the character I'm focusing on.

Current Implementation Status

Since I've been working for a couple of days on this already, a lot of things already work, especially everything non-LLM related is pretty much complete for the MVP stage.

Authentication System

Users can register/login (still have to add OAuth login via GitHub/Google though). There are also different user roles, mainly me being an admin and having access to a dashboard for seeing what's happening on the site. This will probably be enhanced with moderation capabilities since we do allow users to upload random content, which can lead to some abuse.

Game Management

Users can create/browse games, write descriptions in Markdown, and set games to private/public. Once we have some users, it might be sensible to add tags (which should probably be AI-generated) to simplify browsing/searching for particular games.

Version Control

Creating a new release from the current dev state is possible, as is reverting to an old version. Additionally, there is support for downloading/uploading a version using a Zip file in the same format that itch.io/LD uses, mainly for testing the iframe functionality using some old GameJam titles of mine.

Code Editor

There is a manual Editor based on CodeMirror. This isn't really intended to be used by end users, but in the beginning, it is essential since it allows me to check what the LLM is doing and fix the prompts/overall system based on what I see.

It's also nice to allow users to change assets on their own, since especially when it comes to graphics, I've seen many people really taking to it and wanting to do their own art. Here it might be good to create placeholder graphics using AI, with the user then downloading them and using them as templates for doing their own art, since they often need to have a certain format/dimensions to work properly.

AI Assistant

A basic chat is implemented. The way it's supposed to work is that the LLM has a set of commands/tools it can use to modify the codebase. In the system prompt, it is instructed to use these to help the user build their Phaser game. So far it kind of works, but considering I just started on it today, the overall architecture will still change quite a lot.

I still have to make the switch to TypeScript in the future, since right now we're just working with JavaScript, and a lot of errors appear during runtime. Making sure that at least tsc says what we're doing is ok sounds like a good step to make the overall system less frustrating to use. We can automatically send any type error to the LLM, which it'll hopefully fix automatically in a sort of agentic workflow.

So far, I still haven't read up on what others are doing or how editors like Windsurf/Cursor actually work under the hood. When it comes to projects like these, I like to start out unencumbered by knowledge of the usual structure to make sure I can freely explore. I will definitely read up a little after I've gained some experience and seen firsthand what type of problems I run into, since many of them will probably have solutions that are somewhat common knowledge.

Looking Forward

I'm still very excited about this project. There are many challenges ahead that are really difficult for me since at times I'm not sure I can actually pull things off. The asset generation feature is particularly challenging—I'm not even sure whether it can be done at all. It seems possible, but unlike with code where there is Cursor/Windsurf/bolt.new/v0, I don't know of a single tool that generates usable game art from just a prompt. I've searched around a couple of times and could only see some very complicated comfy UI workflows that worked for one particular coder and their game. So far, I haven't seen a general solution.

Adios, べン

Why Build a Custom SSG?

You may be asking why I've suddenly decided to move everything over to a custom SSG after using Zola for a bit. This came about while experimenting with structured data using JSON-LD and giving up in frustration due to the terrible templating system used by Zola.

The Problem with Templating Languages

This is actually a long-held belief of mine that has been solidified due to at least 5 instances where a templating system just wasn't powerful enough, requiring me to use some sort of workaround/escape hatch.

Generally, I'm convinced that all templating languages like Tera (used by Zola), Twig (popular with the PHP crowd) or Fluid (used by Typo3) are a terrible idea and should never be used. They're mainly advertised as being safe against code injection, but whom are they saving us from? It may just be because I've mainly worked with smaller companies, but in 15 years of consulting as a WebDev, I've NEVER had the need for someone needing to alter a template who shouldn't be allowed to alter source code. In general, it's never happened that a non-dev wanted to work on a template, though there were times when there was an employee that knew a little HTML/JS who wanted to do some changes - this only happened once.

So, now that code injection is apparently not much of an issue, what remains? You can either choose a battle-tested language like Scheme/Lisp/Lua/JS for templating or some non-standardized mess cosplaying as a programming language.

Building the Custom SSG

Since I don't like to give up on ideas, I just opened up Cursor and coded my own custom SSG in a couple of hours. Pretty sure it took me less time to rewrite everything with a custom SSG than my experiments with Zola/Tera took yesterday...

It's a terrible codebase, but it's a foundation to build upon, since only the content really matters. So now that everything works, I can just clean up the SSG with every article. Also, just as a reference: the ENTIRE SSG, including a dev server, is currently 621 LoC... that is almost nothing, especially since I haven't really slimmed down/optimized any of it by hand.

The Future of SSGs

I think it doesn't really make much sense anymore to use most SSGs anymore, since there are a ton of libraries out there doing the (somewhat) complicated bits (like parsing Markdown / YAML) and gluing things together has become much simpler with AI Editors like Windsurf/Cursor. Even before it wasn't much of an issue to use a custom SSG, I did so multiple times, once even in Nujel, which was quite the distraction. But since so far TypeScript has proven to be one of the better supported languages in LLMs, I'm sticking to it for now.

Adios, べン

Latest Developments

While the overall idea has stayed the same, pretty much every aspect has been improved and polished.

Structured Learning Path with AI-Enhanced Lessons

Before one could just add vocabulary to the review pile one by one without any sort of general explanation or plan. Now things are quite different in that there are lessons a user must work through one by one. These introduce new concepts as well as vocabulary in a preset path and then add entries to the review pile after finishing a lesson.

Writing process

Just like the normal dictionary entries, a lot of AI was used in the process here. In general, I specified the overall theme of the particular lesson and mostly specified the vocabulary that was to be taught in that particular lesson. I'd then use the AI to actually write a lesson about that, which I'd then reorder and rewrite heavily to fit the overall style. Basically a long back and forth just like the rest of the codebase.

Header Images

All of the lessons also have a header image. While it doesn't add any information, I think it does make things look somewhat nicer and in my opinion greatly helps the overview since you now have an image representing each lesson. Sometimes the image also represents the lesson quite well, while some topics were rather hard to describe in images.

I tried using both Gemini and ChatGPT at first which somewhat worked but wasn't great. In the end, I settled on the following workflow for generating these headers using Gemini/Midjourney:

• Use Gemini 2.5 Pro with a prompt telling it to create a comma-separated list of up to 15 tags specifying an image that represents the following lesson
• Copy and Paste this list and combine it with a custom prefix I've written giving an overall art direction into Midjourney
• Refine in Midjourney by generating variants of good-looking pictures and then editing/upscaling a final version

This worked really well and I still really enjoy the pictures generated via Midjourney, prompting some rather big redesigns.

Enhanced Dictionary Search

I've (well, Claude 3.7 actually) written a dead simple search for the dictionary. While it's not particularly sophisticated, it works well enough for my use case when I need to look something up to check the reading/characters for a particular word.

Improved Social Media Integration

Thanks to url2og, MandoMaster also has very nice OpenGraph images and improved metadata which makes it show up much better in most messaging apps and social media apps.

Flexible Vocabulary Card System

I also generalized the dictionary view into a sort of general vocabulary card with two forms: a big one for the dictionary or the section at the end of a lesson listing all the vocabulary. The other form is an inline form that can be used within the lesson which then automatically shows the pinyin and allows one to hear a voice sample when clicking on the particular card.

SEO and Discoverability Insights

I'm still learning a lot about how to design for searchability, but things are looking somewhat alright. It does get far more impressions/clicks than pretty much any other page I've made, which still isn't great but seems as though my efforts did have some effect. I'll probably stick to it and fine-tune things here and there in the future for various experiments.

Conclusion

Rather fun project, learned a lot about SEO though I'm still somewhat lost. Apart from that, it does work as intended and helps at least me greatly in learning/reviewing my Chinese every day. I've also finally tried out Midjourney due to this and am quite excited about using it in future projects due to the great results I've gotten so far, because I feel that similar to Cursor, it's a great tool which one just has to learn how to use properly.

Adios, べン

What's OpenGraph Anyway?

It's a standard defined by Facebook back in the day on how a website can specify how it looks when shared on social media. These metadata tags, along with some originally intended for Twitter, seem to be the main way to declare your appearance on social media.

In addition to a title and description, a pretty important part is an image representing that particular page. For the most part, I've just not done anything, but if I could be bothered, I'd either create a terrible-looking collage in Gimp or use some sort of AI tool to generate some abstract image I could just plaster on every page - not an ideal solution.

Meet url2og: A Simple Solution

A week or so ago, I read a post by @levelsio on X describing a service he vibe coded in a couple of hours that just takes a screenshot of a particular URL at a popular resolution. A great idea, I thought, and went ahead to do just the same.

How I Built It: The Fun Part

Now, I just created a new directory, initialized git/npm in it, and opened cursor, telling it what I wanted. Things worked great, and in about an hour and a couple of prompts, I got a working service. Over time, I've modified minor nitpicks I've noticed.

The Not-So-Perfect Side of Vibe Coding

One thing that for now is holding back vibe coding considerably is the over-agreeable nature of most LLMs. They are meant to be assistants to you, not to question your decisions, which leads to terrible results when vibe coding.

Generally, the LLM did not help at all when it came to creating a secure/efficient service like this. I generally had to specifically ask it to restructure things in a certain way, or at least tell it to, for example, review the codebase for security issues and harden it at least somewhat. Same for the domain whitelist (it'd have happily let me deploy a service that would just send requests to any server on the web).

Wrapping It Up: What I Learned

Well, apart from that minor excursion, things worked really great. When you pretty much know exactly what you want and just use the LLM for its superior speed in typing out the actual code and knowing how the framework du jour wants you to specify the various endpoints your service needs, then things work excellently.

Now, if you need a service like this as well and are even lazier than I am, then you can just fork/clone everything over on GitHub.

Adios, べン