Ben's blog

rambling through cyberspace

ZZTerm: A Terminal Emulator with Extended Graphics Capabilities

In today's post, I'm excited to share my work on ZZTerm, a next-generation terminal emulator that bridges the gap between text-based terminals and graphical applications. This project, like my previous work on RMake, is something I developed earlier but put on pause. However, I still believe strongly in its potential and am considering reviving development soon.

What is ZZTerm

It's a new terminal emulator, and since that would be rather boring there is something special about it primarily in that apart from the usual ANSI/VT-100 it can switch to a different mode for graphical applications, which allows many layers/planes of text or graphics, simple audio input/output, proper keyboard events (keyup/down events with scancodes instead of characters), mouse/gamepad handling and standard file chooser dialogs.

While this might sound like a lot and/or not necessary since one can somehow get audio working for example, it is much more complicated than just printing some characters in a terminal. My main idea was to have these features be about as complicated as printing characters.

How ZZTerm Compares to Other Terminal Emulators

Most modern terminal emulators like Alacritty, Kitty, and XTerm focus on performance, compatibility, and text rendering quality. Some, like Kitty, have added GPU acceleration and limited graphics capabilities (like inline image display), but they're still fundamentally text-oriented terminals.

What makes ZZTerm different is its explicit support for graphical applications through a standardized JSON interface. While terminals like Kitty allow displaying images, ZZTerm aims to provide a complete framework for building interactive graphical applications that:

This approach bridges the gap between terminal applications and GUI applications while maintaining the simplicity of terminal program distribution and cross-platform compatibility.

How ZZTerm Works

The way this is supposed to work is that we can send a specially escaped command to the terminal which wraps a JSON packet. This object then contains commands for the terminal which allows us to manipulate/switch planes, change the input mode and many more things. An example:

[{
    "T": "keyboardInputMode/set",
    "mode": "JSON"
},{
    "T": "plane/switch",
    "plane": 0
},{
    "T": "plane/getSize",
    "replyId": 123 // Just important if we query multiple planes and need a way to figure out which belongs to which
}]

// Due to the last message the terminal will send the following to STDIN

[{
    "T": "plane/get-size",
    "replyId": 123,
    "width": 1920,
    "height": 1080
}]

Protocol Choices

Still a little unsure about using JSON for this, would rather send S-Expressions but JSON is so super prevalent and library support is amazing so it is probably the best choice.

Why the JSON interface

This is because while working on Nujel I noticed I'd really like to have richer IO facilities, but writing bindings to SDL2 seemed rather complicated considering that I just wanted a simple interface rather than a high performance one.

This got me thinking and I thought a lot of little languages have this problem (and bigger ones as well), but pretty much any serious language can deal with JSON somehow, so why not extend the terminal interface to allow for richer interfaces. Because it is rather nice that a terminal program mostly doesn't need to care whether it is running on Windows/Linux/BSD, the terminal is mostly the same. This might also be nice for WASM programs since you then just need to get text in and out to get a fully graphical interface and one might even port ZZTerm to browsers.

Potential Use Cases

ZZTerm could be particularly valuable for:

The JSON interface makes it possible to create surprisingly complex applications with minimal effort, while maintaining the distribution simplicity of terminal applications.

Ideas I'm hesitant about

There are some ideas which might be good but I'm currently somewhat hesitant about, I'll list them here to hopefully organize my thoughts:

PNG Support

It would be quite practical if one could upload a PNG which then replaces a planes content, or download a PNG with a planes content. I'm mainly concerned about how to up/download binary files using the JSON interface, once I've found a good solution to this then it totally makes sense to support image files.

Tilemap planes

It might be nice to allow a plane to choose its font, with 0 being the system default, but if another one is used then we instead treat the chosen plane as a bitmap font. That way we could very easily create SNES style games.

Language

I'm still swaying between C and Rust, mainly because Rust is not quite as portable as C, but this is improving every day and by now it'd probably be best to just write it in Rust (built a prototype 3-4 years ago).

Future of ZZTerm

This project has seen no development for years, but writing this article has gotten me quite excited. Since I'm currently trying to start/finish one project every month I'll put it in the queue with CodeWaifu and RMake.

Also if you're interested, there is a half-baked prototype on GitHub.

Conclusion

ZZTerm represents an attempt to modernize terminal emulators for today's development needs while staying true to the simplicity that makes terminal applications so appealing. By providing a standardized way to create rich interactive applications through a JSON interface, it could fill an important gap in the developer toolset.

If you have thoughts on this project or would be interested in seeing it developed further, feel free to check out the GitHub repository or reach out with your ideas and feedback. I'd be particularly interested in hearing about potential use cases I might not have considered!