Standardize and you simplify lives: everyone learns the system only once. But don’t standardize too soon; you may be locked into a primitive technology, or you may have introduced rules that turn out to be grossly inefficient, even error-inducing.Don Norman, The Design of Everyday Things
If you set off to design a language today, you probably wouldn’t ask, “Why text?”
It’s the implicit default. For good reason too—all our tools are now based around it. If you change this, you’d have to reinvent editors, compilers, debuggers, version control, and a lot more. But making a new language without considering its medium is like making a submarine without considering water.
Here's an analogy—walkie-talkies send and receive on the same frequency. Only one device can transmit at a time—you can't listen while you're talking. This affects communication in two ways. First, users transmit short messages to minimize use of the shared frequency. Second, they perform communication control themselves, like saying "over" to indicate that the frequency is open again. You don't need to do this in phone calls because everyone can talk at the same time.
A language lives within the constraints of its medium. The medium changes the way we communicate. And here, I want to do the inverse—we know our communication patterns in text. Let's use that to design the perfect medium for programming.
I'm going to look at text from the perspective of humans and machines. To make my points I have used examples from languages I’ve used a lot (C, Python, Go). I don’t mean to imply that these languages are better or worse than others. For my cause, the choice of language doesn’t change the conclusion. CSVs and C++ both inherit the same properties from their communication medium.
Also, this isn’t leading up to visual programming. That’s not where this is going.
Token identity
Escape characters are crucial for parsers. We need these because data and syntax live in the same space and parsers can't tell them apart. Parsers detect token boundaries by a change in character sets. On the other side of this are characters that you can't use in your identifiers—mostly whitespace and operators. Whitespace looks like a token separator to parsers, not data for the identifier. A variable containing an operator in its name would make parsing ambiguous.
We could get rid of these restrictions completely if we stored token boundaries. This would create an obvious distinction between data and syntax for the parser. This is important because you should have the ability to use unconventional naming if it makes the most sense for your application. Whether you use this ability is a decision that should be made by you and your team, not the parser.
Most languagesdon't allow the first character of an identifier to be a number. The parser assumes that a numeric literal is coming up when it sees a number. Reserved words can't be used as identifiers either. It's easier for the parser if keywords have a fixed meaning. In all these cases, notice how you have to manage your tokens for the convenience of parsers. You end up using underscores (or some other convention) to avoid breaking their rules.
As with token boundaries, we could lift these restrictions if token types were explicitly encoded into the source too. It's weird that we give parsers partial information and ask them tofill in the gaps later. Token boundaries and types—you have this information when writing code, but you can't explicitly give this to the parser. Doing so in text would clutter it with token lengths and types. Instead, we pass token identities implicitly using grammar and accept the trade-offs.
Explicit token identities would also solve code injections everywhere by default, but that's for another time.
Case Study: Triple-quoted strings in Python
PEP 295 – Interpretation of multiline string constants proposes a different way to interpret whitespace in triple-quoted strings. Take a look at the following examples.
There is a conflict—the whitespace can belong to either the code block (as indentation) or the string literal (as data); one interpretation isn’t more correct than the other. The root of this conflict is that the whitespace performs two functions—data, and aesthetics.
A common workaround is to remove whitespace at runtime. You improperly store a string and spend CPU cycles to clean it up at runtime. Worse, if this string is in a loop or function—either the clean-up is repeated, or the user initializes the string outside the block. A syntax problem forces you to change program logic.
Python understandably chose backward compatibility and rejected the proposal, but this was a lose-lose situation. Text put us in a position where we had to make a tradeoff and lock ourselves in. This isn’t exclusively a Python problem though. It’s a text problem. Multiline strings in Go and Javascript (and many others) work the same way.
Consider YAML multiline strings where this problem is solved with more syntax. You have six ways to handle newlines and an option to set indentation level explicitly in case your string starts with spaces.
Aesthetics
Whitespace comes with a lot of cognitive load. I just talked about how YAML has a bunch of options to interpret whitespace. Style guides like PEP-8 have extensive suggestions on whitespace use. Did you configure Git to normalize line endings? Look at the feature requests of any language and you will find one asking for support for someone's preferred style.
Encoding program structure unambiguously for parsers while looking aesthetic and readable—these are inherently conflicting goals. More syntax generally means more information about the program structure. Readability generally implies less syntax. Any amount of syntax that you choose for your language will be a tradeoff between these two goals.
Moreover, to work together, we only require a consensus on program structure, not style. Users have a variety of hardware, software, and (subjective) personal preferences. One style cannot satisfy all of them. We enforce formatting rules because style cannot be separated from structure in text.
Tools don't care about aesthetics. In fact, they have to filter out aesthetics to get to the structure. It makes writing tools difficult,
Metadata
Comments are metadata. There are many uses for them. We have conventions for tagging them, and tools like ESLint, Pylint, and Javadoc define their own structure. Every language has comments. Comments can store any amount of metadata, encoded in any format, and can be placed anywhere in code (at least the block variants).
However, comments cannot be mapped to language entities. For humans, this relationship is often apparent through visual cues, but this information does not exist in the data. Tools have to find their own way to associate comments with code.
Also notice how each comment starts with a prefix identifying the tool.
Unlike comments, Python docstrings can be mapped to a module, function, class, or method definition. This relationship is encoded in the program. However, docstrings and comments are disconnected from the code they document in some ways. If a variable or function parameter changes, its reference in the documentation has to be updated manually.
Attributes in C can be unambiguously mapped to almost anything, but these are meant for one-word tags. They are suitable for deprecation warnings and compiler flags, but you wouldn't use them for longer content like documentation, or even TODOs.
Identifiers carry metadata in some languages. Capitalized names are public in Go. Double-underscored names are reserved in C.Underscores for name mangling in Python. Names with an underscore prefix are private, reserved, or ignored, depending on the language. This pattern, where names carry extra information, is used to make syntax less apparent. In Go, this was preferred over access modifiers.
Metadata comes in many forms, but there is no generic way to encode arbitrary metadata into code. We've seen tools extend comments, so this is clearly a requirement that languages do not meet. Another problem with comments and other metadata syntax is that they are entangled with program logic. Even when editors style them differently, they can create clutter in code.
Files
Organizing code
How do you split code into files? Your criteria could be imports, compilation, semantics, or maybe ease of use. Some of these work against each other. The optimal layout for compilation can have suboptimal semantics, for example. Moreover, you're bound to prioritize one concern over others when choosing a layout.
Python creates a namespace for every file, even when a file contains only a class. This is redundant because the file boundary coincides with the class boundary. You have to choose between being forced to make a new namespace or putting all your code in a single file.
Some languages have rules about file locations and names that strongly couple information between files and code. You have to update all your imports when you refactor and move stuff.
In these cases, languages use files boundaries as metadata. Since you can't have code without files, this metadata is involuntary.
Parsing
Parsers can't resolve references on demand. When a parser encounters a function call, it can't directly go to its body because itdoesn't know where the body is located. Parsers need to index all files before mapping references to declarations or definitions. This is how hoisting in Javascript works, for example.
Files are linear. A parsing error at one point stops parsing for the rest of the file. Parsers have to be specially designed to be error tolerant, like in LSPs.
Detecting changes
Tools detect changes at the file-level. Git and Make use file modification times; Zig's incremental builds and incremental parsing in VS Code watch for filesystem events.
Whatever the method, tools only know that a file has changed. In the incremental case, they calculate diffs by parsing modified files and comparing the new (partial) syntax tree with one in their internal state. In other cases, they just do the work all over again.
Diffing is a low-level operation that tools shouldn't have to worry about. This should be handled for them by something lower in the stack. More so because we see so many tools write their own distinct implementations.
User experience
Think of files as a tree of strings that parsers contort into program structure. Files rarely reflect program structure or your mental model of it. For instance,class methods are unordered, but files or editors don't provide an unordered interface to them.
The "Go to Definition" feature in your editor essentially hides file boundaries from you. You don't care what file your code lives in.
Implementation
A weighted directed graph (hereafter: graph) is the perfect substrate for program structure. Language entities like tokens, functions, and classes are stored as vertices. Vertices have types and boundaries, which solves the token identity problem completely.
Edge weights indicate the relative order of successors. These are used to store ordered collections like struct fields and statements/expressions within functions.
Edges with null weights represent unordered relationships like metadata, classes inside modules, and functions inside classes.
Hierarchies are stored using edges, which signify a "has" or "contains" relationship. This lets me remove files and all issues that come with file linearity. Vertices can be fetched on demand and there's no need for an indexing pass to resolve references. Parsers can start parsing from anywhere. If you start from main(), you wouldn't even parse dead code.
Edges partition grammar rules. Errors in one part of the graph are isolated to that sub-graph. Also, you can add new features to the language without affecting existing ones. Remember most vexing parse in C++?
The graph is pure data—it has no aesthetics. Editors are free to render them however they want. Users can pick the style they want without affecting others. Style and aesthetics can evolve independently of program structure.
Every app that reads or modifies the graph is a client. Editors, compilers, linters, language intelligence tools, etc. all get access to the program structure by performing CRUD operations on the vertices and edges of the graph.
Clients can ask the server to notify them through events when vertices or edges of specific types are modified. This is useful for a documentation generator that only needs to do something when a documentation sub-graph has been modified. Or for a compiler that doesn't need to rebuild if only the documentation has changed.
Working together
The Graph acts like a network-shared virtual filesystem.
It's platform-independent—clients get a consistent environment on every host. Clients don't have to worry about Windows/Unix paths or finding home directories.
VS Code Remote-SSH extensionEditors and terminals can connect directly to the Graph over a lightweight connection.
Add an authentication layer and you can VS Code Live Share Zed's multiplayer editing
Grand Unified Data Bus
Graphs can represent data in Blender, KiCad, Godot, digital audio workstations, word processors, and many more. At one level of abstraction, these apps are no different from programming.
The Graph Model presents an opportunity for interoperability—you make one generic tool that works everywhere. The same issue tracker that you use for programming works with your PCB designs and video projects. The same documention tool is compatible with game engines and 3D software.
Notably, these apps can add scripting support by exposing their internal structures. Blender wouldn't need to embed Python—it could integrate with it via the graph. KiCad is already implementing a similar architecture with their IPC API.
In Working together, I talked about how users can share their work. Those benefits extend to all apps that adopt the Model. The server only sees vertices and edges. You make the collaboration system once, and all apps can use it. LibreOffice, for example, could let you share your documents over a network using the Graph.
Lastly—version control. Many of these apps save state in binary files. Some use text files that may as well be binary. They'd be easier to version control on a graph, where binary data can be broken down into smaller, more meaningful chunks.
The Graph solves everything
The Model abstracts low-level operations and simplifies parsing for tools. It gives language tools clean data to work with, and frees them up to solve UX like never before.
This is how I want to make my language.