Kotlin 10
I want to understand DSLs and builders. First, I have to figure out how to make a new project with tests in it. I anticipate an onset of grumpiness as I learn new things.
Building DSLs1 in the builder style is big in Kotlin, and I find the syntax opaque because I’m not yet used to certain Kotlin syntax and idioms, and because I (therefore) don’t understand how they work. So my plan for the morning is to set up a new little Kotlin project where I explore an example DSL to increase my understanding. Feel free to skip out or to tag along, as you wish.
New Project
I want to do this in a new IDEA project, since I’ve already got all kinds of stuff in the one that I’ve been writing about, and it’s evolving in the direction of a game. So I want a clean workspace. The thing is, I don’t know how to set up a Kotlin project sensibly: I’ve never really done it, since I’ve started with various downloads. So this will be amusing, depending on how much help IDEA is.
Above is the hierarchy of the project I’ve been working in. I gather that under src I’ll want main and test folders, with kotlin under those. Let’s see what a new Kotlin project looks like.
OK, that looks like a reasonable start. I can certainly try to add a test file and see what happens. I’m sure I’ll run into trouble soon enough.
I create a class under test and as soon as I type @Test, IDEA offers me a library, and shortly, I have this:
import org.junit.jupiter.api.Test
class HookupTest {
@Test
fun hookup() {
assertThat("bar").isEqualTo("foo")
}
}
Unfortunately, assertThat is not recognized. I need to import something, and I don’t know what it is, nor does IDEA. I’ll crib from history.
I find this in my other project:
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
Unfortunately, in my IDE, the assertj is red. and is flagged unresolved reference.
I ran into that problem at about 0900, maybe 0915. I have it resolved now, plus got a pleasant 15 minute chat with GeePaw Hill, and it is 1020. Not only is all the wind out of my sails, but the sails have rotted away, the lines are fouled, and the mast has been attacked by woodpeckers.
Intercourse this project entirely. I’ve gotta take a break.
Later, at 1245
OK, after some helping from GeePaw Hill, some reading in the very interesting Existential Physics by Sabine Hossenfelder, some quietly drinking my iced chai by Chaikhana Chai, some dozing, and some wondering whether there is something in my eye, all of the above brought to you without compensating me in any fashion, I am ready to get back to my plan.
I have good news. The first bit is that the fix to make IDEA understand how to find my file was to paste this line:
testImplementation("org.assertj:assertj-core:3.23.1")
into the dependencies section of my build.gradle.kt file, which when you think about it should be totally obvious even to the most casual of observers. I also had a pleasant chat with Hill, who had invited me to join him on his porch via the wonders of modern technology. So now I have a running test:
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
class HookupTest {
@Test
fun hookup() {
assertThat("foo").isEqualTo("foo")
}
}
The other good news is that while reading the builder code that I was going to try to understand here, I had the essential insight into how it works, after two days of wondering. I had been reading Html Builder and couldn’t quite see how the thing worked, even in the playground, which is what drove me to writing this article.
Added in Edit
Let me describe a bit what I’m trying to build and therefore a bit about how the DSL has to work. The example I’m working with is a DSL to create html “conveniently”. Note that html is essentially a tree:
<html>
<head>
<title>
Some Title
</title>
</head>
<body>
...
</body>
</html>
The same is true of the window layout, another common use for a DSL like the one we’re doing. The window structure is basically a tree, a main pane with subpanes in it, with widgets in them, and on and on.
When we are writing a program to create such a thing, the code is generally more than a little inconvenient and it’s hard to see the structure that we’re building, some horrible thing like this:
html = HTML()
html.addChild("title")
html.addChild("body")
html.getChild("title").addText("Some Title")
...
result = html.toString()
Our DSL is intended to make this task easier, letting us write something closer to what we might write if we were just typing in the html, or the window tree. We’d like not to have to type the closing tags, since we always forget, so we might like something like this:
html {
head {
title {
"Some Title"
}
}
body {
...
}
}
So our mission is to come up with a little “language” that lets us do this … and in the case of Kotlin, we can do that without resorting to a scripting language of some kind that gets compiled and all … we can come up with a notation that’s pretty good, just using Kotlin capabilities.
Key point: We’re building a tree. Now back to the main thread.
My initial plan was to take the thing apart, into tests, and then build it back up, bit by bit, until I understood how it worked. There are plenty of details in that example, that one might want to understand, but I think I grasp the basics. We’ll find out.
Let’s pretend that we’re going to “build” this thing without reference to that example. We’ll probably have to look at it a few times but when we do, I’ll mention them. Basically I’m going to try to create a little builder-style HTML DSL, based on the limited understanding that I’ll describe here.
I’ll start from a simple spec, but we’re moving toward a tool where we can create an HTML document that looks like this:
<html>
<head>
</head>
</html>
Not particularly fancy but we like to start small. Now we could just type that in, but we’re building a builder even though we don’t know quite how. So we want a bunch of classes and methods to do the job for us. My probably fated plan is to do that here. I am delaying. Let’s write a test.
@Test
internal fun justHtml() {
val h = html()
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n</html>")
}
I’m positing a function, html that returns something whose toString is
<html>
</html>
Seems easy enough so far. Since I only have a vague idea of builder building so if it gets weird for a while, I’m down with that. This isn’t a tutorial on how to build a builder, it’s an example of one person figuring it out from some basic notions.
I could do something truly “just enough to run” but instead I’m going to build a little class.
I create this function, at the top level for now:
fun html() {
val html = HTML()
return html
}
And this class:
class HTML {
override fun toString(): String {
return "<html>\n</html>"
}
}
I rather expect my test to run. Doesn’t. The function declaration is wrong. I’m so used to just returning whatever I want. I think it needs to be:
fun html(): HTML {
val html = HTML()
return html
}
Test runs. Now I’m going to do something that I stole from the example, but … no, let’s hold off a bit and do a harder test:
@Test
internal fun htmlWithHead() {
val h = html {
head
}
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n <head>\n </head>\n</html>")
}
This is my DSL infant. But what does this code even mean? As a Kotlin newbie I didn’t know. We’ll find out but first I need to implement some stuff.
class HTML {
override fun toString(): String {
return "<html>\n</html>"
}
fun head() {
return Head()
}
}
class Head {
override fun toString(): String {
return " <head>\n </head>"
}
}
New method on HTML, head, and new class Head that can return a string that’s roughly like what we want. This code won’t work yet. What will it do? I’m not really sure.
Let me state clearly here: I think I know what I want to do. I don’t quite know how to explain it, or do it, in Kotlin. In Lua, we’d be further along now. Anyway …
When we call html, we’d normally have to give it parameters in parens, as we did in the first test:
@Test
internal fun justHtml() {
val h = html()
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n</html>")
}
- Lightly Edited 13 Aug
-
But Kotlin has a special notation if the last parameter to a [function or] class creation call is a function (lambda): instead of putting the lambda inside the parens, you can include any other parameters in parens, if there are any, and then put the lambda in braces. So the notation in the new test expects that the HTML constructor should expect a function. And if
headis to mean anything, it has to be a function that can be called on the HTML instance, not least because that’s where we put it. - Added 13 Aug
-
We are building a tree of various class instances by calling methods on the parent classes that build instances of child classes under themselves. As we do that …
-
The syntax
name {someCode}means “call the functionname, passing as its only parameter the function defined by{someCode}.”
So …
fun html(init: HTML.() -> Unit): HTML {
val html = HTML()
html.init()
return html
}
After a bit of arguing with Kotlin, I have this for my tests:
@Test
internal fun justHtml() {
val h = html {}
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n</html>")
}
@Test
internal fun htmlWithHead() {
val h = html {
head()
}
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n <head>\n </head>\n</html>")
}
The first one expected a lambda, so I provided one. The second wanted me to call the head, not just name it, so I had to add the parens. The test fails, of course, because while Head knows how to return a string, no one is asking for it.
Here’s where we have to begin to understand how this builder works. We want the HTML object to save “children”, which will be instances of things that can be inside it, including head (and, someday soon, body). Then, when the HTML is told to emit its string, it should emit its own beginning, then whatever its children emit, and then its ending.
In the fullness of time, we’ll make the objects do the same all the way down, and we’ll build some kind of useful hierarchy to remove the inevitable betrayal, er I mean the inevitable duplication.
So, how do we get the HTML to save the head as a child?
Friday 0730
I stopped writing abruptly above, but in the presence of code and tests that will let me pick up where I left off. I don’t recall why I stopped.
What happened was that my second test ran, the one that demands
"<html>\n <head>\n </head>\n</html>"
And the first test broke, saying this:
expected:
"<html>
</html>"
but was:
"<html>
</html>"
The cause of that was immediately clear to me, because I knew what I had just coded, and either I was interrupted to feed the cat, or I was just tired, because I walked away from the keyboard without a word.
Of course, a broken test is exactly the right time to walk away, because it’ll remind us where we were. However, it is a new day and I do have to refresh my mind with where the code stands. I glance at the tests:
@Test
internal fun justHtml() {
val h = html {}
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n</html>")
}
@Test
internal fun htmlWithHead() {
val h = html {
head()
}
val hs = h.toString()
assertThat(hs).isEqualTo("<html>\n <head>\n </head>\n</html>")
}
In the second test, we’re starting to get the outline-style “language” that we want … at some point we will go deeper. Now in the code, we’ll see that what I have is a lot like I wanted, so it was perhaps time to back away slowly, until I was more fresh.
class HTML() {
val children = arrayListOf<Head>()
override fun toString(): String {
var result = "<html>\n"
for (c in children) {
result = result.plus(c.toString())
}
result = result.plus("\n</html>")
return result
}
fun head():Head {
val head = Head()
children.add(head)
return head
}
}
class Head() {
override fun toString(): String {
return " <head>\n </head>"
}
}
So what we see here is that the HTML class understands head, and what that method does is just instantiate a Head object and add it to children. The children collection is the new bit. When we think of what we need, we can see why we do this thing with the children.
When we render the HTML object, we want to put out the <html>, then put out whatever should be inside it, and then put out </html>. So we have to save the “whatever”, so that when we do our toString, it can be like this:
override fun toString(): String {
var result = "<html>\n"
for (c in children) {
result = result.plus(c.toString())
}
result = result.plus("\n</html>")
return result
}
Why did the test break? Well, it’s just that we don’t have a solid agreement on when a given object will put in a newline and when it won’t. Here’s the Head version:
override fun toString(): String {
return " <head>\n </head>"
}
In the output of the opening tag, we are not including a leading return, but in the trailing one,we are. But the HTML one also includes a trailing newline in its opening. We need a consistent rule, and I propose that it is that everyone will emit their opening and closing tag with a leading newline. This is going to mean changing the tests to expect a newline before <html>, but that’s no biggie.
The changes:
override fun toString(): String {
var result = "\n<html>"
for (c in children) {
result = result.plus(c.toString())
}
result = result.plus("\n</html>")
return result
}
override fun toString(): String {
return "\n <head>\n </head>"
}
I expect this to break both tests, because of a missing newline at the beginning. Run them to see:
expected:
"<html>
<head>
</head>
</html>"
but was:
"
<html>
<head>
</head>
</html>"
And:
expected:
"<html>
</html>"
but was:
"
<html>
</html>"
Now, for consistency, I would argue that we should leave the code as it is, always outputting a newline ahead of each line of output, and deal with the initial leading newlines in the tests. For convenience and ease of remembering, I would argue that we should relax our rule and allow the <html> tag to be emitted without the newline prefix.
I think I’ll go with the consistency argument and change the tests. If I’m wrong, I’ll often forget the newlines in future tests, but they shouldn’t be hard to figure out. And I can always change my mind at the cost of adjusting a few tests.
So, fix the tests:
@Test
internal fun justHtml() {
val h = html {}
val hs = h.toString()
assertThat(hs).isEqualTo("\n<html>\n</html>")
}
@Test
internal fun htmlWithHead() {
val h = html {
head()
}
val hs = h.toString()
assertThat(hs).isEqualTo("\n<html>\n <head>\n </head>\n</html>")
}
I notice that there’s no trailing newline, but you can’t have everything.
Let’s step back and review what we’ve done so far.
Retrospective
Recall that our DSL is building a tree, in this case a tree of HTML. When we process a tree, we generally want to do something at the beginning of each node, then do the children, then do something at the end of the node. It’s kind of recursive process, because a tree is a recursive structure.
Now in some cases, we could just generate a bunch of routines that are mutually recursive and with any luck at all, everything would unwind just fine. Here, though, we are trying to build the structure: outputting it as a string is just what this example does. Another example might define a dungeon, something like:
dungeon
room
location(100,100)
size( 10,15)
treasure
chest
gold(20)
mimic
debris
skeleton
vase
damage(5)
room
location(200,150)
...
Here, we wouldn’t print the tree (except for testing and debugging perhaps). We would interpret it and the various rooms and items would be created.
I’m guessing that we often, if not always, have the option of building a tree structure and interpreting it later, or just doing the thing right in place. But … when some of the things in the tree need to reference others … we might prefer to build the tree completely and only then interpret it.
Anyway, that’s what we’re doing here: we’re building the tree and interpreting it at the end.
We only have one object that can have children now, the HTML instance. And we only have one kind of child, the Head instance. But that can and will change.
Hmm. I find myself in an odd position. I think I know how to elaborate this forward, but I’m having difficulty explaining my understanding. Of course, this is my first DSL, so it shouldn’t be a surprise that I don’t have it all clear in my mind. But I’m not used to it: most of the time I’m either deeply confused and have no idea, or I have the idea and can explain and mostly program it. Interesting.
We’ll program further and see whether the code will lead us to clarity.
Let’s try the other option under html, the body. We need a test:
@Test
internal fun htmlWithHeadAndBody() {
val h = html {
head()
body()
}
val hs = h.toString()
assertThat(hs).isEqualTo("\n<html>\n <head>\n </head>\n <body>\n </body>\n</html>" )
}
This will not compile, because Kotlin knows that the HTML object does not understand body and it sees that we are calling that method. Why? Maybe I didn’t make that clear.
When we write
html { foo () }
That means that the html function expects a lambda expression as its final (and only) argument. But it has told us what the type of that expression is:
fun html(init: HTML.() -> Unit): HTML {
val html = HTML()
html.init()
return html
}
The function’s final (and only) parameter is init which is a function of type HTML.() to Unit. The HTML.() means, don’t ask me to explain this, a function defined in HTML (the class). So the compiler knows that head and body are to be methods of HTML, and, so far, body isn’t.
We must fix that to run the test and, of course, we’re going to mimic what we did for head:
fun head():Head {
val head = Head()
children.add(head)
return head
}
Easily done:
fun body():Body {
val body = Body()
children.add(body)
return body
}
Now, of course, it won’t compile because we have no class Body, so we provide that, much like we have for Head:
class Head() {
override fun toString(): String {
return "\n <head>\n </head>"
}
}
class Body() {
override fun toString(): String {
return "\n <body>\n </body>"
}
}
I’d like to find out why this won’t work, so I run the tests. The error surprises me just for a moment:
Type mismatch: inferred type is Body but Head was expected
The message is on this line:
fun body():Body {
val body = Body()
children.add(body) // <---
return body
}
And the bug is that we’ve declared that collection as a collection of Heads:
class HTML() {
val children = arrayListOf<Head>()
Well then there now, we could probably make this a collection of Any and maybe it would work. I don’t know Kotlin well enough to know if this will work, so let’s try it even though it’s arguably evil.
class HTML() {
val children = arrayListOf<Any>()
Ha! Duck typing FTW! The tests all run. Let’s be having a commit up in this baby, OK? Commit: now supporting head and body.
Whoa! Suddenly the commit wants to commit all kinds of random gradle this and that files. .gitignore, gradle this and that, things I’ve never seen show up before. Have I mentioned the frustration I feel when things like this happen? Anyway, I just let it save my files, not the other stuff.
Now what we should really do, I imagine, is to create a common abstract superclass for Head and Body, and let that be the type we use in the arrayList. Shall we do that? It might be premature but I think it’ll be OK.
We’ll call it, I don’t know, Element. (Thing was another possibility, maybe also Item.)
Well, I’m wrong. I can’t quite work out how to make it work with Element being a class, but this works:
interface Element
class Head():Element {
override fun toString(): String {
return "\n <head>\n </head>"
}
}
class Body():Element {
override fun toString(): String {
return "\n <body>\n </body>"
}
}
The messages I got at first told me that if Element was a class it had to be “open” but when I made it open, I started getting messages about needing a constructor. Some research will sort me out but calling it an interface makes my tests go green and allows me a more restrictive kind of children:
class HTML() {
val children = arrayListOf<Element>()
So that’s nice. Commit: Head and Body inherit interface Element, all children have to be Elements.
I may regret this but I just let it commit all that other stuff. This is a throwaway learning project anyway.
I think the next step should probably be to put something inside either head or body. That will drive out the “discovery” that Head and Body instances, like HTML ones, need children of their own. That will create duplication, which we’ll use to figure out a better design. We should perhaps talk about this.
Discovering the Design
Now it’s pretty clear that most of our Elements will have children of their own, so that tells us that they probably do have some kind of superclass in common. We also have this duplication that has arisen already:
class Head():Element {
override fun toString(): String {
return "\n <head>\n </head>"
}
}
class Body():Element {
override fun toString(): String {
return "\n <body>\n </body>"
}
}
The only difference here is the string “head” vs “body”, so there ought to be some way to make that a parameter relating to our methods head and body, rolling that into a common class.
So we see that our design needs a more capable hierarchy. And it would be entirely possible to stop now and design a class hierarchy on paper or iPad or cards, and to decide what common interfaces and classes we should implement. And in my over six decades of programming, I’ve done that often, and often one it rather well. Yet frequent readers will notice that I almost never do that kind of upfront design any more, even when it’s staring us in the face like it is here.
Why don’t you do much upfront design, Ron?
You may wonder why I don’t do much upfront design. Well, even if you don’t, I’m going to tell you.
First, I work that way because in real code, the design we have is rarely the best design we can imagine, and so I like to let the design of my little samples go a bit awry, and then recover them by refactoring. I’m trying to show that we can always improve the design by refactoring: we don’t have to get it right up front.
Second, I’ve almost invariably found that my paper designs don’t hold up. They’ll be close to right, but in the doing, I always discover little things that need to change. Oversights, misplacement of objects, sometimes just lines drawn in the wrong place. When I coded up my own designs, I’d just go “oh, yeah” and do the right thing. But if I had turned that design over to someone else, and they had implemented it … it might not even have worked, and it certainly would have generated a lot of phone calls asking me what the heck.
Kent Beck put it well when he said “I like to let my code participate in my designs”. I push that idea further in my articles, to the point where while I’m always thinking about design, I try never to put something in before I need it.
I do often fail at that … and I think I’ve even put in something before I needed it in this article, certainly in this series, because when I’ve got an idea, sometimes I just run with it.
I never said I was perfect, or even consistently imperfect. I’m just some guy, sitting at his computer, giving what he’s got. And a big part of what I’ve got, one of the better parts, is that I’ve found that when I go in small steps, with moves that are not obviously stupid, and when I pay attention to how the design is evolving, I can guide the code to a better place, while keeping it working and becoming more capable. That sentence is long … because that’s a big deal, the difference between my code getting worse and worse, slowing me own, and staying clean, so that I can move smoothly, with style and grace, almost all the time.
Almost. No process, no person, is perfect. At least not in this house.
Anyway, we’re discovering the design in a very real sense. We create something, we look at it, we observe that it could be better, and we find a way to make it better. To me it feels like discovery. I’m not bearing down, forcing a design out of my head. I take simple ideas implement them, then use other simple ideas to improve things. Simple ideas just come to us almost without effort. It’s like we find them in our head. We don’t have to grind them out.
Or so it seems to me.
Where Are We?
OK, enough blue sky. Where are we? We have a rudimentary but not unattractive tree-building “DSL”. I can already see, and I hope that you can, that, elaborated, this thing will be able to deal with all the html we want to push into it.
More to the point, the technique of using these class constructors that end with a lambda function is general enough to use for a wide range of problems, including creating windows (which is why TornadoFX is the way it is), creating game layouts, probably anything where we have containers containing containers.
The essential trick, I think, is that each kind of node xyz in the tree will be represented by a class XYZ, and when that class is created, it expects a lambda of the form
XYZ.()->Unit
That is, making calls to the XYZ instance. When I look at the code in IDEA, it looks like this:
val h = html { this: HTML
head()
body()
}
Inside those brackets calls like head() and body() are interpreted as methods on HTML. When we get there, similarly, calls inside our head or body in the outline will be interpreted as calls to methods in Head class or Body class, respectively.
Are you finding this hard to think about? Frankly, I am, or at least hard to write about. It’s as if there’s a bit too much to keep in mind all at the same time.
My old friend Gene Somdahl, one of the best programmers I’ve ever known, used to say “You can hold three things in two hands, if you keep one of the things in the air”. This comment usually prefixed a patch of assembly code that somehow managed to move three things through two registers without losing any of them. I’m sure Gene was right, and in this case, I can’t keep the three things in my head at the same time. Fortunately, the code doesn’t care. It keeps track for me and once I get the layout right, it just works.
We Had Better Stop
It’s 1010, so I can legitimately pause, and the article is going over 700 lines of typing, so it’s surely too long. Let’s sum up and continue at a later date or time.
We have two different methods on HTML doing similar things, going into HTML’s children as implementors of an interface Element. We can see the shape of the rest of the DSL coming into view. Each node will be represented by an instance of some class (we’ll pretend for now that they’re all different), and that class will provide a method for each node that can occur under it, just as HTML provides for head and body. We could just rinse, repeat, until done.
However, we already see some duplication cropping up, and there will surely be more. That tells us that we will probably benefit from some inheritance, and perhaps some helper classes, to eliminate the duplication, since as I always2 used to say back in Rome, “Ceterum autem censeo duplicatio esse delendam”.
We’ll push forward next time. This is enough for now.
See you anon!
-
Domain Specific Language. A formal syntax for accomplishing some thing, like laying out a window or a dungeon, implemented as if it were a little language all of its own. Famous for confusing readers because they’ve never seen that particular one before. Nearly impossible to use until you know them, and nearly impossible to learn if you don’t already know them. Hmm, grumpy already … ↩
-
I used to end all my Senate speeches that way. Sometimes, for short, I’d say “Duplicatio delenda est”. ↩