Python Asteroids+Invaders on GitHub

Why do we work as we do? How might we improve? We look back at our process and practices in the large. We accidentally invent “Agile”.

Lookin’ into history back

Over the course of over 250 articles, this project has worked out has worked quite delightfully. We now have one program containing one basic engine, and depending on what objects we prime the engine with, we get either Asteroids or Space Invaders, two very different games, all the way down to their graphical approaches. Asteroids is vector-oriented, Invaders is pixels and blit oriented. We have simulated a large, ongoing project, including a rather surprising change of direction near the end: “Hey, we have a new game that we need!”

What have we encountered along the way, what have we learned? What general conclusions can we draw? What specific improvements would we like to make in how we work?

Overall Development Style

The overall style of development has been to slice off stories that make sense as part of the game and implement them. The order of things has been nearly arbitrary, though there is a general trend to leave things like drawing and moving objects done early, with things like scoring left until later. The result of this is that we might have a game with one asteroid, then an asteroid and a ship, then a few asteroids. This is in contrast to a full end-to-end “walking skeleton” with a ship, an asteroid, a score, and so on.

I would say that the game was usually “complete”, in that there were always things on the screen moving and possibly interacting. It just wasn’t as interesting a game today as it would be tomorrow. And this is in contrast to a game where the team thinks they have to build a game engine first and then somewhere around the day before release, they get around to building the first asteroid. OK, I exaggerate, but I’ve worked on engine-first efforts and they always came to a crunch at the end because absolutely necessary things had been left until too late, while capabilities were being built that we could have lived without. I’ve made a lot of mistakes in my life.

So the game grew, visibly becoming a game more and more as time passed. I tried to do stories in the order that a business person or game designer would call for, driven by the next best thing to make this like a fun game.

Stories First

We proceeded, generally, “stories first”. We’d look at the current game, think what new capability would make it better, and work on that for a while. After a while, that thing would be good enough and we might work on another, but we try to focus on the next most important story. (In the articles, I do spend a lot of time refactoring, and I try to make clear that I do that to show what’s possible, and I try always to mention that on a real project we’d do that refactoring spread over time, not all in a convenient article-sized lump.)

I work stories first in these articles for two main reasons.

First, when we work on visible features that the business-side people see as most important, we tend to keep them satisfied, at least compared to the old style I used to use, where we’d work on the engine, with few visible saleable capabilities, until bad things happened to me. If we deliver what they want in the order they want it, they see progress, and they feel that we are responsive to their desires. They are less likely to do something bad. The only problem is, at least some of us (like me, for example) never thought it was possible to work that way and still get a decent design.

So, second, I do things story first to show that it is perfectly possible, perhaps even easier, to evolve the design, from a simple one at the beginning to a more capable, well-crafted one at the end, while delivering features all along.

Since stories first is politically better, and since it’s perfectly technically feasible, I think it’s the way to go. I grant that it isn’t easy, although the other way isn’t easy either, we just think it is, until it isn’t. We’ll talk below about some of the things we need if we’re to work this way.

Incremental Development

The essence of this approach is that key parts of the design and implementation mature more or less in sync with the features being implemented, instead of being finalized first. Often design even lags features a little bit. After we get some asteroids and a ship, we want missiles to shoot down the asteroids, so we let all the asteroids interact with all the missiles and we figure out distances and kill radii, and such more or less “just in time”. (I don’t remember what specific order we really did those things in, that’s just an example of the kind of thing that happens.)

When we work this way, we consciously try to build only what we need, rather than building for the ages. We do try to keep things separated into sensible objects, but early on the objects are pretty simple.

If I’m not mistaken, in the early stages of Asteroids, all the asteroids were kept in one simple list, and the ship was a separate thing, and the missiles had their own list. There was code that compared all the asteroids against all the missiles, and code that compared the asteroids against the ship, and so on. Later on, we changed over to use of smarter objects than vanilla lists, ultimately building a few different version of the object currently called Fleets, which is one object holding all the Flyers in one big list. At other stages of evolution, Fleets held them all separately. I think it has switched back and forth at least once.

When we build “just enough”, or perhaps even better “not quite enough”, we are, of course, guaranteeing that we will need to change the code later. It is tempting, oh so tempting, to try to get it right the first time. We imagine that if we’d just spend a little more time, we could get this part right and then never have to “do it over”.

Frankly, this trick rarely works. Why? Primarily because at this moment we know less about what we need than we ever will again. We imagine what we will need, and our imagination is invariably wrong. We generally imagine all kinds of things to cater for that never happen, and we generally miss perfectly useful things that will be needed. Even worse, because “hey, we settled on how this would work”, we resist changing this object, because we spent so much on it and we think of it as done.

As my brother Hill puts it: We are in the business of changing code.

My projects go well because I never consider any part of them to be “done”. I try to get each part more and more cohesive, less and less coupled, more and more clear, less and less confusing … but never done.

Changes Break Things

One reason that we hesitate to change code is that we know that when we change things, we often break them. Programming is complicated and difficult at the best of times, in the best of situations, and we still make lots of mistakes. When we treat every component as “not done yet”, we make small changes all over, and some of these will inevitably be wrong.

Therefore, we cannot work incrementally? No. We have to work incrementally, even if we don’t work stories first, but we’re working stories first and so we must unquestionably work incrementally.

Since we might change anything, anything might break. (“We might break anything” would be a better phrasing.) But we must progress, therefore we must test everything, all the time. Otherwise, the program will get worse, not better.

Continuous Testing

It follows that we need to test everything all the time. If we did that with skilled testers at keyboards, it would take forever. So while I hope we have some skilled testers at keyboards, we need to precede their testing by comprehensive automated tests that we run all the time.

Tests First

Because we are programmers, we do not like to write tests for our existing code. We have convinced ourselves that it works, and therefore writing tests for it is clearly unnecessary. Besides, when we do write post-hoc tests, they just find problems in the perfect code we wrote, and that is depressing and damages our tender psyches.

But, #$@!, we need those tests, because we’re going to change that code and somehow, even though we really do our best, some of it is going to break. (“We’re going to break some of it.”) So, some programmer got a clever idea: write the test first. That’s actually kind of fun. It’s like reading a programming puzzle and then coding up the answer. And you get to make up the puzzle, so you can make it just hard enough to be interesting but not so hard that you can’t solve it.

We can get quite good at the test first thing. There are some great tools out there to help us write them, and to help us see quickly what has gone awry.

What we learn, if we are paying attention, is that the smaller we make the step between each test and the next, the smoother we progress. Small steps all tend to take the same small amount of time, and larger steps are far more variable. Most of them are kind of in proportion to size, but too often a big step gets out of control and we spend hours debugging or even days trying to get done. And the longer we spend, the more we bear down, the more we have to lose if we roll back, and we are in a hole and we keep digging.

Brother Hill puts it this way: take Many More Much Smaller Steps. MMMSS. Amen, brother!

Small Steps, Small Messes

When we work with code, we invariably leave a bit of a mess. We don’t get the names quite right, we might have some capability in the wrong object, we may have written some bits less clearly than we could, and so on. When we get it working, when our test passes, it is meet and just that we commit the code, take a break, have a few drinks, go out and knock over a bank, you know, the usual restful things. What, only me again? OK. Anyway, we are right to stop and take a break. The code may not be great, but it’s working.

Design Improvement: Refactoring

Working incrementally, in small steps, supported by tests, we will still grind to a halt because of all the small messes that we leave in the code. And, quite honestly, I think those messes are inevitable. They are due, partly, to the fact that the design is evolving, and so some aspects aren’t quite done and other code needs to compensate for a while. They are also due to the fact that by the time we finish some element, we are getting tired and we will inevitably leave things a bit rough and unfinished. And they are due to the fact that our first draft always needs editing. The design works: it is not expressed as well as it could be.

Refactoring, as everyone knows, is “improving the design of existing code”. As our system grows and evolves, its design evolves and grows with it, and if we are not to bog down in small messes everywhere, we absolutely must refactor—improve the design of the code—as we go.

TDD: Test-Driven Development

This results in a cycle. We take some goal. We express part of it as a small test.The test doesn’t run yet. We make that test work. We do another test, another. The code starts getting messy. We refactor to make it better. We have invented the famous Test-Driven Development cycle: RED, GREEN, REFACTOR.

We don’t do TDD because someone said we would be bad people if we don’t do it. We do TDD because it helps us do what we want to do, build our system incrementally, story by story, evolving the design, taking small tested steps, one at a time, until we get each story done. TDD isn’t something imposed as some kind of discipline or punishment: it is a combination of tools and practices that enables us to do what we need to do.

Look! We Invented True Agile!

Here’s what we see so far:

Story by Story

For survival, we want to proceed story by story, because that gives us our best shot at a healthy relationship with our business-side people and those who need our product.

Small Stories

The smaller the story we do, the sooner we get feedback, and when we inevitably discover the need to change the story, we’ll have less code to change, less feeling of sunk costs.

Small Programming Steps

We work in small programming steps, because larger steps are disproportionately variable with occasional horribly long delays, while small steps tend to go wrong less often.

Preceded by Tests

We need ongoing confidence that things are still working, and we find that writing tests first gives us better focus, is easier to do than writing them afterward, and provides a growing safety net as we evolve the code.

Punctuated by Refactoring

We inevitably leave small messes as we work. Refactoring cleans these up.

As the program grows, our understanding of the design grows. Refactoring lets us put our new understanding into the old code.

TDD
The practice of writing small tests, making them work, and refactoring the code as needed is called Test-Driven Development. Books, articles, tools, practices and even rituals have grown up around the “TDD” moniker. They are all just formulations that help us do what we must: evolve the code bit by bit, without ever grinding to a halt.

Lessons from Astervaders and Spaceteroids

Now I’d like to write about the things that I myself have learned or relearned, so far, as this quite pleasant project has gone one for Lo! these 254 articles so far.

Cover System Objects
Almost every time I have covered a system list with a little object aimed at the kind of list it is, it has paid off. I have found a place to put specific behavior appropriate to that list. It never hurts to cover a list, and it almost always helps.

It often pays off to cover a string to make it a Command, or an integer to make it a Money. I believe that collections are most important to cover, but I want to pay more attention to covering other objects. As an example from this program, I recently created the Masker object, covering a mask, with an immediate improvement to the code that used masks.

I want to cover system objects more often.

Test More
I do not test as often as would be ideal. I tend to wake up with an object or code in my head, and too often, I just want to code it. And yet, when I do use TDD, I never regret it and it seems that I invariably have a better experience. You’d think I was a slow learner or something.

I’ve noticed in a few recent cases that the code is better when I do this. This program is rather “event-driven” in style, so often I’ll have some event message contain code that just does whatever is needed, and that also manages the relationship with the event parameters. When I go to test such a thing, I have to provide a rather full environment for it. Often, with a bit more care, testing first, I can see a method that can be tested without setting up the whole big scheme of things. It goes faster and produces better code.

I want to start with tests more often.

Smaller Steps
No matter how small I make my tests, no matter what tiny steps I take between working, through not working, back to working again, things seem to go better. Days when I’ve committed the code a dozen times in two hours have been better on all dimensions than those with two or three or five commits. Better code, better tests, more pleasure, less stress. Just plain better.

I do not always think naturally in tiny steps. Sometimes a whole approach to something comes to mind, and I am often too ready to try to swallow the whole thing at once.

I want to try to take a moment to think of some smaller step, and then to think again of one even smaller still.

Smaller Objects
I would argue that my objects are generally fairly cohesive. The decentralized design tends to cause that: all the Asteroid-related behavior is in Asteroid class. It isn’t allowed to contaminate the Game and it can’t be in Ship, because the objects are all rather independent.

But Asteroid itself might have a few different things that it does and might benefit from some help. I’ve found, as this project wears on and on, furrowing into your soul, that providing some small helper objects makes the code better. Given the speed of our computers today, it’s hard to object to using more objects.

I want to practice making more small helpful objects.

Continued Discovery
The most recent absolutely new object came into the system yesterday. The most recent new general purpose object, TimeCapsule, three days ago. Before that, five days ago we invented Masker, the cover for Pygame masks. I’ve enjoyed discovering new and better ways to do things, and have found that they generally make things better and almost never force me into long change cycles retrofitting things. Existing things work fine, new things work better.

I’ve enjoyed learning and trying new things from Python, PyCharm, Pytest, and Pygame. I could stick with the tried and true things that I already know. Sometimes it slows me down for a minute or an hour to learn something new, but almost always it pays off. Learning usually lets me go faster, more smoothly, with less hassle and less stress.

I want to continue to discover new techniques and new objects that help me program.

Summary

This long project has, I think, demonstrated some “big important” things: we can work stories first; we can design all the tie; we do best when we test first; we do best when we refactor between tests. This is how “Agile”, works. By “Agile” I mean the Pure Quill Agile we wrote about at the turn of the century, not the ███ █████ high-pressure unSafe fakeScrum ████████ that is too often called “Agile” instead of its True Name: Dark Agile.

Excuse me. I got excited for a moment there. Ahem. I apologize for my bad language.

Point is, the real thing, real Agile, as sketched here, works well. It works better than anything I’ve ever tried in my over six decades of software development. This little project tells us a bit about how, and why, it works.

And it continues to teach me. Mostly it teaches me to go in smaller steps, testing more.

What should you do? You should do as you see fit. Would these ideas help you? I certainly hope so: I wouldn’t write them otherwise. But you get to decide. You must decide. I just wish that you will find a way have as much fun in your work as I do. Maybe these ideas will help.

See you next time!