The Ann Arbor Kanban Agile Lean Society had invited Kate to address the group at their monthly meeting. She agreed, but with the proviso that she wanted to get questions from the group and discuss them, rather than present some prepared topic. Since Kate’s Q&A sessions were often thought to be the best part of her evenings, KALe readily agreed.
The first arrivals weren’t even surprised to find Kate already sitting at the table in the front of the room when the doors were unlocked. Somehow people rarely saw her coming and going.
“Wow, you’re here already,” said Mark Hopkins, the KALe president.
“Hi. Yes, I was early so I just popped in. I had a little work to do, so I used the club wifi.”
“How did you get in here?” asked Mark. “The door was locked.”
“Was it? I didn’t notice,” said Kate. “I don’t quite recall. Maybe I came in through the bathroom window.”
“I see that you don’t have your standard Diet Coke. Can I get you one while we wait?”
“Yes, please, that would be lovely.”
Mark quickly returned with drinks from the treats table and they chatted while people filed in. Soon it was time to start.
Mark did a quick introduction, threw the meeting open to questions, and took a seat in the front row.
Kate made a few opening remarks, then asked for questions and topics.
There were the usual suspects. Mark did some notes on the whiteboard:
- “How should we estimate stories?”
- “The best way is ‘don’t’. Slice them small, do most important things first, count them if you need a sense of speed. But mainly, always be ready to ship.”
- “How can we prioritize refactoring?”
- “Again, don’t. It’s nearly impossible to compare the importance of refactoring with that of a feature story. Do refactoring as part of the daily work. If the code we’re working on for a story needs improvement, improve it then and there. If you must estimate, you know the code well enough to provide a proper estimate to allow for the code improvement as part of that story.”
- “What about stories that take longer than one Sprint?”
- “Never do that. Slice all stories to fit into a single Sprint.”
- “But sometimes the Product Champion wants more than that one slice.”
- “No problem. Make each slice ready to ship. Be sure to do the next most important slice next. Try to pick them so they entice everyone to release now. The idea is to be always ready to release, and the ideal is probably something like releasing every time something’s done.”
- “What if we don’t want users to see a partial feature? Then we can’t release.”
- “Look into “feature toggles”, switches in the code that turn features on and off. A toggle might gray out a menu item, or not display it at all, and so on. When the Product Champion is ready, flip the switch and there’s your feature.
Time for one more tough one
As the time for the meeting to end drew closer. Kate said, “Let’s do one more question before we close. Has anyone got a tough one?
Angel Romero raised his hand. “Kate, I work at Climate Control Corp. We’re a largish company and we’re using SAFe.”
“I’m sorry, Angel, that is a tough question. How can I help make it better?”
“Well, Kate, it’s the darn red threads. We were taught in our all-hands planning to show dependency connections between teams with red threads, and I’m in a mess with three other teams.
“We’re building a new system for large office climate control, where we’ll use parallel divided ductwork, one duct with warm air, one with cold. The idea is to provide better localized control by doing “smart mixing” in smaller areas of the building. That should help with the old hot on one side, freezing on the other problem.”
Kate laughed: “Can I sign up for the Beta? Oak River’s building isn’t very big but it seems I need to bring both a parka and a bikini to be comfortable anywhere in the building.”
Angel smiled. “I’ll keep that in mind. I would certainly enjoy helping you with that problem. I mean: CCC would enjoy helping.”
“Anyway, Angel, what’s the issue?
Angel went to the whiteboard. “We have four groups whose solutions have to go together to make this happen.
- We need centralized logic that’s aware of the whole-building duct layout, to make strategic decisions about how much hot and cold air to move into each plenum.
- We need changes to our in-duct fiber network nodes, to support what turns out to be a whole new kind of messaging. The existing nodes are just too special-purpose to be used as is.
- We need similar changes to the wireless system that is also in place. Often we don’t run wires, we use in house wifi or a dedicated channel for that. Again we have to sort out messages and formats.
- We have sensors and actuators that open and close baffles in the ductwork and at the outlets.
“My group has the centralized logic responsibility and the coordination responsibility. And getting the other groups lined up for the design changes is a nightmare.”
Kate thought for a moment and asked: “Why not have the engineers from the four groups work together?”
“We do work together,” said Angel. “I have meetings with the other three product owners every week, and we figure out stories for this project. Each of those groups is fairly large, though, and they have their own existing backlog of stories that they need to show progress on.
“So we wind up in a loop where one group takes a Sprint to write a spec, then another group (sometimes two) gets the spec, then they work on it, then we get their build and try to use – and usually debug – the implementation, and it repeats over and over.
“The simplest change is taking us three Sprints, one to spec, one to implement, and one to debug. Meanwhile, the other groups can either work randomly on guesswork, or just pause my project and work on their old backlog. It’s a nightmare.”
Kate thought for a moment and asked again: “Why not have designated engineers from the four groups work together?”
Angel frowned: “You mean transfer them into my department or something? That’ll never fly.”
Special team room
A hand was waving in the crowd. “Yes?” said Kate.
“We had a smaller case at my company,” said Barbara Thomson.
“The managers of the departments, no way were they going to transfer people. What we wound up doing was we formed a special team, with engineers from all the departments, on temporary assignment.”
“Sounds like it made sense. How did it work out?” asked Kate.
Barbara said, “It was fantastic. The project got done much more smoothly and quickly than anything before. It went so well that people kept attributing its success to it being an easy project. But it wasn’t: it was an easier way to work. We’ve replicated the approach often since then. We call it Team Room.”
“Thanks,” Kate said. “Angel, what about that? Why not have designated engineers from the four groups work together? – Is there an echo in here?”
Angel frowned. “Honestly, I’ll try. But CCC is all about following the existing SAFe process and in their interpretation that doesn’t allow for this sort of skunk-works effort. And all the managers would be concerned that the assigned engineers would go rogue and foul up our carefully crafted architecture.”
“You mean the carefully-crafted architecture that’s requiring you to rewrite your networking, build a new centralized control system and update all your sensors and controllers?”
Angel laughed. “Yeah, that carefully-crafted architecture.”
The whole audience laughed. Kate raised her hand and said In fairness, let’s see the hands of everyone who’s been faced with a carefully-crafted architecture that needed major work.”
The laughs turned a bit rueful and many hands went up.
Kate went on. “OK, so Angel can try to build a special team, but it may be politically infeasible, and it may take a long time. What else could he try?”
Hands went up again. Kate picked one and Laura Huntington said “What about Test Doubles? We had good luck with a simpler case where we had requirements on another group, and instead of writing a spec, we created Test Doubles.
“At first they were pretty simple. We’d send a canned message to a Double, representing a call to their as-yet-undefined software, and returning a canned result. We passed those tests to the other group, and in a meeting, we went over what the messages meant to us and what we needed to get back.”
“How did that go?” asked Kate.
“It went pretty well. At first our calls didn’t include all the information that the other team needed, so we iterated on that, and we even extended our Test Doubles to have enough logic to show how we expected the replies to change based on our inputs.
“The upshot was that as the other team built their component, they could plug the real component in and run our tests against the real software. When the tests failed, as they often did at first, we’d look to see if it was a defect in the new code, or a misunderstanding on our part of the growing protocol.”
Kate said, “So how did it go overall? Do you think it went faster or better?”
“It was quite a bit faster, mostly because the requirements were clear. But I don’t think it was as fast as it would have been with the Team Room idea, because we usually had to wait one Sprint between when we’d write tests for a new segment, and when we’d get the updated library. We learned early on that the actual work only took a day or so, and most of the delay was in the Sprint cycle.”
“Did you try a more continuous release approach?” Kate asked. “Having them slip you a copy before Sprint end?”
“We did, after a while, but there was still a lot of delay in turnaround, especially when something wasn’t right. They really didn’t have solid fully integrated software until the end of the Sprint.
“Even when they thought our bit was ready, their engineer would have moved on to another card, and couldn’t get back to our stuff for a day or so if we found anything wrong with it. The Test Double tests helped with that, when they were correct, but we were always learning and often discovered places where we needed to evolve.”
“Yes, thanks,” said Kate. “You can imagine, had the engineers from the two teams been working in a Team Room, they could have paired on the feature, creating the test and making it run right away. They’d have stuck to it until the new feature worked. Fewer handoffs, faster turnaround.
“That’s why the Team Room idea is so powerful. Something about individuals and interactions, something about working together, something about self-organization, blah blah, you know the drill.
“Angel, what do you think?”
“Let me see if I understand. When we’re ready to define some part of the conversation between a sensor or controller and our central unit, we create, what did you call it, a Test Double, that acts, in a sort of canned in-out fashion, like we expect the controller or sensor to act. We might request info from the sensor, or maybe make it seem to b sending us information autonomously, as a thermostat might. Then we send this Test Double our proposed commands to do whatever, open or close controls usually. If the controls are supposed to answer back, we have the Test Double just send back a canned message. Is that the idea, Laura?”
Laura said, “Yes, that’s basically it. Over time, the Doubles get smarter, and we often put in a sort of “command access” that let us select a scenario to test. For example, you might want to test what happens when the controller succeeds in opening the flap or whatever it is, and another one when it fails. We just grew into that as we went forward.”
Angel nodded. “I get it. I think we could do that, since we’re laying out written designs now, and frankly evolving them in code would be more to our liking anyway. And then once those test scenarios exist, the other teams can run them, look at the messages, advise us if we need to change them, and with any luck most of the changes will be formatting or adding additional parameters to the message.”
Kate said, “Yes, and meanwhile, the network teams can use those same tests to see what information needs to be in these new messages. I expect they’ll need you to do something special so as to preserve their old legacy format while allowing these new messages.”
“Yes,” Angel said. “And maybe, as we begin to work through a scenario with another team, it’ll make sense to just casually suggest that a couple of engineers sit down together and work out the changes jointly. We might even more toward better organization that way.”
Building based on test doubles
Kate said “We have just a bit more time. Anyone have anything to add?”
Cameron Barker raised their hand. Kate said, “Cam?”
“We did something a lot like that and it evolved very nicely,” Cam began.
“We had an app that needed to store a lot of fairly simple information persistently. We weren’t sure what kind of database we should use, and some people argued one way and some another,” they went on.
“Finally,” they said, “we built a little kind of Test Double or “Mock Object”, not so much for testing as for designing our protocol. We knew we couldn’t embed any database code in the app itself, since we didn’t know what database it would talk to.”
Kate said, “What happened, Cam?”
Cam said, “The little protocol language for access was quite simple. We just had a few queries, by name, by date or something, and basically only one write operation to set a record.”
They went on: “So our little fake object let us work out that protocol, and of course we had tests against it, saying what the gozintas and gozoutas were. Then we built a simple in-memory store, so we could run scenarios through our confirmation examples. We had a very solid script of how the thing should work.”
Angel said, “I can see how that would give you confidence in the protocol. What happened when you finally went to a database?”
Cam said, “We actually did four different ones for different installations. One went to a regular SQL server over HTTP, one went to a NoSQL database, and we built one for AWS. Most interesting, I thought, was that we took our in-memory one and made it just dump records sequentially to files on the server. While it was running normally it was incredibly fast, and if it ever crashed, it just reloaded from the files and took right off again.
“The main idea for you, I think, is that our little fake object and its tests give us a perfect way to drive our implementations of the database connections, and our confirmation examples gave us a lot of certainty that each implementation was really working.”
Angel said “Thanks! That’s what I got. And we do have at least the two protocols, the wireless one, which is vanilla HTTP, we think, and the wired one, which is a tune of our own invention from last century.
“Good stuff, thanks!”
Kate said, “It’s 8 o’clock, so let’s wrap up.
“With a multi-team effort with dependencies, it seems like we all agree that the best thing we know of is to get developers from all the teams working physically together. When that happens, things that take days or weeks get resolved in a day or even an hour. Things that would require a lot of written communication get resolved with tests and code. So working together is the best we know of.
“Then we’ve got the Test Double idea, where the Doubles help us design the protocols we need, and then also serve to communicate among the various departments, with clear and concrete tests rather than vague written descriptions. So that’s a decent fallback.
“Then Cam brought up a thing they did. They started with Test Doubles, or Mock Objects, then once they had the protocol worked out, actually built multiple implementations of their back end. Very powerful idea. You can even imagine going parallel on that part of the implementation.”
Kate asked, “What else?”
Angel said, “We touched on another idea that I’m taking home. Once we get some of these Doubles being used between the teams, we can just set up a temporary kind of Team Room meeting with a couple of our people and a couple of theirs. We’d get a similar advantage to full team room, and maybe it would break the ice on teams sharing the load instead of handing it off.”
“Good summary, folks. Thanks for coming!” said Kate.
Mark stood up. “Remember, for those who want a little relaxation before heading home, Kate has agreed to join us at the Bombay Bicycle Club game space across the parking lot.”
* * *
The after-hours group had reconvened around a table at the Bombay Bicycle Club, across the parking lot from the meeting area. There were game tables and players all around. Backgammon, chess, checkers, darts, old pinball machines: Bombay had them all. The KALe groups chatted about work, movies, and the songs playing over the sound system.
Allen Carter had stepped away from the table. When he came back, he said:
“Kate, I heard while I was up that you are this place’s darts champion. Would you like to try a quick game?”
“Allen, I don’t even compete in the tournaments. I’ve no idea how that idea even got started. But I’m willing to try one short go. How about we each throw three darts, high score wins? That shouldn’t distract us from these important matters of music and film.”
Allen agreed, and the group moved over the darts area to watch the competition. A board soon became available. There seemed to be general interest in the upcoming game among all the patrons.
Kate said, “Allen, you go first. Throw three darts for high score. Then I’ll try to beat you.”
Allen smiled, and pulled a package of custom darts out of his pocket. “Did I mention that I actually am the champion at my local darts club?”
Kate laughed. “No shame in losing to you, then. Go ahead.”
Allen tossed his fancy darts in that slow arcing style that most good players use. His accuracy was good: he hit a triple 20, then a 25.
“Going for the bullseye, were you?” Kate said.
“I’ll get it this time,” said Alan and made his final throw. Twenty, right on the edge of the 25 ring. The crowd all moaned.
“Hmm, OK, 60, 85, 105 total, is that right?” said Kate. “Hand me some darts please.”
Allen offered his fancy ones but Kate demurred. “No, just the regular ones from the tray. I don’t want to get sucked into a sport that requires expensive equipment.”
Kate lined up her chair. She threw very differently from Allen: she laid a dart in her hand, the fletching between her first and second fingers, the point outward. She threw overhand from the elbow, rather briskly, with a lot of wrist, and very little arc. Her first dart hit the double 20.
“Great beginning for me,” she said. “Stand back, I’ve got this.”
Her next throw buried its point in triple 20. “Woot, 100 already! Beginner’s luck! What were the stakes again?”
Allen said “I think it’s fortunate for me that we didn’t name any. Maybe loser buys the next round of sodas?”
Kate said “Fair enough. Now I’m going for the bullseye to finish you off with a tidy 150.”
She threw again. “Oops!” Her third dart barely missed the 25 ring, landing in the 4 wedge.
Kate smiled. “Wow, bummer, only 104 to your 105. You win! The next round’s on me!”
Allen said “Rematch, double or nothing?”
Kate said, “Maybe another time. It would probably just turn out like this time. Good game, thanks!”