AdrianCarr.com ^1.0

For some strange reason, I think a lot about software, business value, economics, work environments, technology, and mostly about how all these things come together. Sometimes, to help me make better sense of my thoughts and to keep them from getting all jumbled together, I write them down.

Not too long ago, I started sharing them out on the internet. At the time, I really didn't know why I decided to share them. Most (probably all) of the people around me are smarter than I am, and I really don't write very well, and I'm definitely not a great programmer (when compared to the amazingly smart people around me). So why should I write about and share these things when there are others that are much better at this than I am?

Well, I'm glad you asked.

One of the reasons is that I know from experience that it is a very, very bad idea to go off and work on software for a long period of time and then eventually hand it over to the customer and hope you got everything right. For some people (and me at one time), this is a revelation. I mean, how could we not get everything right when our business analysts spent so much time interviewing and meeting with the customer and getting every last detail agreed to and then signed off on? This concept became very clear to me several years ago when working on a multi-year project where the company I worked for provided a very large software package for Toyota Financial Services.

Talented Business Analysts.

First of all, we had extremely talented and very thorough business analysts. These business analysts were smart and knew their stuff. They had years of experience in the banking industry, and they knew our product really well. They went to California and worked with Toyota business analysts and gathered the requirements and spelled them out in great detail. It was really important that the requirements be spelled out because the contract, including pricing and timeline, was based on these requirements. Since the contract was based on the detailed requirements, changes were strongly discouraged so the requirements had to be right from the start. It seemed like they haggled and negotiated over the contract for months, but eventually, it was finally signed off by both parties.

Great Project Managers.

We also had great project managers. The project was broken into three phases that were scheduled over a couple years. Project managers created very detailed charts and timelines and strategies for success. The requirements for the first phase were handed off to us developers. We designed and developed the software for months and months. The project managers created beautiful progress charts from the weekly status reports we filled out, and distributed them to the managers and to project managers at Toyota. We all felt great about it for quite a while. We all smiled, the company threw parties, and life was great.

These status reports remind me of a presentation by Scott Ambler about agile practices that I attended at a Dr. Dobb's Best Practices conference in Boston. Someone stopped him and asked "What about the status reports?" Scott calmly turned to everyone in the room and asked them: "How many of you have to fill out status reports?" Around 80% of the people raised their hands. Scott then asked them: "How many of you trust the status reports? Two or three hands went up. This was in an audience of about a hundred people. It was immediately obvious. Status reports are too often inaccurate, and people are usually making the numbers "look right". I used to do it, thinking that I'll have a breakthrough soon where I make a giant leap forward. Traditional status reports are a waste of time and often give people a false sense of security.

The Development Process.

After a while, I became a team lead, and soon realized that I wasn't the only developer stuck at a status of 80-85% done for week after week. It seemed like that last 20% took half of the time, especially when we ran into some ambiguous detail that a person could read in a couple different ways.

"Your entree comes with a salad and fries or green beans." 

Wait a minute. Does that mean I always get a salad and I can choose between the fries or beans? Or, does that mean I can either have a salad or I can have the fries and beans?
On a software project, one way might mean a couple days of work. The other way might mean a couple weeks or months of work. So clarification is important, but sometimes takes a lot of time, especially when the communication has to filter its way through lots of people. Then, what happens when the clarification comes back and everyone on our side wrongly assumed that it meant the easier option? Or, what about when the you reach a technical limitation and you aren't going to be able to deliver the exact feature as described? Uh oh. Change management time, and arguing over the contract, who's going to pay for the increased time to develop it, and who meant what when they wrote the requirement, whose fault it is that the misunderstanding occurred, etc. It's not pretty.

Late to QA and Getting Later.

Eventually, we finished the last little bits (well most of them, with the unfinished part to be tacked on to the next QA cycle) and handed it over to our QA department. Of course, by this time, we were running late. Then, an unexpected (at least, unexpected to me) thing happened. They found all kinds of bugs. I mean LOTS of bugs. Big bugs, little bugs, hairy bugs. So we fixed most of them and gave it back to them. They found more bugs, and we fixed them and gave it back again. We did this over and over and over until we finally had a pretty stable release ready. We sent off a release to Toyota to install and for them to do some quality assurance on.

The QA staff at Toyota now had their turn, and a surprising (at least to me) thing happened. They found some bugs. All kinds of bugs. Not nearly as many as our QA department had found, but quite a few nonetheless. Well, we fixed most of them and gave a version to our QA department who verified that they were fixed, and we gave Toyota another, cleaner version.

Of course, by this time, we were running pretty far behind schedule. The managers and project managers were wringing their hands and having lots of meetings and I heard some rumblings about the contract, but the old timers on the development staff didn't seem too surprised by all this, and looked at it as quite normal.

Delivery Time.

Once the release was ready and all the details worked out, our managers asked for volunteers to travel to Toyota to support the install and release. I signed up to go. I had never been to California, I enjoy traveling, and like a good adventure from time to time, so why not?

User Acceptance Testing.

While I was there, I watched people use the software for the very first time. I showed up bright and early the first day, and waited with nervous anticipation for the install and everything to be turned on. After a couple small hiccups that we were able to overcome and document and promise would never happen again, we turned on the system and a few people logged in. One of the first people to login for the UAT was a really nice lady. I don't know what her normal job was, but clearly, she was used to doing some job that our new system was going to have an effect on and she had been asked to evaluate it. She used our software for about five minutes before she called me over because she wanted to show me something. Imagine my surprise when she told me that our new, fancy, multi-year, multi-million dollar system wouldn't work for her and couldn't possibly go to production. I was floored. I mean completely flabbergasted. I couldn't hardly believe her. But, she patiently showed me why it wouldn't work, and explained her job to me and how her old system worked, and how the new system worked. A Toyota business analyst came over and looked at it, and studied it for a long time, and eventually agreed with her.

I didn't know what to do, but a light bulb came on for me. I thought to myself: "How could we have missed that? If the developer who did that work knew what the end user really wanted and had worked with them, they would have gotten it right the first time, with ease." This was the first time I realized that maybe something wasn't quite right with this long discovery-document-design-development-test-deliver strategy. But, that thought didn't last long, because my next thought was: "Hmmmm..... I have the source code, and I could fix that in about half an hour." After some meetings, some explaining, negotiating, documenting, and hand wringing, they actually let me make the change and recompile right then on my laptop. I was pleasantly surprised, but glad that we could keep the project moving forward.

Production, and Phase two

We eventually got the first phase of our system in production about a year after starting the project, much later than originally planned. It didn't take long for new bug reports to come in, and feature changes to be requested. We had lots of situations where the detailed specification just didn't catch all that was needed, or the programmer completed it as described, but not in a way that was really useful to the actual end user who had to sit down and work with it day after day. Now, we had a dilemma. Since we were already behind, we needed to get right to work on the next phase of the project. But, we also had more work that came out of the release than we expected. Once again, many meetings, org changes, wringing of hands by project managers, and contract renegotiation. It wasn't pretty.

Back to the grindstone.

Eventually, the project managers figured it all out and we got back to work. We worked through the bug list, and added the feature requests and unexpected changes to the list of planned enhancements. It wasn't long before we were back in the same situation we were before, where we were running behind, lots of bugs, endless QA cycles, painful change management meetings, and eventually, loooooong days and some weekends.

The Eventual Outcome.

I won't bore you with the rest of the details, but suffice it to say that we basically repeated the process from phase one. The project went downhill, ran later and later, ended up costing lots more than originally planned, the company lost lots of money on it, and Toyota ended up suing us. Parts of the project were in production, and parts never made it. Millions of dollars worth of hard work, careful planning, and functionality never made it to production. Millions of dollars spent without the business value realized.

Could this project have been successful?

I think so. I saw both sides, and besides being a developer and team lead, I filled in as a business analyst, and even did a short stint in a project manager role. I may be dreaming, but I really think I could take this same project and make it successful with what I know today. If I'm completely wrong, it would at least fail faster, and as a result, save millions of dollars, some bad feelings, and a lawsuit.

What would I do different?

The first thing that I would do differently is that I would run the project with agile principles instead of as a traditional project.
As a review, here is what the Manifesto for Agile Software Development says:

We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Get the people together.

One of the first things that I would do is get the people on the projects together, even if it was only temporary and occasional. Since Toyota Financial Services is headquartered in California, and we were in Georgia, this would have been difficult and expensive, but worth it. I want people on a project to know who they are working with and developing for, not just what they are developing. I would have the people from Toyota show the old system to the developers, and show them how the business works and talk about what they really hoped to accomplish with the new system. When developers know whose problem they are trying to solve, and what that problem looks like from an end user's perspective, they typically do better work.

Create a better contract.

With traditional contracts where the cost, scope, and time are fixed, someone is going to get the bad end of the deal almost every time on a large project. This has worked several times for me on small projects, but on large projects, it's a losing game. Think about it. Let's assume you're going to sign a contract with me that says you will do "x" for "y" dollars in "z" time before we ever start the project. You have a motivation to make sure you don't get the bad end of the deal, so you make a lot of assumptions and then "pad" the amount of time and the cost upwards to make sure you don't lose money. From my point of view, I feel good because I know what my expenses are going to be. Now, the financial risk is on you. However, I also know that I have to get my requirements defined up front, and while we will probably have some small amount of changes specified in the contract, I have to make sure I get almost all of my functionality defined before we even sign the contract. Now, stop to think about this for a minute. At what phase of a project does everyone know the least about it? Answer:At the start of the project. Why do we think it makes sense to try to define everything and contractually obligate ourselves at the point where we know the least? This is stupid. What it leads to is a lot of guessing and assumptions on everyone's part so they don't get a bad deal. It also has the side effect of adding unneeded functionality to the list of requirements. If change management is resisted so hard, and the contract doesn't allow large changes later, I have to add things to the requirements that I may or may not need, because if I figure out that I do need that functionality later, it is too expensive or difficult to add it after the project has started. Several studies have shown that a huge percentage of features in large software systems are never used. (You can read more about this here. )This is a waste of money and time. Agile projects allow the business to get the highest value features and functionality in production soon, before all of the less valuable stuff is implemented.

I would still have a contract, and I won't go into a lot of detail here, but it would be crafted in a way that would encourage us to work together, rather than foster an "us" vs. "them" mentality. I wouldn't fix the time or the scope. If they needed to put a cap on the cost, that would be fine, but the contract would require regular payments as functionality was delivered, and we would keep delivering until the cost cap was reached. Hopefully, by the time the price limit was reached, we would either be done, or would have built a level of trust that allowed us to continue working together. This encourages us to deliver useful, working software early and often. There would be requirements and rewards for regular delivery of useful software. There would be some protections for each "side", but encouragement to keep the project moving forward in a manner that serves both parties. Changes would be welcomed, and encouraged. To sum it up, the contract would be written in a way that encourages "customer collaboration over contract negotiation".

Build Quality In

Our QA department was pretty good, but they never got to do any testing until the end of the development cycles. Often, I'd get a bug report and have to go back and fix it, and I hadn't worked on that code in months. I'd long since moved on to lots of other features and changes. It sometimes took a while just to remember what exactly I had done and why I had done it. Not only that, but almost every time, we had to cut the planned QA time drastically, and they often didn't have time to test everything.

I would also integrate the QA staff onto small teams with the developers and business analysts and have them all sit near each other and work together daily from the start. As discussions were happening between the developer and business analyst, the QA person would hear what was being discussed. I'd get the QA person involved in design and database discussions so they understood things better.

Automate testing. I would require that my QA people be able to write automated tests. I would also require that my developers write unit tests. Now, this would have been particularly difficult on this project because the code base was huge, and old, and written in Visual Basic 6. However, it can be done. As more and more unit tests get created, you get more and more code coverage, and you get more confidence in changing complex business logic, and faster feedback when things are broken.
Here's a great quote from a member of the Agile Atlanta User's Group:

Top-producing teams don't correct errors, they prevent them, using all tools at their disposal, including their own attitudes; that's why they are top-producing teams. -Joseph Beckenbach 

Frequent Delivery of Working Software

In the project described above, the primary measure of progress was Gantt charts, documentation, training manuals, and promises of working software. The primary measure of progress on an agile project is working software. I wouldn't even think of going a year without delivering software to production. I would work with the client to figure out what valuable features we could get out the door quickly. It isn't until the actual end users get to touch and feel and use the system that they figure out what works and what doesn't. This doesn't mean that we have to deploy the whole thing at once. I would figure out a way to make lots of small deliveries along the way, each time building on the last. As people see that working software really is being created, trust is established. There was no trust on that project, and this was one of the primary reasons for its failure.

I've seen a better way.

So, that's it. That's why I write. Because I've been very involved on both sides of the coin, and seen a better way of doing things. Here's the inverse of the Agile manifesto, and exactly the way we ran the project I've described here. It's a terrible way of delivering software, and a terrible way of working. It even sounds stupid, but it is exactly the way most software projects are run.

• Processes and Tools over individuals and interactions
• Comprehensive documentation over working software
• Contract negotiation over customer collaboration
• Following a plan over responding to change

I hope by writing these thoughts and experiences down that I will help others avoid these same mistakes. I view it as similar to contributing to the free and open source software community, except instead of software, these are my thoughts and experiences for free. Do watch out, because just like in free open source software, sometimes you get what you pay for. (;~)