Software design: underrated and not practiced
As a result of my long career in Information Technology (IT) I have made some important observations regarding software development. One of those observations is that quite a lot of software development teams, in companies both big and small, jump directly to the software implementation part of the Software Development Life Cycle (SDLC) skipping software design. These software teams may sometimes have a very quick talk about the system architecture accompanied by a quick drawing on a whiteboard. This drawing may stay on the board for a few weeks or longer but it is finally erased. Other times software design is skipped completely jumping directly to coding. Skipping software design is a serious neglect with negative consequences on the quality of the software system produced.
Early in my career, many years ago, during my first few days of employment as a software developer in a new company, they asked me to change part of an existing system. My immediate response was,‘Sure, in order to start can I see the design diagrams to get an idea about the system?’ I still remember the baffled look on my new colleague’s face, he stared at me for a few seconds not knowing what to reply, then he said, ‘What diagrams?!’
Me: ‘You know the software architecture and module design diagrams.’
Colleague: ‘Oh we don’t do those, we do Agile here, Agile doesn’t support documentation.’
Unfortunately this type or similar responses, are quite common excuses for avoiding software design (and code commenting or code documentation in general but I would like to discuss this in another blog article). Many software development teams erroneously think that if they are using a methodology based on Agile they should not be doing software design! This type of thinking is still prevalent despite the fact that nowadays there is even something called Agile Design [5]. I can imagine several reasons why these concepts about Agile exist. One reason is the fact that in its manifesto, Agile states that it values ‘Working software over comprehensive documentation [1].’ This is quite often wrongly translated as, Agile does not support documentation. Since the final product of design is a number of comprehensive documents and diagrams, then this is in turn also translated as no design as well. This is of course a misconception.
The Agile manifesto [3] was published in 2001 by people who worked during the previous decades. Back then most software were developed using the Waterfall model where a comprehensive and exhaustive set of documents (such as the Software Requirements Specification, architecture, design blueprints etc), had to first be prepared, agreed and signed before development could even begin. So the general idea behind the principle ‘Working software over comprehensive documentation’ is that instead of spending time on comprehensive documentation and then start development, it would be better if development teams prepare less documentation and start development as early as possible. The advantage of this method is that pretty early there is a working prototype which the team can show to the stakeholders for feedback. However, this does not mean no design and no documentation at all. Some design must still be done before starting.
I understand that typically programmers like coding more than writing documents. I know this because personally I also prefer it. That is why I became a programmer not a secretary, but we should not let our emotions affect our judgment. First, the Agile manifesto does not support zero documentation, and second, design is not just diagrams, blueprints and documents. A lot of thinking, discussion and decision making which will affect the quality of the product go into those documents and keeping them around also affects the software’s lifetime and maintainability. These in turn affect cost. Actually principle number nine of the manifesto states the following: ‘Continuous attention to technical excellence and good design enhances agility [2].’ So, the Agile manifesto actually emphasizes ‘good design’ and ‘working software’ but that does not necessarily mean completely excluding documentation and design.
Another excuse I heard is the following. ‘Agile emphasizes face-to-face conversation therefore, there is no need for diagrams and blueprints.’ Put differently, the architecture and design of the system is in our heads and we do not need to waste time writing it down, if we need to we communicate that information face-to-face because Agile emphasizes conversation.
This excuse may be inferred from an Agile principle which states, ‘The most efficient and effective method of conveying information to and within a development team is face-to-face conversation [2].’ When the Agile manifesto was published people communicated with memos, letters and regular landline telephones. In large bureaucratic organizations there was a need to record much of the communication between employees. Quite often employees had to type a memo or a letter when they communicated with another so that the conversation was recorded. This practice wasted time, and communicating with a document was not as efficient as communicating in person. This was the environment within which the writers of the Agile manifesto were working in and the context and meaning of the above mentioned principle. If a development team spends time thinking, discussing and deciding about software architecture, design and system structure then this should be documented for future reference. If this information is stored only in the brains of the various individuals who thought or discussed about it then what will happen if for example one of them is absent due to sickness or holidays? What if some of the original employees leave the company completely or new ones are hired? By studying the design documents new employees can immediately start working on the system. If there are no design documents then experienced programmers, if they are still around, would have to spend precious time training the new recruits. Having the design recorded and preserved has many advantages.
‘We prioritize fast delivery, we don’t have the time to produce documents and diagrams for software design’ is another excuse I heard, after all Agile supports, ‘Simplicity - the art of maximizing the amount of work not done - is essential.’ My response to this is simple. If you want to save time by skipping a step then skip something else not design. If you skip design it will cost you more in the long run, in both time and money. I cannot cover all the excuses I heard over the years for not doing design but I would like to mention this final one because it is one of the worst, ‘We are experienced we don’t need it.’ This probably relates to some of the Agile principles I mentioned earlier, and everything I said for the previous excuses also apply here, but I would also like to add something else. If someone or some group believe that they are so experienced that they do not need to do any design then obviously they are not experienced! In other words this is an oxymoron. Professionals who have experienced the existence and lack of design have learned to appreciate it.
As many (not all) of these excuses are based on Agile principles, it seems that Agile which is indeed a blessing to software development can also be used as an excuse for creating bad software or writing low quality code! Of course I have to point out that not all companies neglect software design. Some of them take design seriously practice it and are rewarded for it.
Programming by patching
The result of not doing any or very little software design is code which looks like patch work. The code does not really have a structure or plan. If it does have some sort of structure this may not be recorded anywhere, it could be instead spread in the minds of different individuals many of whom may have already left the company. If you want to change the software or fix a bug, it is like trying to reach a destination in a labyrinth without possessing its diagram. As you do not have a design it is very difficult to know where to add or change the code so you finally decide to add a patch somewhere without knowing if you will be affecting something else elsewhere. If you discover that something else has been affected, then you add another patch to fix that, but then something else is affected and so on. This is what I call ‘programming by patching.’
Systems developed without a design may be functional but they cost more. At the beginning progress seems to be fast because you skip steps and save some time, but as the program grows, it becomes harder and harder to proceed, debug, add new features or maintain. Because the system functions, management is unaware of the underneath mess and that they are employing twenty programmers to maintain it while five should really be enough. As the program grows the mess builds up and productivity goes even further down. Management needs to hire more and more staff. Because there is lack of a design it is very difficult for the new programmers to immediately start work on the system. Then the changes they introduce to fix something breaks something else creating a further mess. In the end the team informs management that it is impossible to work with such complicated code, that the system has (prematurely) reached the end of its life and it has to be refactored. This of course implies further costs.
This problem is described eloquently by Martin C. Robert in his famous book, Clean Code:
‘Over the span of a year or two, teams that were moving very fast at the beginning of a project can find themselves moving at a snail’s pace. Every change they make to the code breaks two or three other parts of the code. No change is trivial. Every addition or modification to the system requires that the tangles, twists, and knots be “understood” so that more tangles, twists, and knots can be added. Over time the mess becomes so big and so deep and so tall, they can not clean it up. There is no way at all.
Eventually the team rebels. They inform management that they cannot continue to develop in this odious code base. They demand a redesign.
A new tiger team is selected. Everyone wants to be on this team because it’s a greenfield project. They get to start over and create something truly beautiful. But only the best and brightest are chosen for the tiger team. Everyone else must continue to maintain the current system.
Now the two teams are in a race. The tiger team must build a new system that does everything that the old system does. Not only that, they have to keep up with the changes that are continuously being made to the old system. Management will not replace the old system until the new system can do everything that the old system does.
This race can go on for a very long time. I’ve seen it take 10 years. And by the time it’s done, the original members of the tiger team are long gone, and the current members are demanding that the new system be redesigned because it’s such a mess.
If you have experienced even one small part of the story I just told, then you already know that spending time keeping your code clean is not just cost effective; it’s a matter of professional survival [4].’
As we can see there are plenty of good reasons for software development teams to spend some time designing their systems. The advantages of design are documented in thousands of information technology books and articles. The most obvious are that the developed system will be flexible and easily maintainable. A designed system is flexible because it is easy to change. If you change something here then you know it will affect something else down there. System quality goes up because your code is structured, organized, clear, easy to understand, therefore you can find bugs easy. Security is improved. Having a design helps you express and explain ideas about the system to your developers, stakeholders and clients. It makes your software easier to implement and easier to expand. This reduces the need for major changes later on as mistakes are discovered early. All these factors lower cost.
A documented design has many employee related benefits. Having a design translates to lower cost because a company needs fewer programmers to develop and maintain the system. Without a design an organization depends on the original programmers who analysed and developed it. Since the design is not documented and they are the only ones who know it, if some of them leave, a part of the design leaves with them. It is very difficult to replace such programmers therefore they are usually paid higher so that they are not tempted to leave. This is an additional cost. But even if the original programmers remain in the company they still have to go on holidays or get sick and so be absent for a number of weeks every year. However, if you have the design written down then employee absence or turnover is not important. Even a new programmer can consult the documents and diagrams and immediately take over. Without a design it would take months for a new programmer to produce useful work. It would also incur and additional cost in work hours for the experienced coders who will have to train them.
When a company wants to hire new programmers, I often hear management say that they are looking for experienced people so that they can immediately produce work. This idea is wrong. If the company’s system lacks design it is hard even for experienced programmers to immediately produce useful work. If they do start work on a system with no design then they will be doing ‘programming by patching’. They will be patching an existing mess making it an even bigger mess! Alternatively, if the company has a documented design, even new programmers can immediately produce useful work. As already mentioned, design does not only affect cost but also quality. A software system is like an engine, it must be designed, and the design must be documented for future reference. Like there is a mechanical engineering discipline which dictates how an engine is build, there is also a software engineering discipline to guide us on how to build quality software systems. Experienced programmers alone cannot produce quality software. The best software are created because an organization has a specific development culture which places emphasis on quality and good design.
[1] https://agilemanifesto.org
[2] https://agilemanifesto.org/principles.html
[3] Kent Beck; James Grenning; Robert C. Martin; Mike Beedle; Jim Highsmith; Steve Mellor; Arie van Bennekum; Andrew Hunt; Ken Schwaber; Alistair Cockburn; Ron Jeffries; Jeff Sutherland; Ward Cunningham; Jon Kern; Dave Thomas; Martin Fowler; Brian Marick (2001). "Manifesto for Agile Software Development"
[4] Martin C. R., “Clean Code: A Handbook of Agile Software Craftmanship.” Pearson Education, Inc., Boston MA, 1996 pp. 4-5
[5] http://agilemodeling.com/essays/agileDesign.htm
Author: Dr. Sotiris Fanou