How to “play the game” of software development in a team fashion is not a physics or math based law. Given we have humans of varying skills and degrees of discipline working together we get highly variable outcomes, so it is unlikely we can find some single way of doing things. Giving up given such variability is unacceptable. We apply software engineering principles and processes as a way to increase the probability of a positive outcome (See Thinking in Bets). Such models have evolved over the years and it is my strong belief that we should know all the broad approaches and when to employ one or the other as opposed to rigidly embracing whatever is currently fashionable at our organization or within the broader industry.
The included activities in the lifecycle of building software do not vary and include:
Requirements
Design
Development
Validation (aka Testing)
Deployment
Operations
Various continuous activities in management also tend to be ongoing and lead to an evolutionary thought about software. In other words software grows like a garden as opposed to is built like a building.
Top-Down Design - start with high level and break down towards small pieces. The common reductionist style
Bottom-Up Design - using interactions, data, etc. to grow upwards
Middle-Out - a surprising concept that has us trying things sometimes completely out of order and still finding value via exploration
Generally I think you need to move between all the techniques as appropriate.
An easy way to recall that realism of all approaches being useful might be to think of writing a book. Start with concept and outline - top-down. Write chapters and realize outline needs revision - bottom up. Get stuck and try sections or examples out of order - middle out
Incremental solutions focus on doing pieces at a time
An all at once solution suggests we do the whole thing in one go and release
Generally incremental solutions provide more safety (check as we d, but truthfully a system often isn’t viable until many increments are done. Think a Jet Plane a tail, or cockpit might be an increment but we need many increments before we have something workable
Incrementalism tends to associate with iterative processes since we often do something release, get feedback and adjust.
Incremental approaches on the surface feel much safer because you can see progress and get feedback as opposed to doing something for a really long time and release all at once.
My general experience with this is somewhat varied. Start-up companies very much find this approach useful, but some types of software it doesn’t make much sense at it is too risky. (Think space probe software)
Metaphors from little to big don’t seem to hold up and often there are large scale changes. Recall the caterpillar to butterfly idea or skateboard to car evolution. Incrementalism is rarely as neat and tidy and risk free as many people advocate
One problem solving approach is more of a step at a time to completion linearly approaching some done state.
This linear approach eventually can be boiled down to a check list, recipe, dare I say algorithm once we understand a problem well enough. The problem is that such problems can and should be automated.
More interesting problems probably can’t be linearly procedurized at least not at first so we may need to cycle over the problem many times. An iterative approach has us creating a little bit of the code at a time evolving it if you like.
Many people focus on the idea of evolving or growing software as opposed to assembling software.
I actually am fine with both as I think iteration is very much part of design engineering where a linear is very much a process of production engineering. Software contains both of these forms of engineering so I prefer more of a this AND that as opposed to this OR that.
The Iron Triangle represents what most consider a fairly inflexible law of project management especially software project management. The graphic below shows a representation of the triangle with a sense one can pull on each of the handles
The idea here is that if you pull COST to be low it reduces quality. You pull scope it probably effects cost (higher) and schedule (longer) and may reduce quality.
For me I think the key points is trying to fix from 1 - 3 of these so you can then accurately focus on quality. If cost is set then it limits scope and schedule to what can reasonably be done with a defined quality. If schedule is set same thing. If scope is set same thing. However, when you set more than 1 it gets us to a place where you might have fancy features (big scope), but low budge (too little cost), or no time left because of procrastination (poor schedule)
My best advice is not focus on the pull points, but focus on the acceptable quality. Without defining quality we are somewhat lost unfortunately.
TL;DR - fix one or more of these and assume that if you pull you give things up.
The iron triangle reminds us that a key aspect of engineering is embracing trade-offs. We must admit we can’t have it all and we try to balance our requirements with our situation as best we can to make a “good enough” solution as opposed to some perfect solution.
Consider even if you can make a perfect solution, it likely will only be perfect under a particular context and for a particular period of time. If context changes open what is a “perfect” solution changes.
Common Trade-off tensions include:
performance vs security - high performance, lower security vs higher security vs lower performance
security vs usability / ux - more security tends to degrade ux and inversely ease of use might sacrifice security.
Acknowledge that the market does effect what we do as SEs. It is a common topic in the course, but we see in the process model space consultants pitching classes and books to take their waterfall+ or agile’ twist and sell you on that it solves everything. This is not an unusual concept and likely anything sold has that to some degree, but why this is a point to consider with process models is that software projects are tough and risky so management can and does fall for fashionable solutions. When you encounter such things you need to work at how to describe and explore these fashionable solutions to see if there are novel or just trendy.
A key aspect of process models that management in particular cares about is being able to estimate when a projects will be finished. Management would want some certainty like what this shows. In a school environment that indeed might be something like a set deadline. Companies can do this as well. However, to meet that deadline we likely may need to sacrifice quality and features which then is likely reflected in our grade or industry lower user-acceptance or even profit.
The truth of estimating and completion when you don’t have a fixed date is that things tend to shift to hit the desired quality mark. This chart shows the more likely reality and your goal is to be closed to the dotted line with the quality required.
There are many examples of people being the #1 force in Software Engineering from focusing on individual abilities, the role of the 10x developer, group dynamics, decision making, user centered thinking, requirements gathering, user acceptance testing, and more people are at the center of nearly every aspect of SE.
The big challenge with people is that they frankly introduce a form of variability that is simply quite difficult to account for. The certainty of the programming activity for example is made hard by the uncertainty of your fellow humans in the form of your stake holders (product owners), team members, or end users.
The fact that people dominate SE is not really that controversial and numerous books and works have validated this idea. However, we note in the discussion this truism because as we get more in process and eventually tools and code we want things to be more predictable and less fuzzy, but that sadly isn’t very true.
The cone of uncertainty is a useful way to remind ourselves that when starting something we are mostly uncertain and the longer time or examination continues especially as we build software the more certain and specific we get.
The process models and estimation aspects of SE suggest a certain amount of prediction. Prediction of the future is of course not a science. There are various heuristics including the obvious observing past behavior to estimate future outcomes, but given the variability of the world this will not always work. A good way to address this is to employ a thinking in bets type strategy.
The basic idea of thinking in bets is instead of 1 / 0 certainty of pass or fail you adopt more a percentage outcome of pass fail like 86%/14% or something. Thinking of poker hands where you have a high chance to win but no guarantee is a good way to recall how to employ this thinking.
The spectrum process runs from pre-planned very formal linear processes to ad hoc and near choatic processes like “hacking” or “cowboy coding”. Recall that DUF = Design Up Front and this just showing how much work we do before we start the development.
A key point of the spectrum is to understand that the need to employ one process or another might depend more on the risk appetite, project type, time availability, team maturity, organization needs, and such as opposed to one particular right or wrong way. The diagram here is an alternative representation of this spectrum with some region markers and examples to make more sense of the thought.
The idea of the waterfall as a process model is to break up a software development projects into a set of steps running through designing, planning, executing, verifying and maintaining. The are various versions of this model. Some have 5 steps, some 7, some 8, etc. The names also vary. This one is a simple one we can use for understanding purposes
For testing purposes I’ll rely on on my own variations which adds a few modifications to the standard waterfall thinking. First, it breaks up the design process into more steps. Second it interlaces things like testing into development in particular unit testing as it should come during that process as opposed to after, and finally it shows arrows up suggesting a return to previous steps if warranted.
I generally prefer a modified waterfall where I have added a design whirlpool effort before entering the build process. I have found this a good way to launch projects with less risk in practice. Once done though we tend to fall into a spiral or agile style for further iteration work.
Waterfall approaches do not handle change well, often do not get user feedback early and often enough without modification (ex. whirlpool idea), and frankly may delay progress or gratification in a world that demands to see that. Agile approaches such as Kanban, XP, etc. attempt to change this with more of an iterative approach which focuses on breaking the project into mini work efforts typically called sprints and growing your software incrementally with constant feedback.
Like waterfall/linear approaches there are also many versions of the iterative approach as shown in this graphic.
The Spiral Model for Boehm is one of the earliest well known iterative approach. Note that there are the waterfall style steps (recall these are just the SE activities) performed over and over again.
Depending on your degree of formality this ranges from very systematic to nearly ad hoc. Consider for example the “Sad JAD” (Joint Application Design) idea which one could say is iterative. This is the idea of getting vague requirements from a customer. Building and then having the customer react and repeating. This is Agile like but is incredibly inefficient and lacking in guardrails like estimates (story points), defined work units (sprints), user centeredness (user stories), and so on.
Our main focus here is simply to understand the range of iterative approaches which range from more linear/waterfall like to more ad hoc like. An up close look at Agile in our particular class varietal which attempts to reach some happy middle ground is in a separated deck and document.
A major point of the process model overview deck is to acknowledge there is a possibility of using an over engineered solution for a particular job (formal model for hello world) and under engineered or dangerous solution for another job (cowboy hacking or rapid iteration for a life and limb application like a heart monitor or autonomous driving system).
Developing a pragmatic attitude is a key aspect of a mature software engineer. It is unlikely we can come up with some one true way to build software for all situations. However, it is quite likely that thru experience and holding a pragmatic attitude you may find your way to adopting the appropriate solution to the team, situation and job you face.
If any one thing is learned about the process model it is that the how (process model) is only about supporting the creation of the what (software). If the what can be properly created meeting the organization constraints (cost, time, etc.) and quality goals than that is an appropriate solution.
Some final points about process models is that a group of humans creating something using a methodology is 100% not unique to software and you are encouraged to look at other group activities from a kitchen at a restaurant, an operating room, a flight deck, a war room, a soccer pitch, a movie production, a magazine or newspaper issue being made, a construction site, <insert dozens of others> for ideas. The idea that what we are doing is so utterly unique in my view is a bit arrogant and restrictive in terms of our growth as an industry. We can should learn from others.
Even if we stay insular to our space many ideas are long stated but not truly admitted to our adhered to, for example
“Knowing what to build is typically harder than how to build it” (Paraphrased)
“Requirements and ideas are not always clear to people and extracting them takes time and iteration” (Paraphrased and related to “Users/Customers aren’t designers”)
“Grow, don’t build software”
All these quotes are 50 years old and from titans of SE.
TL;DR - we’ve known many of the problems with process for a long time, but struggle as an industry to get consensus.
The point of a process model is to add organization to our problem solving in software engineering. However, problem solving is not unique to the development of software. We started first with understanding generalized thinking top down, bottom up, middle out as well as incremental and iterative concepts. Hopefully we have begun to adopt an engineering mindset of pragmatism. A mindset that accepts trade-offs and understands the iron triangle law. Our mindset focuses less on perfect eternal solutions and more on situational appropriate solutions. The concept of risk mitigation, appropriate model for situation and thinking in bets were different ways to express that type of situational pragmatism. Finally we should be very concerned about one true way and understand that from waterfall to agile to full on hacking we kind of need all of the approaches at our disposal. We must be quite careful to not believe that our form of problem solving is so utterly unique, because at the end of the day it is just a bunch of humans working to solve problems. We SEs do need to find both what is different with software, but also understand what is the same otherwise we miss easy improvements in our process of building things people actually want.
Process models are really more responses to conditions, at least they should be, than things that live outside of any context. Waterfall was a product of how software was made, who funded it, and its costs. (Bigger corporate, government, etc.). Agile arose out a different period that favored a “first mover” idea and lack the previously required formality. Clearly Agile wasn’t dreamed up in some Platonic perfect way is to deny the dotCom context. Context is important and its is likely as conditions change via market, regulation, social expectations, the rise of AI, etc. process models will likely respond and react.
TL;DR - these are often lagging not leading ideas.