By Shirly Ronen-Harel
Shirly Ronen-Harel is a coach and consultant in Agile / Lean methods. She is also author of ‘The coaching Booster Book , and the ‘Agile Kids’ books and currently writing her new book.
What does DevOps have to do with the software testing mindset?
If you ask people from the industry what DevOps is, you get many different and interesting answers. And no wonder this is the case, since DevOps is still a relatively young approach (2009), born out of the painful need to deliver fast, high-quality product in a complex and rapidly changing environment. Personally, I have adopted the standpoint that sees DevOps as an integrated approach, combining culture and tools aimed to get things done, within complex fast moving environments, with an emphasis on culture and mindset.
The software testing practices and mindset have radically changed since the early days of Agile and Lean. Today there is no doubt that the software testing practices and mindset are an inseparable part of DevOps culture. There are people who will argue that there is no difference between the Agile software testing mindset and the DevOps culture mindset, and to an extent I can agree; nevertheless, in this post I have decided to put a special spotlight on the software testing mindset within the DevOps world, to try and understand this mindset, see where these practices came from, how they have developed and how it is all connected to a single culture.
So I will divide my thoughts into 3 parts
Part 1 : Why culture is part of DevOps
Part2 – What does testing have to do with it and how has the testing mindset evolved
Part3 – What are the keys to an agile DevOps testing mindset?
So let’s get started.
Part 1 : Why culture is part of DevOps
‘Dev ops’, ‘Mindset’, ‘Software Testing’ – each of these three terms has an entire world of practice and culture relating to it. Before taking them all three together as one topic (‘DevOps in the software testing mindset”), I would like to explain each one separately, giving each its respective place.
Let’s start with the mindset.
What is a mindset?
Please take few minutes to watch this brilliant video, “Always #LikeAGirl”.
“In my work as a documentarian, I have witnessed the confidence crisis among girls and the negative impact of stereotypes first-hand,” said Lauren Greenfield, filmmaker and director of the #LikeAGirl video. “When the words ‘like a girl’ are used to mean something bad, it is profoundly disempowering. I am proud to partner with Always to shed light on how this simple phrase can have a significant and long-lasting impact on girls and women. I am excited to be a part of the movement to redefine ‘like a girl’ into a positive affirmation.”
This video illustrates how one’s mindset and preexisting set of assumptions can change behavior, self-esteem, the way we act and what we do.
From Wikipedia : Mindset is defined in professional literature as
…a set of assumptions, methods, or notations held by one or more people or groups of people that is so established that it creates a powerful incentive within these people or groups to continue to adopt or accept prior behaviors, choices, or tools . (http://en.wikipedia.org/wiki/Mindset)
This is a mindset. A mindset that will allow certain behaviors to perpetuate, a mindset that will lead to certain types of legislation, maybe even to violence or death, it will determine the way in which this society treats woman and human rights. It determines the way people explain behavior or treat rape. Just by reading the comments attached to this article one can witness this mindset and realize just how far-fetched it seems when examined from another culture’s mindset.
The preexisting assumptions and beliefs that are held by developers, testers, analysts, product managers or anyone else involved in producing working software will dictate the way they believe they should or should not work together. It will ”dictate” the tools they will choose to use as well as their team members, methods and procedures.
In a world of rapidly changing environments, where DevOps is a necessity, we will need to create the right mindset to enable DevOps to answer these new environments’ needs: producing high-quality product fast.
But what is DevOps?
My friend Thomas Cagley @TCagley in his article DevOps Primer: Definition
States that If you ask 20 people the definition of DevOps you will get 12 definitions and 10 people that don’t know the term DevOps.
He then quotes: “David Herron, provided a simple definition, “(DevOps is)…an integrated approach to software delivery that include process, tools, technology, resources.””
Why is that?
In a fast moving environment, where the customer has very little patience to wait but quality is not negotiable, organizations needs to find the way to move from A (the point of the idea) to B (the point of delivery) as fast as they can with the highest quality. Satisfying customers and at the same time reacting to rapid new demands at the right speed, quality and with grace. It’s no simple feat to accomplish.
Can we do it? Can we release changes to customers every month? Week? Maybe every day? How can we make sure that quality will not be compromised and that the changes are well integrated?
First we need to remove some barriers. The barriers between Dev (that wants to change the code, add to it) and Ops (that wants to keep everything stable). We need to break and adjust the boundaries between the developers and the operations people, we want them to work together. We want to deliver something with good quality, fast time to production and at the same time create trust between those groups.
So we can add tools. We can add them everywhere, making sure things are not breaking down on the way to production. Tools will surely do a great deal of work in building awareness to quality. We can create continuous integration for every line of code, continuous delivery for any functionality added and tested, health monitoring, better logs and more.
But I’m afraid tools are not enough. They are great but they are not enough. Why?
Because we need to know where to employ these tools. What will be a valuable application of these tools? How do we determine this? We need to know our production line. We need to identify production line bottlenecks, to understand the process and set the right process. Imagine we are a chocolate factory, we can add cocoa but still not deliver chocolate because the mixing machine is not working properly.
And tools, processes and barriers are only a small part of it, why? Because we need the people to be part of it. And they need to have the right mindset to be able to deliver the right feedback.
Why? Because in a rapidly changing environment we need the people to give us feedback from every part of the production line at any time needed. In this complex environment, things may come up, change and develop differently all throughout the production line. Therefore we need the people to be aware, to raise flags. We need the problems to be identified fast, as well as the solutions. We can only do so when we have the right mindset. Only when they are on board, know what they need to do and why. In this complex environment we need all the different professional perspectives to be available to us for fast decision making. We need to know where to use our tools according to accurate feedback and recognized bottlenecks, what to change and why. Without the right people and mindset we will be limited in our ability to do so. In order to reach this state all these people need to work together to create this culture of collaboration for delivering high value fast. Without the right mindset we may still succeed, but we still be generating much waste in the process and reach our goals slower then we could have.
In some extreme cases (and I have seen such) the entire organization will be stuck due to lack of collaboration, to the point of not being able to deliver.
So what is DevOps?
“DevOps means that there are no walls, no gates, no transitions, and no ceremony between Development and Operations. They are seamlessly integrated (when viewed from ’above‘) into a single, value delivering, IT entity.”
“(DevOps is)…an integrated approach to software delivery that include process, tools, technology, resources.”
DevOps includes rapid collaborative development without hassles, continuous testing, continuous deployment and continuous monitoring and it can’t work without a proper mindset in place, one that aims to allow everyone to collaborate to deliver value faster.
Part 2: What does testing have to do with it and how has the testing mindset evolved?
In part one I covered the basic terms of DevOps and the related mindset. The need to react to a complex, fast-paced and ever changing environment, make ongoing changes to high-quality working software, created the DevOps culture and framework that enables all of this to happen. I also discussed the need to establish the right mindset or culture, regardless of which tools or processes we use in order to achieve the goal of delivering quality working software in a complex, rapidly changing environment.
In part two I will cover the changes that software testing went through during the last few years, which have led to the rise of an Agile testing mindset and up to the point of a DevOps culture.
DevOps aims to break the barriers between Dev and Ops. This also necessitates breaking the barriers between Dev, Test and Ops. Nothing stays the same for a traditional testing operation.
Testing operations undergo a major change in a DevOps culture. The traditional testing methods, tools, approaches and mindset will not do. Automation testing is the glue where all this rapid delivery of value can exist. Automation testing becomes a critical and major component of being able to deliver rapid value and to react to changes. Therefore everything changes for testing. The tester’s skillset changes, methods change, the way we test, what we test, when we test and the way we plan and execute, all change completely.
**Right before I start covering some of the testing practices applied throughout history, please keep in mind that thousands of pages were written on the subject and I probably gave more attention to some things over others. Please forgive me for that and feel free to add comments, points of view or landmarks.
Agile testing procedures and mindset were starting to take shape even before the term DevOps was coined. The traditional approach to testing had to remove a few layers and change some other layers in the procedures, while adopting a different type of mindset. This new mindset was particularly crucial, both for testers, developers and product owners, as well as executives and all others involved. It could not work without it.
Being (and using) Agile was overwhelming to testers and developers and in order to achieve the correct outcome, i.e. frequent working software, new practices and skills were developed by emerging professionals in this field, in addition to a shifting mindset that was crucial in order to produce working software in a rapidly changing environment.
Let’s look at this shift that testing operations went through in the last few years, considering methods, tools, and mindset, which has brought us here – to the exact point of what we call a DevOps culture.
In short, looking back at the traditional testing approach we can immediately identify its characteristics.
- It was mostly a step in the process. This step was implemented towards the end of the process
- The notion of separating between development, testing and product activities was prevalent.
- The notion that testing is meant to control the quality of the product, detect defects and prevent flaws from appearing in the end product
The problem with this was that the quality-related feedback came too late – at the end of the process.
- Documentation was an essential part of every testing operation, something testers would refuse to work without it. Why? One reason is because in the traditional waterfall approach (in the many shapes and forms it can take) we mainly experienced the gap between requirement-design-coding and testing. This was a relatively big gap (considering that today a one day gap is too long, imagine what it was like when having gaps that lasted for weeks or month was the default state) , and when changes were made, naturally before going to testing they could not relay on anything but good documentation. Everything testing did at the beginning was to review what had been changed from the requirements to the time of testing. This was a hugely time consuming and wasteful, but inevitable operation. We had to do it, we were located at the end of the process, accountable for every bug found, and how could we even think not to compare documentation at the end?
Direct communication and planning instead of big documentation and plans were not a common or regular process or mindset in the development production line.
- We usually didn’t have time to test. It was always too late, and we always had to compromise on quality in order to be able to keep the timelines.
- We spent time and resources on large-scale meetings, trying to analyze huge amount of defects and move them to the next big release, thus contributing to continuously low quality.
- Developers were passing pieces of code marked as done in big chunks.
Everyone working in testing knew that “90% of development effort done” meant nothing about the project’s actual state – it could all revert to 0% progress once the product began being tested.
- Automation depended on heavy tools, usually using GUI (the top-down method of testing). The prevalent approach was using GUI automation but it was the hardest component to test, the most expensive, and the most unstable component with the latest feedback possible. Analyzing and engineering automated tests in the top-down method, from the GUI to particular scenarios, trying to cover as much as we can in the process, was tedious, expensive and difficult. And everything was done “after” we really needed the feedback – at the end – making it too late and too expensive.
- Many siloes, handoffs, documentation to follow those handoffs.
- Moving information back and forth between siloes created a major setback in project schedule. Discoveries made late in the development cycles often required changes to code, requirements or design. Problems that were identified in late testing stages changed everything. Unexpected requirements or changes also needed to be integrated in the process and caused changes that were discovered late during testing, imposing a heavy practical and emotional burden in dealing with changing plans, managing defects and more…
But this was how everyone did it. To the majority of software practitioners it didn’t seem wrong.
But the fact of the matter was that 90% of the software projects failed (low quality, delivery dates missed, different content, over budget and more…) Things started to change once customers began refusing to wait, once quality was something customers refuses to miss, once the entire environment started to demand things faster (otherwise we would not survive as a product), once changes needed to be integrated into software faster than the organization could normally react while maintaining high quality and addressing rapid customer demands.
What was wrong with everything we had done that far?
Too slow, too late feedback, too expensive, less effective than it could be.
So what did we try to do?
Even before Agile, Scrum or any other tools started to emerge; some practices for solving those problems began gaining traction. And how couldn’t they? It was a necessity.
The Agile methods that were being introduced were helpful, but were not that easy to implement and were definitely not a ‘silver bullet’ solution for coping with the complex, rapidly changing environment. Amongst them the most popular tool was Scrum (although there were additional methods that adopted a somewhat similar mindset in order to make it work). We adopted Scrum in an attempt to shorten delivery time, while in the process introducing the testers to the Scrum team daily activity schedule.
If we wanted to move fast and produce small chunks of working software, we had to shorten the cycles and integrate the testers the Scrum team methodology. It was not easy for anyone. The mindset needed to change as well as the practices.
Testers where overwhelmed by the fact the developers will be accountable for quality, that developers will ‘manage’ testing. How will they work? What about regression? What about defects? Who will make the decision on quality and many more… We needed to jumpstart the mindset of working together and the shared accountability for quality, as well as some additional attitudes which I will detail in part 3.
At that stage we needed to create a mindset of collaboration, of mutual understanding, of shared ownership and shared accountability, in regards quality and more…
But this was not enough: it’s not just the testers who were introduced to the Scrum team, it’s the testing activities that were introduced to the Scrum team.
What does this mean?
The goal was that development managers would be accountable for E2E delivery, having all the tools needed at their disposal = developers and testers as one whole, complete group.
It means that a Scrum team is accountable E2E to deliver quality working software, including all the tasks needed for this to happen. Including the testing. It’s a mindset of accountability for quality. Testing became a task rather than a person’s job description.
We also started examining the process and in order to move faster we tried to reduce critical destructive siloes which created big risks for the entire software release and necessitated time-consuming, wasteful efforts to be made.
What were they?
There were two points of friction
Both points where referred to as “integration \ deployment hell” why? Because a huge amount of defects, inconsistencies, and changes were introduced in these stages, forcing the entire production line to reopen a lot of code, and backtrack to make fixes and changes. This was a hellish period of uncertainty and not knowing when and how all of this will actually be done.
Overcoming this obstacle began with including integration and deployment tasks earlier in the process of software development: customer data tests, smoke tests, continuous integration along with testing and more…
At those times you would hear the developers complaining that it is impossible to create working software fast, especially at the end of a sprint, without automation. It was also hard to include integration and deployment early in the process without proper tools or proper automation (the latter, when implemented at the end of the entire development cycle, was ineffective for this purpose).
And at that time, they were right. But automation techniques and practices had already started to change to accommodate this new and rapidly evolving environment.
Agile automation testing (which we might refer to as the ‘glue’ of modern DevOps) introduced a completely opposite approach to testing compared to the traditional one. It means start testing from the first line of code, working your way up in the testing pyramid while continuously integrating, delivering and deploying working software, constantly dealing with defects, getting early feedback and keeping quality high at all times.
The big change came from early feedback on the software’s quality, the ability to fix early, change fast; to produce a safety net where developers can fail early and fix fast. Testing is everywhere. Instead of detecting defects, providing early feedback on the software’s quality.
It moved testing to the early stages of development.
If you wish to read more about the testing pyramid you can do so here: http://blog.goneopen.com/2010/08/test-automation-pyramid-review/
Best practices for enabling this automation to operate within an Agile environment were also developed. This enabled continuous testing, feedback and highly effective and fast responsiveness along with quality of delivery.
I will not go too deeply into this, but you can read more here if you like:
But all of these wonderful theories and tools could not work without a proper mindset: a culture of collaboration, trust and respect. It does not matter how well you define the process, the responsibilities, the tools, it’s the mindset that will make the difference.
You cannot test early if developers are not tuned to quality. You cannot get or react to early feedback if all testers want is to detect defects instead of preventing them. You cannot move fast if a developer does not understand the need to deliver working software in small chunks while maintaining direct communication with the tester. If they don’t understand the WHY, they will not become hooked to the practices. These are just few examples that demonstrate why the mindset needed to change in order to make this happen.
Let’s proceed to look at the Agile testing and DevOps mindset in part 3:
Part 3: What are the keys to an agile DevOps testing mindset?
In part one I covered the basic terms of DevOps and the proper mindset. The need to react to a fast and ever-changing environment, deliver ongoing changes to high quality working software led to the DevOps culture and framework that enables all of this to happen. I also discussed the crucial need for establishing the right mindset or culture, no matter what tool or processes we employ, in order to to achieve the goals of delivering quality working software in a complex and rapidly changing environment.
In part two I covered the changes that software testing went through during the last few years, which have led to the rise of the Agile testing mindset and to the development of DevOps culture.
In part three – I will take a look at the actual mindset and culture we need to implement and cover real-life practices that will promote Agile testing and DevOps culture for delivering quality software, fast.
Well, having good tools, “good” processes and well-defined responsibilities are just one part of it. If we want to achieve effective testing in a DevOps world we need to establish the right mindset.
There are actually three basic things we want to achieve.
- Remove barriers between silos
- More quality
- Fast progress and reaction
Those goals needs to be broken down into some practical instructions to become actionable. It will also not be enough to merely “say” these messages, we will need to follow them in practice and make sure to connect the dots whenever we encounter behavior that contradicts or fits the mindset we wish to achieve.
We wish to have a culture of:
- No walls, no gates, no transitions …
- Trust and collaboration
- Early feedback
- Continuous improvement
Let’s break some of these elements down into more practical instructions and take a closer look at the bricks that hold this entire culture together, including:
Collaboration is a huge topic. Some of its meaning and significance is also relevant for other components of the mindset that appear in this article – after all, it’s all about collaboration.
When we want collaboration we need to mean it. First we need to say the word itself clearly; COLLABORATION! We need every one to hear it and to act upon it.
Collaboration is a big word, it needs to be taken apart to actual behaviors, e.g.:
We want people to communicate directly. We want them to sit and talk. We want them to approach each other when they have questions and we want them to be pleasant and approachable themselves. And it should be OK. No one should get angry that a tester interrupted a developer with a question.
The organization must supply the right infrastructure for collaboration: encourage daily meetings, create spaces for communication, promote pre-planned meetings, planning ahead, and retrospective examination. Everything that brings people together to collaborate on the same tasks.
There are so many methods today that offer effective techniques for better collaboration.
For example: sitting together makes it easier for teams to collaborate. Placing developers, testers and others close to each other will make it easier. When a tester has a problem he can reach out to a developer, just by asking. Being part of the same team, sharing the same tasks and goals makes it easy to communicate on a relevant level without bothering the other,
Direct communication, sharing knowledge leads to faster production as well as a better understanding of what we need to move faster (tools, skills)
I hold the managers accountable and responsible for making this happen. They are trend-setters. They should lead, first by personal example and also by guiding, encouraging, and teaching collaboration. It is unacceptable for an R&D manager to shout at Support staff for failing to handle defects. He must understand that the appropriate point of view is to focus on how we can help Support understand the customer’s defects and take it from there to prevention. Maybe we can add better log analysis, maybe better monitoring or other related tools. He should also encourage direct communication and understanding between these two departments.
I can create collaborative tools such as monitoring and effective data visibility, but the tools are always second to the understanding and the actual practice of joint collaboration.
DevOps means that we are all collaborating together – PO, Dev, Tester, analyst, architect… There is no ego.
Imagine a Formula 1 team– without full collaboration and understanding no one can win the race and no one can get better at it. (I will return to F1 later on)
I was once working in a company that was located in an industrial area. We were always on the hunt for a good place to get a fast meal during our short lunch break. Our offices were surrounded by restaurants whose entire business targeted the few hours each day between 11:00-14:00, when they would feed the hungry tech workers. These places needed to be very efficient and be able to serve a lot of people, and as many as they could in a relatively short period of time. This was their bread and butter and they had to do it fast, provide high quality food and feed as many people as they could.
One of these places was so efficient at this, that when you entered the place in took only 1.5 min to be seated and served. Every 10 seconds a person was out the door.
You started the line by choosing which food you wanted. There was a running production line of two people who filled your plates. . From there you proceeded to the registry, where a lady would take your card and make the bill while offering quick packaging and drinks.
This was a very efficient production line.
The problem began when a customer who had no corporate meal card or was not familiar with the flow came in. This customer would keep the cashier busy by asking questions, paying in cash, etc. Treys of food started piling up near the cashier. As a customer it was very irritating to wait, all I wanted is for her to take my card, close the bill and let me get out of there.
But they were good, so good (Kanban-good) that when they noticed a 3-tray queue forming at the cashier’s, one of the plate-fillers would leave his station and efficiently handle the waiting customers, help the cashier and get back to his station once the pressure resided.
In addition to an effective understanding of flow, this is a good example of collaboration. Similarly, when we have a bottleneck, a developer can switch for a while and test the application just because it will get us faster to delivery.
- Testing is everywhere
Instead of being a step at the end of a development cycle, testing becomes something that happens everywhere. Everyone needs to understand that we are no longer compartmentalized.
What does ’everywhere’ mean in this context?
It means that it starts from the first line of code to the smallest functionality, through integration, regression and also in production. It never stops.
In order to do it everywhere, we need skilled testers who know their automation. Testers can no longer be unfamiliar with technical issues. They are collocated with R&D, sharing the same knowledge and goal to get things done.
But it’s not just that, it’s also means that if we want testing to be everywhere we need testing to be done by everyone.
The tester cannot be the one who writes every test, sometimes it’s easier and cheaper for a developer to do it. A developer that refuses to write tests is a problem. We will create the right tools to do so, but the understanding that everyone writes tests everywhere is a message that is crucial for continuous integration, delivery and deployment.
The mindset is that I am not waiting for someone else to test. If needed I will do it myself.
- Build something that has the quality of our brand written all over it
Quality is non-negotiable.
It’s not something that the tester owns or protects with his life. If we mess up with quality, we can’t blame the tester.
Everyone is accountable for quality and everyone needs to do everything to maintain it.
Statements like ” we should expect a massive amount of defects since we are a startup and that’s just how these things go” are unacceptable. We need to find the way and the tools to prevent this.
You cannot write untested code. Having problems testing it? Let us know. We will create the right tools but the information and flags needs to be in everyone’s minds.
No more “I am a developer, I am not doing a tester’s job” – you are doing a developer’s job by making sure nothing that is of low quality leaves your workstation.
- Start thinking of QA as an enabler rather than a gatekeeper
Instead of a gatekeeper holding things back from going to production, the tester becomes an enabler, helping good code rapidly flow into production. He has his hands all over the entire flow. Engaging with developers as well as with the Ops teams, perhaps performing smoke tests in the production environment, helping to identify monitoring needs, identifying more precise needs and better techniques for automation – everything that enables a better and faster flow to production.
We must banish the assumption that a tester should not be involved in production. Testers should understand the business aspects as well as the technical and product aspects, and should use this knowledge to enable afast and high quality flow to production.
- Always-working software
Working software is one of our primary goals. If it doesn’t work, it’s worthless. We should all strive to create working software, not only testers.
Working software means coding + testing.
In every point of the development flow, the software should be in a “working” state. From the first line of code. This is a mindset that is hard to maintain. We will build the tools that enable the developer to check their code at any time to make sure it didn’t break anything.
The business should have the ability to change whenever it desires, to access the code at any time, change it and fix it to address a customer issue and quickly move on to the next assignment without big regression cycles and rollbacks, and without throwing away valuable code that still didn’t reach “working status” and might need much additional work alter on, or without any hassles.
We need to instill this approach in everyone who is part of the production cycle, since changes will inevitably be required and the ability to react fast is crucial.
I remember working in a company where we used to have this big management meeting every Thursday, wherein we would report the development status to management. I have to repeat this: 90% done is not done. “We finished all the code, now all we have to do is test it” is a dangerous statement. We all know that everything can turn into “non-working” in one minute. And it usually does.
Working with small chunks with the right safety nets will help keep this mindset in place: continuous integration, red build approach, proper visualization, direct communication regarding problems and more.
A red build policy is crucial. Nothing is added to the build once it is red. It runs every all day with all tests being applied and no team, leader, manager or developer should ignore this.
We fix the bugs and then continue.
It’s like in a car factory. If a machine produces a faulty carburetor, the entire production line stops. Why? Because it is a faulty item and it will cost us. It can cost us as much as a recall of an entire line of cars or severe damage to our reputation. The costs of fixing and disturbing the entire production line are very high, and hence it is crucial to ensure everything works during production. This is a common approach in the automobile industry and in other industries. But somehow it is harder to grasp in software development.
Tools are created to spot these areas of faults, but it is the people’s responsibility not to ignore them.
I have seen many organizations ignoring the red build policy for various reasons from “oh, it’s a regression bug“ to “oh, it’s an old bug”.
Keeping good quality depends on keeping this policy in mind.
The zero defects approach is also critical. We stop and fix on the spot. We don’t ignore defects and leave them to the next version or to end-state integration.
- Lean thinking
Lean thinking is a huge topic, we already set some references points for Lean in some of the previous points. But to maintain it, we expect everyone to start thinking in terms of production line flow. For example:
- Reduce siloes
- Remove waste
- Identify bottlenecks
- Value driven thinking
- Continuous integration, Don’t wait for the “integration hell” period
Meaning: identify siloes that slow us down, identify unnecessary handoffs between people, and identify bottlenecks or waste.
As for waste, we define waste as anything that does not add value to the process. For example, in a furniture factory, producing tables without the ability to ship them to the customer creates inventory waste. Why is it waste? Because I need to pay for storage, it can become unusable over time. It can become damaged along the way.
Defects are another type of waste – why? Because I need to fix it, maybe recall fulfilled shipping orders, and replace it. It costs money.
Any waste in the development cycle?
Testers that are waiting for a build to test; extensive documentation that no one reads; extensive documentation that needs to be updated due to requirements, design or code changing towards the end (inventory).
A manager whose approval is required before anything moves forward; software that is waiting to be tested; code that is waiting for review that no one takes care of due to lack of time and meanwhile no code is being release.
In one company I worked in there was a testing bottleneck inside a Scrum team. Software did not ship out because “it was waiting for testing”. After a few discussions with my colleagues we decided to just pull the tester out of the team. No tester = no bottleneck. It is a radical step but it sent a message: no waiting. The team needs to learn how to balance their work in a way that less code is being written, more is being tested and code is being shipped out rather than piling and waiting.
In Agile and DevOps culture we look at the entire production line, we do not calculate the cost of each unit separately, but instead look at the entire value flow from beginning to end. It does not matter that a developer costs more than a tester if the entire line of production can’t produce working software due to lack of testing. What matters is that we have a piece of code that is getting older, harder to maintain and harder to test and repair later so we better get it moving and release it.
We expect the team to take ownership and promote stuck tasks, to identify and solve bottlenecks and to suggest the solution.
There’s a lot more to Lean then just that. But the point is looking at everything as one production line, identifying bottlenecks, removing waste and continuously improving for better performance.
Only people who take active part in the work can identify the production line’s actual problems. They usually know the solution.
- We are all in this together
The first step will be to open up the lines of communication between all teams, not only between Dev and Ops but also between Test, Dev and Ops so we can all deliver E2E.
It means that we are all accountable for E2E delivery. As a developer I am in charge of testing my code. I even report on the testing status in the Daily. I am accountable. When we work together so closely and things are continually being developed and tested it’s easy to take E2E ownership because I know what’s going on, there are no siloes.
In this environment you will see that the testing environment is the developer’s “problem” and concern, while the production environment is the tester’s concern.
The tester is not the sole person responsible for bugs in the end product, we are all responsible for the problem and the solution.
Imagine you are calling a help desk service for your cable TV. And imagine you accidently reach the wrong person, and the representative asks you to call again, or to call another service, they leave you empty handed and frustrated instead of helping you with your issue, regardless of the fact that it is not their “job” to solve it. This is not a mindset of “we are all in this together”; this is a mindset of “it’s not my problem, I am closing this service call”.
‘We are all in this together” attitude generates feedback regarding tools we need to implement, what we need to test and why. From the former example we may be able to move faster if the service rep had the option to direct us to the right person. We may even monitor the time it takes and the amount of times it happens so we can think of a better solution in the future.
In a rapidly changing environment, we need frequent information available to as in different areas. We need everyone to be aware to the fact that we are all in this together and losing one of us may cost us dearly.
This is a true story: I remember one offices of a major car rental company located in one of the biggest cities in Israel. The headquarters marketing department decided to go with a promotional offer that seemed likely to generate a lot of revenue for the company. The numbers looked amazing and would highly improve the figures we present to the public. What they did was to rent cars for the weekend for only 30$~ for the duration of the entire weekend. The promotional offer was aimed at a targeted audience of Orthodox Jews. Why? Since they don’t drive during Sabbath anyway, the cars would be safely parked around the city and returned after the weekend, and money will be entering the cashier without the cars even moving. Needless to say that what mattered was the actual revenue.
It took a lot of field employee pain and a lot of time, meetings and shouting till they realized they are losing money… And big time… But since they didn’t discuss their proposal with anyone, and were not aware of the entire production line, and were not fully cooperative with other departments, or didn’t know how to properly assess the costs of their actions, and were only concerned about the bottom line income, they didn’t know what was coming for them. Instead of earning more money the company lost big time.
The local rental offices suffered from a severe parking problem. So in order to keep up with the promotion the cars needed to be shipped from other nearby cities around to the small local office (and this costs). Parking spaces were very limited due to the crowded nature of the city, so cars started getting parking tickets. The crowd that reacted to this attractive sale was huge so the office needed to hire another secretary to deal with this crowd every weekend for many consecutive weekends, at least at the beginning, in order to relieve the load on existing personnel. They also needed to hire another employee to deal with the municipal authorities in order to to pay or cancel all these parking tickets (usually to pay). Dealing with parking tickets and a crowded office became a regular event at the end and beginning of every week, when a large amount of cars was being rented or returned.
Regular flow of customers’ began to deteriorate since the small office could not keep up the pace and service suffered. Now, you do the math. Money was being lost.
- Shared ownership
Shared ownership means what’s yours is mine and what’s mine is yours.
Like brothers, your room is mine and my room is yours… ho… Just kidding. But this is what it feels like when we start mentioning the topic.
What was mine? As a testing manager the entire automation operation was mine. Testing mangers used to raise this talented child called `automation`, with talented skilled automation developers placing the framework and building tests. How many times have we heard developers complaining that the quality of code in automation is subpar, that it needs engineering review? I heard it from time to time. And most of the time they were right. The separation from developers was not very helpful for automation code engineering and quality.
I also experienced the control testing mangers had over the entire automation operation to the point where in one organization developers created their own automation tool just to shorten the cycles and were given a slap on the wrist for doing so without the authorization of the “BIG” testing manager.
Well, no more. Automation belongs to everyone. There are places where the actual coding decisions and management are under shared development-testing leadership.
It is bilateral. Any developer can activate, write, or review automation code.
This is heartbreaking for testing managers.
But, when a developer start dealing with automation and makes it work, he sees how hard and complicated it is.
But the other way around is also something we need to accept. A tester can change product code.Writing automation code sometimes requires special coding standards that will allow this entire cycle to run. Sometimes the code itself needs to change. There is no need to wait for a developer to make the changes that the testers need to continue building automation. Testers can change code. Yes. I saw this happening in Scrum teams and it works fine. As long as we are all on the same page, working towards the same sprint goals, meeting daily and discussing technical issues, code changes done by testers become trivial, a non-issue.
- Early feedback
Early feedback means that testing, rather than being the last step of the process, which creates a gap in quality analysis (too late, at the end, when it’s expensive to fix or change), becomes a means of providing early feedback throughout the entire process. Remember, we test everywhere and all the time. But not only the testers, everyone is expected to work toward providing early feedback.
Our goal as testers is to look for feedback to provide all the time and provide it in a way that allows for early corrections and preventive measures to be implemented.
Testers provide feedback from business all the time, as they are the hands legs and eyes of the business inside the Scrum team. They do the same in everything that is being developed.
This mindset leads to developing and testing small chunks of software with frequent integration and frequent tests. Why? Because when the feedback comes earlier it’s easy to relate to, easy to fix, easy to detect or easy to retest.
This mindset is the reason we will place this log instead of the other, it’s the reason we will automate one process instead of the other, just because we need this feedback earlier than the other. This mindset will lead us to continuously monitor system performance, and not leave it to the end of development for a specific phase of performance testing. Why? Because assuming we are continuously building and testing every day, any deviation from expected performance is immediately detected and the relevant code was added just a short while ago.
Visibility is a powerful tool. Correct usage of visibility can create the expected behavior. We expect everything that can point us in the right direction or provide early feedback to be plainly visible.
Let’s illustrate the value of visibility with an example: Pretend that in your town, there’s a problem of speeding drivers. And as you are a responsible driver yourself, you know that’s dangerous.
To solve the problem, police set up hidden speed cameras in strategic locations. According to them, this is meant to catch dangerous drivers who exceed the speed limit, and so reduce the number of accidents. But as it turned out, drivers didn’t actually drive any slower, as they didn’t see any visible deterrent (hidden speed cameras, remember?)
So was there any value in setting up the cameras? Apparently not. Our taxpayer money was spent on buying and installing them, but the actual driving speed didn’t go down. So there was a challenge (speeding drivers) and a quick reaction (hidden cameras), which didn’t actually produce any value.
But was there ANOTHER value in setting up the cameras?
Yes, and you can probably guess what it was. With the hidden cameras in place, the police department (and the town council) made buckets of money from collecting fines.
The problem, of course, is that’s not what we were looking for. We wanted to reduce the number of accidents by making people drive slower. THAT was the intended value.
When the hidden cameras were taken down, and VISIBLE cameras were set up, people started slowing down. And that reduced the number of accidents.
See? Value, and the value we were looking for.
The following is an example of changing driving speed by means of clear visualization:
The following is an easy way to control large crowds:
Control your inventory easily:
Control your child?
And here is how it looks like when you deal with DevOps. Everything is visible, and the relevant information that needs to change is presented:
- Continuous improvement
We expect everything to grow and improve. This means our ability to move fast, our skills, our insights and our performance. How? By adopting the mindset and tools that allow us to continuously improve.
Performance matures over time, it is not born excellent. Experience is learned but only when we aim to learn from past events and apply our conclusions to future events.
Retrospective, for example, is one of the tools used to improve. We examine our performance and deduce the next actions toward improvement.
This mindset should emphasize failure and mistakes as something we learn from. We will not tolerate blaming. Testers are not to blame if a bug was found and developers are not to blame if a mistake was made. This is easier said than done. People will often quickly turn to anger and blame, and hence attention and constantly reminding this point is crucial.
When we work in small chunks, it’s easy to detect mistakes earlier, it’s easy to fix them, it’s cheaper to fix them and easy to learn from them and correct them. Therefore stopping the race from time to time for the sake of retrospect (or having kai-zen events) is a helpful practice to adopt.
Growing and improving can only happen in a nurturing environment. Respect for each other’s role, trust and collaboration should be built.
Let’s go back to F1. They race 20 times a year. Only 20 races. Each race is “to be or not to be”. To reach the point of an excellent performance requires a great deal of team collaboration, openness, respect and a lot of continuous improvement. They didn’t start from the point of being able to stop for 2 seconds in the pit stop, they grew to that point by continuously improving their ways. The way they change a wheel, the way they stand, the colors they highlight, the people that need to be present in a pit stop – everything is examined and improved.
You can see for yourself in the next two short videos, how they looked like back in the 50s and how they look like today… and how they got there.
Have you heard of Team Sky? The won the Tour the France. Why? Because their manager’s approach was to retrospectively examine each race and continuously improve by making changes in small steps. They got to the point where they were taking their pillow with them to a race since it makes them perform better the next day. This is how champions get to be the best.
Who is the tester then?
- In every one of us
- A skilled developer
- A skilled business representative
- Automation everywhere
- Manual tests are saved for exploratory purposes
What is the culture?
- No walls, no gates, no transitions …
- Trust and collaboration
- Early feedback
- Continues improvement
Old world vs. new world DevOps approach and mindset.
One last thing:
What do you need to make this happen?
- Management’s active support and appropriate mindset
- Understand the fact that we need to react to a rapidly changing environment
- Always be on top of technological changes
- Highly skilled testers
- Highly skilled developers
- Hire team players
- Start even without the mindset fully developed – because mindset is something that grows over time.
- Hire a Supernanny (Agile coach). Do you know the Supernanny? Well , she is always around, advocating the right culture and coaching, teaching or mentoring on the right culture for desired behaviors