Why (Batch) Size Matters

At one company, we worked to bring the development batch size down from multiple quarters across the whole organization to a few weeks of each team’s efforts.  To enable this endeavor, I defined a batch as development work completed to an organizational-wide, agreed-upon definition of done and customer outcomes.  The batch size was calculated by the time and effort required from start to  finish of the agreed upon customer outcomes by people or teams involved.  

Even with this simple definition of a batch, a number of the company’s leaders engaged in passive aggressive behaviors to undermine the movement to smaller batches.  The executive sponsors focused on surfacing the passive aggressives’ reasoning and systematically addressed their concerns over time.  In doing so, the development batch size was reduced, code quality increased, and predictability improved.

I found the passive aggressives’ reactions and thinking informative of the challenge facing organizations when moving from traditional, large batch development to agile, small batch development.  Allow me to represent their thinking here.

There are two vastly different principle sets at work behind traditional, large batch development and agile, small batch development.  The traditional development principles are based upon the efficiency gained in development by aligning partial work outcomes across all teams.  For example, aligning key requirements by a requirements completion date, aligning designs by a design completion date, aligning implementation by a functional completion date, aligning testing by a test completion date, and aligning system delivery by a system test completion date.  

The agile development principles are based upon the efficiency gained in development by always returning each small batch to a high quality of completeness and delivering the batch to final system testing or to the customer many times within a sprint.  

Both are valid claims to efficiency yet diametrically opposed on how they gain efficiencies of development.

Changing an organization from traditional, large batch development to agile, small batch development requires a different level of thinking across roles/teams, processes, architecture, requirements, validation and delivery.  At the root of the push back, was an element of the change in development principle sets.  

Claim: that’s not how we do work.  Response: Correct, small development batches cannot be achieved by doing what we did with large development batches.  In Scrum and Scrum@Scale, the organization structures and individual/team roles are very different.  We need to reset expectations of people’s roles and their metrics including the leaders’ roles.

Claim: We need to make design changes as we implement and test features.  Response: We need to spend more ensuring that APIs are correctly designed, validated and backward/forward compatible so incremental changes can be made with each small batch.  We have to design in perceived and conceptual integrity.

Claim: we lack the build and test speeds.  Response: let’s prioritize time to speed up builds and test automation so we can reduce the development batch size over time.  Otherwise, we’ll be wasting team members’ time waiting.  The payback in increased velocity will offset the upfront costs.

Claim: Features take a lot of cross team communication. Response: The increased cross team communication is a potential sign that our APIs are poorly designed, documented and/or validated.  Let’s spend time incrementally improving these where cross team communication is costly and can be significantly reduced by an improved API.

Claim: Features are bigger than a single sprint.  Response: Yes, features are bigger than a sprint, and more importantly, than a batch when the batch size is smaller than a sprint.  This means that our grooming has to take into account time for understanding overall customer outcomes, needed validation investments, good APIs and breaking up customer outcomes so they can be incrementally delivered.  

Claim: We have too many bugs to fix prior to release so we’ll defer them and fix them in the next batch.  Response: Let’s step back, why are we deferring defects?  They are deferred because we poorly estimate how much work can be completed in a batch.  The larger the development batch, the less confident we’re in our estimation.  Shifting to small development batches, better APIs, better grooming and consistent definition of done, we increase our confidence and decrease our error margins.  We won’t have to defer defect fixes.  Equally, a batch is not finished until it meets the definition of done and accepted by the product owner per the agreed upon outcome.

Claim: Customer will never accept increased frequency of product delivery.  Response: Why won’t they?  Is it because our infrequent complex releases, labor/resource intensive updates and deferred defects create poor product update experiences?  We’ll address these by prioritizing non-disruptive-update capabilities, use the improved update capabilities to speed up our validation automation, never defer defects, improve design integrity, and incrementally show improvement in customer update experiences.

As we worked through these claims, the responses and changes, the passive aggressives started to sign on to small batches, albeit, slowly.  The best part was that customers noticed these improvements as the new releases came available.

Footnote for executives: note how investments in people, process, tools, methods and architecture are key to moving to an agile development principle set.  These can be done incrementally, however, must be done with forethought and purpose.  As noted in the answers, these investments are codependent with grooming the backlog, inspection of outcomes, and resolution of impediments.  Transparency and inclusion of these investments helped this organization address the concerns and move forward together.

Constraints: The Good, The Bad, and The Ugly

 A frequent conversation with executives who are entwined with an agile transition is about the often stated claim that an executive cannot tell the agile development team what to do, how to do it, or when it must be done.  I ask her from where did the teams pick up these claims?  Her response is that she is unsure.  She says that it maybe was stated during the Scrum team’s training and that their Scrum Master is claiming this to be true.  I ask given this situation how does she communicate status with her stakeholders and boss.  After a pained expression, she says that’s why I’m talking to you.

Talk about constraints!  My executive friend seems to be the one constrained by her teams and team leaders.  Constraints are an “applicable restriction or limitation, either external or internal to a project, which will affect the performance of a project or process”, as defined by CrossLead, an executive coaching firm.  For me, teams forcing constraints on executives seems a bit odd and are likely the ugly side of constraints.

So, are constraints good?  Let’s use CrossLead’s definition to rephrase this question.  Are ‘applicable restrictions and limitations, either external or internal to a project, which will affect the performance of a project or process’ good?  Let’s try this example out.  Let’s say that we have an agreement with the company’s  investors for a specific level of funding for a specified period of time to deliver a capability.  Is this a good constraint?  I would argue that if reasonable and real, knowing this is a constraint is helpful during teams and leaders’ decision making.  Yes, it’s a good constraint, especially, when compared to not having investors or funding.

Other examples of good constraints are: the requirement that one collaboration and work tracking tool is used consistently across all teams, the definition of done for completeness of work, the timing of Sprint standups, planning, review and retrospective meetings, the structure and priorities of various customer and business outcomes in the backlog, the definition of teams/roles across the organization, the offering architecture and design, and the build/devops methods.

These good constraints provide the correct context for teams to efficiently and effectively operate together in creation of high-integrity customer outcomes.  In other words, these restrictions and limitations help teams in delivering the desired outcomes together.  They act as a binding function bringing teams together.  You might consider these good constraints akin to a collaboration framework.

So, what type of constraints are bad?  Clearly those that inhibit teams from delivering their outcomes.  Many times, these constraints are undocumented, unspoken and/or out-of-date decisions that have become inculturated into an organization’s thinking.  

When I hear comments such as, that’s not how we work, that’s against policy, that’s not what our customer is now demanding before they buy, etc, I immediately question where these constraints are coming from and how visible/valid they are today.

  

My first investigation approach with constraints like these is to ask, ‘where are these constraints documented?’  The answer is more than not that the constraints aren’t documented.  I guide the teams forward by asking them to develop the right constraints (if any) that will take their place, document all of the constraints in force, and widely communicate them.  If they are documented, then I approach the leadership team to discuss if they are aware of the constraints and if the constraints remain in force.  If the constraints are no longer in force, we actively communicate the situation, why the constraint is not in force and what has taken its place.  If the constraints are in force, we add it to our constraints and ensure that every team and leader understands the constraint and its reason for being.

Back to the ugly constraints.  These are constraints that are half-truths or based in ignorance that are accepted by the organization and unquestioned by leadership.  For example, take the three listed my executive friend, an executive cannot tell the agile development team what to do, how to do it, or when it must be done.  While partially true, that an executive cannot tell the agile development team what to do, every team has stakeholders and has agreed to a prioritized backlog with customer outcomes and a common definition of done.  This is how an executive works with teams to reach agreement with the teams.  The executive has to know how the teams decide what to do and has to participate in these methods for everyone to be aligned in what will be done.  

The executive cannot tell the team how to do it is a half-truth because it doesn’t inform the executive that she is able to inspect the outcomes at every sprint review meeting as a stakeholder and to inform her of potential changes in what’s desired for the upcoming sprints.  If there’s a gap, backlog prioritization and grooming will help bring the stakeholders and team together.

Lastly, on executives not saying when the work must be done is a confusion of constraints.  Mature agile teams are able to communicate the rate of completion of customer outcomes, or the velocity of their work, and are able to project potential future outcomes based upon their properly groomed backlog.  Mature agile executives should know how to use this information to correctly communicate to their stakeholders.  If there are any concerns, the executives and teams have the grooming mechanism to identify and address them together, while maintaining excellent agile methods.

Notice the pattern here, the executives are active in the identification, disposition, communication and articulation of constraints.  By doing so, the executives positively affect the performance of their teams.

Yacht Racing and Agile Transformations

An organization’s agile leadership team reached out to me to review their agile transformation approach.  Their Senior VP who was leading the organization wanted their software development to be state-of-the-art.  They reasoned that agile development was state-of-the-art, so a transformation initiative was underway to adopt agile development  methods.  They had experienced agilists leading the transformation effort with a well constructed, long-term approach.  We spent time together every other Friday to discuss various issues and concerns raised as they proceeded forward.

One day a business analyst in the organization reached out to me asking how I would answer the Senior VP’s question regarding how other organizations calculated ROI of an agile transformation.  Great question.  I should have responded with ‘Yacht racing!’ as in the America’s Cup Race yacht race.  I wasn’t as pithy at that time.

Why ‘Yacht racing!’?  Not that I’m into yacht racing.  I sail a 16 foot, center-board, Vanguard 420 sailboat on small lakes.  Oceans are too big for my sailing abilities.   I have a general interest in the America’s Cup yacht racing tournament.  As you may be aware, the racing world was in shock when 1988 Dennis Connor and the San Diego Yacht Club responded to a challenge race with a multihull catamaran.  This started the monohull vs multihull design innovation race.  Near 2010, yacht racing took another turn as hydrofoils and rigid sails were added to the design, doubling racing speeds.  Even today, yacht designs are changing as they add dynamic hydrofoils to monohull yacht racing akin to a shovel-snouted lizard lifting their legs to cool off.  Each America’s Cup, yacht racing design teams have to accept the defending team’s yacht design terms otherwise they don’t race.   

Equally, agile development methods can be key to how certain markets operate.   For example, a leading micro-services, REST-API, as-a-service, DevOps offering in the customer relationship management (CRM) space, would likely use mature agile development methods to constantly release a stream of customer value while staying always-in-production.  To participate in the same market as our mythical CRM as-as-Service offering, competitors would likely have to match the leader’s capabilities and innovate better, faster and smarter to take market share.  Agile development methods are table stakes to participate in the market.

Let’s assume that before we undergo a transformation, that we’re a capable competitor in our market using reasonably mature traditional development methods (waterfall, spiral, sequential, etc).  What are the costs of a transformation?  New organization, new decision making methods, new development methods, new validation methods, new roles, new leadership behaviors, new architecture, and new delivery methods are just a few costs to incur.  However, the highest cost items are time and talent.  My estimate is that an agile transition will take on the order of 3 to 5 years from start to maturity and will yield, at least, a 20% turnover in key, talented people.

Under what circumstances does something this costly merit consideration?  I can only think of two circumstances.  The first is we want to disrupt the market before our competitors disrupt the market, like Dennis Connor’s move to multihull yachts did for the America’s Cup race.  The other is that our market has been disrupted and we are forced to follow the new leader.  These are big gambles and require strategic consideration, investment and commitment.

Let’s head back to ROI of the ‘desire to be state-of-the-art’ reason for going with an agile transformation.  Both traditional and agile development methods can be state-of-the-art, in their own way, and deliver market-leading customer value.  The assumption that a development methodology is only state-of-the-art if it is an agile development method is incorrect.  From my experience and reasoning (that’s another series of blog posts to be written), the difference in customer delivered quality between traditional and agile methods operating at full maturity is relatively small.  With the cost of an agile transition in effort, time and talent being extremely high, the ROI is low when shifting from one state-of-the-art method to another is the sole benefit.

What’s much more critical is when the market is defined by agile development characteristics.  For example, in our mythical CRM market leader described above.  This fictional CRM market, micro-services, REST-API, as-a-service, DevOps, constant stream of customer value and staying always-in-production are aligned with agile development methods.  Hence, even with state-of-the-art traditional methods, market participation and leadership would be extremely difficult without a transition to agile development methods.

If an agile transformation won’t help you redefine your market, making improvements to your traditional development capabilities will likely deliver better returns.

Why do an agile transformation?  Remember Yacht racing.  Be the winner in applying agile development methods in your market.

In Appreciation of Mary and Tom

Shortly after starting a position, I attended my boss’s first off-site staff meeting at a remote office.  My new peers, boss and I were in the conference room together.  One of my peers could not attend and was connected with me via IM.

We did a round of introductions.  During this time, I mentioned that as part of my new role, I would also be doing an agile transformation in how the products in my new organization would be developed.  My boss responded with, “You’re going to do what?”  I was taken aback a bit and responded to my boss that as we had discussed during our multiple interview discussions, I wanted to do an agile transformation.  A peer flashed up the Agile Manifesto on the projector screen.  My boss pointed to the screen and asked, “You’re going to do that?”  

My remote peer’s IM popped up on my laptop in front of me, “Stop now.  You’re getting him upset.  It’s best you don’t.”  I responded to my boss with an apologetic claim of misunderstanding.  My goal was to deliver world-class products.  I would do this consistent with the organization’s best practices.  My boss accepted this.  He moved to the next agenda item.  I knew that damage to my credibility had been done.

Back home at the building where my boss and I worked, my boss came into my office early in the morning.  He sat down.  He needed to talk about this agile thing since I had never mentioned this to him during the interviewing process.  I tried to explain that I had.  I could tell from his reaction that either I had not or he had not understood what I had said during our discussions. His dejected response was that maybe he had mishired. Defeated, I accepted responsibility for the misunderstanding and would resign immediately due to my miscommunication of my intentions. 

I asked for one favor before I resigned.  I asked him if he would read Mary and Tom Poppendieck’s book, Lean Software Development: An Agile Toolkit.  I had found Mary and Tom’s work profound in outlining core agile principles.  My boss readily agreed.  He left my office with the book.

To his credit, he returned to my office an hour later.  His finger pointing to Tool 8: The Last Responsible Moment.  He claimed that he agreed with the Last Responsible Moment principle and all what he had read so far.  He left my office.  I was speechless and encouraged.  I was grateful and amazed by his immediate reading of Mary and Tom’s book.

Another hour later, he returned again.  His finger pointing to Tool 19: Refactoring.  He was shaking his head.  He claimed he had always agreed with this.  He knew that he should do it but never did it in his past projects.  He left my office reading as he walked.  My hopes were rising that my resignation would not be needed.

Another hour later, he returned yet again.  He tossed the book on my desk.  He claimed, if this is what I intended to do, then I have his support.  He said he would remain skeptical.  I would have to prove that this agile stuff does work.

I set out on my mission to bring about a state-of-the-art agile development organization by embracing my new organization where they were and bringing about the understanding of agile principles.  I used Mary and Tom’s book as my starting point with every team I met.  We ensured everything built was based on these principles.  As other organizations saw our results and invited us to explain, I used Mary and Tom’s book as the baseline.  I encouraged them to read it before anything else. 

Years later as I was leaving that company, my boss told the whole organization as he expressed his appreciation for my contributions, that I demonstrated how agile really can work and helped the organization understand those principles by embodying them in everything we did.  I responded by thanking my boss for entrusting us all to change and deliver results consistent with the agile principles.

Prior to walking into my next job, I made it very clear in the hiring process that I was all about agile transformations.  I immediately asked my new boss and subordinates to read Mary and Tom’s book.  My new boss read the book during a trip to the beach.  She internalized the principles quickly.  She and I worked on multiple successful agile transformations while at that company.  I started every transition with Mary and Tom’s book.

Mary and Tom explain the right, complete list of agile principles of which Scrum and Scrum@Scale are built upon.  Their work is extremely approachable and referenceable. That is why their book is first on my blog reading list.

The companies that I’ve worked for have purchased 100s of copies of their book.  I doubt that even with those book sales, I have returned the favor of what their published work has done for my teams and me.

Mary and Tom, thank you for your inspiring and, in my case, job-saving work!

A Road Less Traveled

Recently, I spent some serious reflection time with a senior executive on how he ‘phased in’ agile into his large organization of 400 plus engineers.  His ‘phased’ approach was persistently done across his entire organization over multiple years.  These were bold steps communicated consistently to everyone.  

His phased steps were:  the first directional decision (if you will a north star of where they were all heading), organization assessments, training, scrum team formations, PO assignments, more frequent releases, modernizing tools, building the Scrum@Scale structure, moving big-batch to small-batch, single definition of done and more.  The next step was taken as the organization could tolerate that step.  He was always focused on the business and customer outcomes knowing that he had to keep people and product value moving forward together.  

At the time of our conversation, his organization was gaining on velocity and maturity, so there wasn’t really any pressure to do something radically different.  He was just taking time to step-back and reflect.

I asked, what if he would have at one point, at most any point, told his organization, now that we’re underway, we’re trained and we’re doing what we can set out to do, we have an additional change for everyone in the organization.  On a particular Friday, everyone’s old job roles, old ways of working, old projects, old commitments, old tools, etc are terminated.  Those old things are now gone.  On the following Monday, everyone is hired into our new organization, new job roles (as defined in our maturity model based upon Scrum and Scrum@Scale), new tools, new small batches, new definition of ready for grooming, new definition of done for completeness, etc.  As with every newly hired individual, our previous experience is valued, but the new jobs-at-hand defines us, as will our future customer value.

He responded with, why, would he ever want to do that?

We were noodling the timeline of the changes that he made and how the timing selected for the next phased step seemed to take longer than desired.  We discussed the passive aggressiveness, or collective resistance to the changes.  Often the ‘that’s not how we work’ pushback kept popping up.  When new tools were introduced, many in his organization would claim, ‘the tool won’t work because it can’t do what the previous tool did’.  When Scrum@Scale structure was introduced, his managers initially relabeled their existing staffs as Scrum-of-Scrums or metaScrums and kept working as they historically worked.  When releases were shorted to quarterly, they quickly devised a plan for staged development and test quarters for six month or larger work batches.  

Each of these initial responses were really a rejection of the decisions to move forward.  Some responses were covert.  Some were overt.  Regardless, the responses slowed the organization’s movement forward and had to be addressed.   Now, to his and his leadership team’s credit, they coached and nurtured his teams and managers through the resistance.  They got them on the right track towards their north star decision.

I asked, what if given what he knows of the resistance to change, he, metaphorically, terminated everyone’s old positions over the weekend and, metaphorically, rehired them into the new positions, would that have helped ease the resistance?  Who starts a new position at a new company by saying, ‘that’s not the way they did it at their previous job’?  Doesn’t everyone who joins a new company or takes a new position, start with the usual 30-days of listening, 30-days of planning and 30-days of taking initial steps?  They learn before they act anew.

He reflected upon these ‘what if’ questions.  His response was, being clear that the old jobs, the old ways of working, the old ways of delivering customer value, the old ways of making decisions and the old ways of thinking were gone would have helped everyone reset expectations.  The reset would have provided everyone a clear, clean break with the past.  This would have helped everyone resist together the passive aggressives’ claims and behaviors.  He said that he should have told everyone that it applied to his job too.   In retrospect, that would have been a good step to take.  He didn’t just see it at the time.

I responded, we’re all learning together.  

Knowing him, he’ll be leading more agile transformations.  He’ll continue to reflect on his past experiences to learn.  He’ll now have another path to consider, a different road less traveled.

Innovation Drives the Next Generation Jobs

I was recently asked what is the most difficult agile transformation task facing executives today?  My response was holding their people accountable to the requirements of the new agile jobs, such as product owner, scrum master, scrum teams, etc.  I got quite a puzzled look in response and then an ‘aha’ smile from the questioner.

Technology innovations have been creating new job roles ever since the first technology was placed in the hands of early craftsmen.  In The Perfectionists: How Precision Engineers Created the Modern World By Simon Winchester, Simon shows how the evolution of measurement and precision changed engineering and manufacturing over time.  As you might imagine, the evolution also changed the jobs that people held.  Prior to the innovation of carving out a large steam cylinder out of a single piece of iron done by the first machinists, blacksmiths were employed to hammer a sheet of iron into a cylinder held together with rivets.

Equally, technology innovations have changed jobs in our high-tech industries.  Remember typists using  typewriters, operators using word processors, and executive assistants using email who each handled document creation and distribution based upon fundamentally different tools.  Now, that document creation and distribution is pretty much done by all of us due to technology innovations and ever improving tools.  Who knows, maybe with Zoom, we won’t be using written words in the future?

Each new innovation follows an ‘S’ curve of adoption where adoption is slow at the beginning, speeds up through the transition point and slows again at saturation or maturity.  You could imagine part of the adoption curve to be limited or enabled by the rate that humans are trained and hired into the newly created jobs that utilizes that technology.  As a manager of a new, emerging job type, you would be expected to hire high-potential, yet untrained, employees and train them on the new tool or technology.  Think of being that first manager over the factory milling large steam engine cylinders where all your potential new hires were blacksmiths with their hammers in tow.

Nice, but what does iron milling have to do with agile development?  Let’s focus on job types that are different between traditional or sequential development programs and agile development programs. 

With traditional development, we have programs lasting quarters, if not multiple years that span across various organizations’ functions: business planning, product planning, design, engineering, testing, manufacturing, support, marketing and sales.  Each organization will have specific job types for their area of expertise, like software development engineer, quality engineer, product manager.  Each organization will also have similar job types shared across the whole organization, for example, program managers and group managers.

Each traditional development program job type is consistent with the long duration programs, that is, their work is based upon long-duration-program methods, or technologies.  Program managers use requirement documents, estimates, gantt charts, dependency management, commitment tracking, check points and status summaries.  Each function’s role, knows and when to plan, work and communicate their deliverables.  All of the group managers were likely promoted within their organization based upon their ability to do their organization’s jobs well.  They know how to coach new employees in each of these roles.

Shifting to agile development programs, the new jobs are based upon very different technologies and methods.  The organizations have self-managing teams, scrum masters, product owners, small increments frequently released to production (like every few hours… or, yikes, minutes) and velocity.  The people in these new jobs hold or attend daily standups, sprint planning and review meetings, scrum of scrums and meta scrum meetings.  The organizations operate in a constant flow of production changes and improvements.  There are no checkpoints, gantt charts, or estimations.  There is no equivalent of design, implementation, or test phase gates.  While group managers exist, their jobs are totally different with self-managing teams, scrum of scrums and metascrums.

All well and good, but what’s does this have to do with holding people accountable to their new job requirements?

Many executive leaders are not aware that job types dynamically change when the organization starts on an agile transformation.  If they are aware, they may face resistance from HR when communicating the new jobs now exist.  Equally important, even if the new job can be coded in the existing HR system, the actual work done by the person is very different from the traditional development jobs.  For example, in traditional development, software engineers often focus on implementation of code per the committed requirements and designs, leaving it to quality to demonstrate that those commitments were met.  However, in agile development, software engineers, as a team, have to demonstrate that the desired customer value increment was delivered to production quality for each and every increment.  

Let’s imagine, we are the first hiring manager for our new milling machinists.  We would describe the new job, require skills, necessary training and evaluate our new hires against these needs to help them continuously improve and grow.  As demand for our new steam engines grew we would be holding ourselves accountable to mastery of our whole organization against these new milling jobs and tools.

Equally, executives have to hold their organizations, their managers and their people accountable to the new agile job requirements. In doing so, the organization will work towards mastery of the new jobs, tools and technology.

Back In The Saddle Again

Great to be back!  

I'm returning to cloud savvy after spending the past six years at EMC and Dell Technologies Dell EMC.  The time at Dell EMC was amazing due to the breath of offerings in fast changing markets.  I owe many debts of gratitude to the Dell EMC leaders, managers, and team members.  Each had many lessons to teach me helping me refine the future direction for cloud savvy.

Where to go from here?  

Over the past 15+ years, many technologies and methodologies have emerged based upon a new agile principle set that are very different and distinct from the preceding traditional principle set.  Comprehension of the new principles is difficult for leaders because of their success with the traditional principles.  When development projects run into trouble, their back-up styles often undermine their organizations' progress towards mastery of the new principles.

Equally, teams and individuals follow rituals without understanding the principles behind them.  When the rituals fail to deliver, instead of changing behaviors consistent with the new principles, they modify the ritual.  The modifications continue without clear benefit as results lag expectations.

I've decided to dedicate cloud savvy to the mission of helping executives view their situation with the lens of both the traditional and agile principle sets to help guide them forward to the right outcome for the business.  Understanding both principles sets provides the executives with the right framing to guide their teams forward.

What's next?

I'll be sharing new thoughts and insights here for everyone to read and review.  For those who want to share feedback, it will be welcomed.  For those who want more insight or help, they can reach out to me directly and we can discuss steps forward.