It Depends

 As part of an agile transformation, I was guiding an organization towards smaller batches enabling more frequent inspections of the organization’s progress against a ‘known good’ definition-of-done.  For complex agile projects, this approach validates that the correct collective progress is being made, and ensures issues are addressed as they are uncovered.

The organization was insistent on retaining their traditional development mindset and keeping their dependencies documented using ‘depends on’ links, effectively generating a dynamic Gantt chart.  Whenever a team issued a new dependency link, the dependency got an immediate high priority from leadership until the two teams agreed upon a path forward together.  A new dependency would disrupt the receiving team who had other critical work to complete.  Over time, the teams reverted back to on-going partial work with ever increasing dependencies on other teams.  The onslaught of dependencies disrupted teams from reaching their sprint goal, lowered their definition-of-done to maintain velocity, and undermined their predictability.

How one handles dependencies are completely different in traditional development compared to agile development. I have struggled to help leaders understand that there are two ways of delivering complex programs, that these two ways are based upon different principle sets, and that they are executed with different expectations and rituals.  

Traditional development follows a series of milestones of progressive decision making over time, for example, serial agreements to business case, product requirements, architecture & design, implementation, validation, market readiness and finally product launch decisions.  A traditional development Gantt chart maps out the program plan with dependencies within the phases showing where one team’s outcome enables another team’s work.  This is consistent with traditional development’s making a plan and executing the plan.  The use of the phases helps provide early detection that something’s amiss.  For example, if a team cannot complete all of their architecture & design decisions by the design complete date, executives know that they have an issue early in the program.

Agile development is different, first and foremost, because the product is in the continuous state of being ‘ready to ship’, or ‘done’.  To do this, means that product backlog and architecture has been constructed such that the teams can work independently and incrementally in design, development and validation.   At any point in time, all teams can operate and inspect the current ‘production’ system.  When everyone agrees that sufficient customer value has been achieved, the system is immediately released to customers.  Equally important, no one team is allowed to cause builds of the whole system to fail, validation to become stalled, or deployments to the staging environment to pause.  Even during early development, any of the aforementioned events is equal to a production outage and teams do everything to return the system to ‘production’ quality.

So, how are the interim agile development dependencies handled across teams?  Think of a traditional program’s Gantt chart as being a horizontal relationship map over time.  Now rotate the Gantt chart 90 degrees where an incremental agile program outcome is at the top and all necessary work cascades downward to stories that teams complete during the same sprint (or a few sprints).  This is called, a hierarchically structured backlog, where stories are completed to enable an epic, epics are completed to deliver a theme, where each issue type of the hierarchy is ‘done’ and inspected.  Each and all teams’ progress can be inspected and tracked.  

But using vertical dependencies, or a cross-organization hierarchically structured backlog, planning is focused on a series of inspectable, incremental, cross-team outcomes that keeps the system operational even when feature-poor.  This allows surfacing unknown risks early and uniform inspection of progress across the whole organization.  When there is a failure, the identification of the root cause and corrective action can happen quickly to enable the current incremental outcome and adjust the future outcomes.

Agile development’s known-good, production quality first over traditional development’s feature design and coding first allows leadership to have a common standard to inspect and understand progress.  Traditional development’s strict phase-driven decision making allows leadership to have a different standard to inspect and understand progress.  However, a mixture of low quality implementation and loose phase-driven decision making means pure development chaos.

Transforming Architecture

 Architecture, an architect’s role and their relationship to agile principles are seldom defined when I start an agile transformation.  There are those who believe architecture is a lofty set of future technical needs or desires.  For example, architects provide solutions to address past technical debt that will never be implemented.  There are those who believe that architects live in an ivory tower and profess designs that have no possibility of being implemented within the constraints of the -ilities (affordability, scalability, securability, sustainability, supportability, etc).  There are those who know that agile means do whatever, whenever and however they want until there is something new and urgent that needs to be done instead.  When all of these beliefs are held in an organization, architecture is dismissed.

Given this baggage, I start an agile transformation by asking, how do you define architecture and the role of architects?  As expected, I get a mixture of responses across the organization depending upon who’s answering.  Engineering managers tend to define the role subservient to their role as managers, as in technical designs that fit into their envisioned schedule and feature set. Product managers define the role as subservient to their roles balancing stakeholders and customers’ needs, as in technical designs that support the whim of sales or customers’ desires.  Support or quality leads define the role as ensuring supportability or delivering reliability.  Even the technical leads will define their own roles as being subservient to everyone else.  

When I ask, if a technical infeasibility surfaces, who raises it and resolves it?  I’m surprised by their willingness to immediately take on technical debt and paper over the infeasibility.  Or, worst yet, toss the technical issue on another team or organization, and move forward with an unrealistic, low-quality solution.

To reset the discussion, I define architecture as ‘technical decisions that we hold ourselves accountable to’.  This means that the decisions (or agreements) are technical in nature and made by engineers, technical leads and/or architects.  Architecture decisions relate to but are not management, product, quality or support decisions.  Everyone is held accountable to these decisions which means that the decisions have to be written down and tested/checked against.   If there’s a discrepancy, we either change the implementation to be consistent with the technical decision(s), or we change the technical decision(s) for everyone.  Architects, or technical leads, are responsible for cultivating, documenting and, when necessary, making these technical decisions.  Architecture and architects stand as a separate task and job role.  They are part of the collaboration between Product Management, Engineering Management and the team.

This definition avoids the pitfalls of the above baggage by centering architecture in the space of documenting and brokering technical decisions that are written down and used.  Normally, engineering managers, product managers, support, and quality don’t want to cultivate or maintain a set of technical documentation.  Equally important, most everyone will agree that given the pragmatic nature of the definition, it avoids the ivory tower and irrelevance concerns.

I can hear the screams of the agile purists, ‘working software over comprehensive documentation’!  Most agree that ‘over’ does not mean ‘instead of’.  Both working software and documentation are highly valued and important.

Let’s step back and use Mary Poppendieck’s ‘Build Integrity In’ Tools; Perceived Integrity, and Conceptual Integrity.  Perceived Integrity is the consistency in how we present abstractions and interactions to our users.  This means that our abstractions, or user design decisions are expressed in both internal and user documentation, and are carefully cultivated and validated with our users. Equally, this means that we have explicitly decided and documented who our users, or user personas, are. Therefore, I place the decisions of user design and user personas in the realm of architects’ technical domain.   Any change in user personas has a massive impact on every aspect of the product and needs to be carefully considered and controlled.  Architects tend to understand these impacts on Perceived Integrity better than the other roles.

Turning to Conceptual Integrity as it comes more naturally to technical leaders.  Conceptual Integrity deals with the speeds, feeds, interfaces, APIs and functionality of the code itself.  As we’ve learned, APIs are contracts between components.  Neither component can change the contract without an agreement of all parties involved and a phased transition to the new agreement.  A lot of modern language and API styles have helped to ease these transitions but regardless, the contract has to be kept to keep the system operating.  When these are documented, maintained and tested against, the resulting system tends towards resilience. 

Back to the ‘working software over comprehensive documentation’ concern.  Recent innovations in validation, APIs, languages, CI/CD, tooling and UI development, have moved us towards the reality when comprehensive documentation is also working software.  As we maintain software, we can maintain architecture and our technical agreements.  With each software change, we know if we’re staying aligned with those agreements.

Another benefit of this approach, by clearly defining and using user personas, all teams can now align their user stories and outcomes to identically the user personas defined for Conceptual Integrity.  So as Product Managers, Engineering Managers and teams discuss potential user value, they have a common understanding of who they are discussing for precisely the intended outcome that is consistent with past delivered value.

During an agile transformation, it takes a while for the organization to grow accustomed to this clearly defined architecture role, responsibility and accountability.  Some organizations have created a ‘triad’ collaboration where Engineering Leadership, Architecture and Product Owners engage the team as a single voice during the team’s refinement, planning and execution.  Teams benefit because they know what’s expected of them so they can focus on creating the highest value for the customer.