Ops at Scale Maturity Assessment¶

Estimated time to read: 12 minutes

Based on the provided information, here's a summary of the different levels of maturity across various aspects of software delivery. This framework complements the DORA and SPACE Metrics and the Observability Maturity Assessment. For context on the three disciplines covered, see DevOps, SRE, and Platform Engineering.

Culture - Team Structure & Product Alignment¶

Baseline: Uncommunicated E2E vision, low collaboration, sub-innovating, multiple teams in silos, reactive.

Novice: Clear delivery requirements, rapid intra-team communication, innovation by necessity, teams organised around deliverables, written knowledge.

Intermediate: Autonomous teams, articulated business goals, clear project requirements, rapid inter-team communication, innovation by design, maintaining scripts and creating orchestrations, automation as documentation.

Advanced: Articulated business vision, clear product requirements, frequent collaborative communication, strategic innovation, shared responsibilities within the autonomous team, shared knowledge.

Expert: Articulated business strategy, clear business requirements, rapid and continuous feedback, ownership mindset, proactive, common knowledge transfer.

Deployment Automation¶

Baseline: Manual deployments, fully scripted deployments.

Novice: Push-button deployments to test environments, manual deployments to production.

Intermediate: Immutable structure defined, standard deployments across environments, push-button deployments to production.

Advanced: Automated deployments to production after tests pass, zero-downtime deployments, database deployments, independent multi-tier deployments.

Expert: Ability to implement continuous deployment.

Continuous Integration¶

The capabilities below are expanded into a practical implementation guide in CI/CD Pipeline Design: From Baseline to Expert.

Baseline: Official builds are performed on developers' machines, build artifacts are managed, some manual scripts still used.

Novice: Automated unit tests embedded in the CI phase, coverage is measured and breaks the CI phase, system polls source control and builds on commit, official builds are not performed on developers' machines, build artifacts are automatically managed, dependencies are managed in a repository, build status notification is sent to committer, build only once.

Intermediate: Mocked smoke, contract and functional test as part of the CI, SAST is part of the CI, source control tells system when to build, no polling.

Advanced: Mocked DAST as part of the CI, code analysis quality gates break the CI phase, SAST and DAST quality gates break the CI phase, container builds become the main CI outcome, trend reports are automatically generated from build server events, fast feedback to developers.

Expert: Gated commits to mainline based on CI phase result.

Loosely Coupled Architecture¶

Baseline: Monolithic application, risky deployments as all software is deployed/released together.

Novice: Monolithic application is broken into smaller modules/components, all product tests must be run on each commit, scaling application requires a complete copy of all components.

Intermediate: Pyramid of testing, independent release of components, teams become autonomous, components are containerised (e.g., in Docker), devs/testers need only basic docker commands to run the entire product.

Advanced: Different versions of individual components can be run in parallel, support for canary and blue/green release by component, scaling of application at microservice level and more efficient use of resources, inter-service tests use mocking and services can be tested in isolation and in parallel, commit to an individual microservice only runs related tests, faster MTTR.

Expert: Loosely coupled microservices, each component can be deployed and released independently, low-risk deployments/releases, contract testing ensures that if tests are passed, the release of microservices is safe.

Version Control & Trunk-based Deployment¶

Baseline: No single source of truth, long-lived branches, and disconnected sources of truth.

Intermediate: Branches are used for isolating work, and trunk-based branching methods are employed.

Advanced: Short-lived branches, single source of truth, and mainline branch is pristine.

Expert: All commits are tied to tasks/stories, and pre-release tagging and traceability are in place.

Cutting-edge: Product and multi-component interconnected sources of truth, and traceability analysis with auto-generated release notes.

Continuous Testing¶

Baseline: Automatic unit testing with every build, and manual functional testing.

Intermediate: Code coverage is measured, and peer-reviews are conducted.

Advanced: Automated smoke tests, functional and integration testing, security static code analysis (SAST), and automated end-to-end/regression tests in target environments.

Expert: Automated performance and security tests (DAST/AST) in target environments, integrated management and maintenance of test data, and shift-right paradigm with test monitoring, alerting, and logging tools.

Cutting-edge: Production testing, post-release A/B testing, chaos testing, and business impact testing.

Test Data Management¶

Baseline: Shared test data environments and manual data anonymisation.

Novice: Isolated test data environments and automated data anonymisation.

Intermediate: Test data management strategy in place, synthetic data generation, and data anonymisation tools.

Advanced: Test data management integrated into the pipeline, automated synthetic data generation, and consistent data anonymisation.

Expert: Test data strategy as code, on-demand test data provisioning, and data anonymisation integrated into the pipeline.

Monitoring & Observability¶

Baseline: Whitebox monitoring performed, basic logs and metrics are recorded locally.

Intermediate: Detailed metrics and log data are recorded centrally.

Advanced: Blackbox monitoring, access to logs and metric data for teams, and automatic alerts and alarms triggered by metric thresholds.

Expert: Constant deployment feedback, profiling against data, predicting potential issues, and debugging implicit failure modes.

Cutting-edge: Patterns, traces, and anomalies detected automatically, proactive actions, and dependency analysis of services.

Database Automation¶

Baseline: Manual data/schema migrations and manual rollback-enabled data/schema migrations.

Intermediate: Automated un-versioned data/schema migrations.

Advanced: Database schema changes are version controlled, and DBA team does peer review.

Expert: Automated database deployments, data migration in production environments, test data management, and automated rollback-enabled data/schema migrations.

Cutting-edge: Zero downtime database releases using canary.

Security¶

Baseline: Manual penetration testing.

Intermediate: Account & access key compromise detection, compliance scanning, storage monitoring, and configuration monitoring.

Advanced: Code vulnerabilities detected and measured (SAST), privilege activity monitoring, VPC/security group/firewall monitoring, encryption, and key rotation monitoring.

Expert: Application functional and performance vulnerabilities detected and measured, real-time network traffic visibility, suspicious traffic detection, automated penetration testing (DAST/IAST), and cybersecurity integrated into product design and delivery.

Cutting-edge: Production environments monitored and protected by a security engine, and auto-remediation in place.

Security & Compliance¶

Baseline: Reactive security processes and manual security testing.

Novice: Regular security audits, basic security training, and automated security testing.

Intermediate: Proactive security processes, security as code, and security integrated into the pipeline.

Advanced: Shift-left security and compliance, security champions, and automated security gates in the pipeline.

Expert: Continuous security and compliance, security and compliance metrics, and automated security and compliance remediation.

Metrics¶

Baseline: Cycle time and build frequency not measured, and no real-time feedback loops.

Intermediate: Trend reports are manually generated, and stakeholders are notified of MTTR, change failure trend, and availability.

Advanced: Cycle time and build frequency measured in real-time, automated trend reports, and team-wide notifications.

Expert: Low MTTR, high availability, real-time dashboards for stakeholders, and real-time feedback loops.

Cutting-edge: Cross-team data mining and analysis, - and business impact and performance connected to technical performance.

Infrastructure¶

The path from manual virtualisation to serverless and Internal Developer Platforms is detailed in our IDP Implementation Guide.

Baseline: Manual DR Active-Passive failover architecture, static VMs Infrastructure, manual patching, firewall and network management, and DNS record creation.

Intermediate: Infrastructure as code in place for VMs and configuration management systems in place.

Advanced: Environment standardisation, immutable and ephemeral environments, automated deployments, cloud infrastructure for VMs (IaaS), automated patching, firewall and network management.

Expert: Cloud infrastructure for containers (CaaS or PaaS), infrastructure as code for containers, zero downtime through cloud-based continuous operations, automated DNS record generation, and CDN in place.

Cutting-edge: Continuous operations, serverless infrastructure, and traffic shaping in production environments.

Release Management¶

Baseline: Infrequent releases with manual review and coordination, no end-to-end relationship between request and release.

Intermediate: Partially automated releases requiring manual intervention.

Advanced: Basic Dev-to-Ops traceability, velocity, and release feedback.

Expert: Frequent releases with defined manual and automated orchestration and calendaring, full traceability from request to deployment.

Cutting-edge: Just-in-time or on-demand releases, every change is deployed to production, and immediate feedback and triggers.

Risk Mitigation¶

Baseline: Risk controls are done manually as part of the delivery process.

Intermediate: Lean Change management controls in place and automated CR process.

Advanced: Quality gates enforced during the CI phase, test controls enforced pre-release, and 100% test pass rate implemented.

Expert: Security SAST gates enforced during the CI phase, DAST and AST enforced pre-release, database automation practices, and full audit trail of the delivery process.

Cutting-edge: Security engine protects production state, compliance controls in place, proactive auto-healing and auto-scaling in production, and traffic shaping to ensure zero downtime deployments and reduced risk of outages.

Tools¶

Baseline: Code build done on CI server like Jenkins using Maven and NPM, committing the code to a source control system like GitHub, and automated unit tests using JUnit and ESLint.

Intermediate: Code scanning using SonarQube, functional tests using Rest Assured, Selenium, and Cucumber, artifact repository management using Nexus, configuration management using Ansible, collaboration tools like Confluence and Slack, and project management tools like Jira.

Advanced: Static Application Security Testing (SAST) using Checkmarx, deploying VMs in GCP using Terraform, logging through Splunk, containerisation (Docker), service discovery and version management using Consul, secret management using Vault, API mocking using Wiremock, database schema management using Flyway or Liquibase, dependency vulnerability check (OWASP), and contract test frameworks in place (Karate).

Expert: Containerisation orchestration using Kubernetes, AST/DAST using Contrast Security or OWASP ZAP, data logs visualisation using Kibana, open-source monitoring using Prometheus, and performance test frameworks in place like JMeter.

Cutting-edge: Service mesh using Istio, open-source data visualisation using Grafana, service mesh observability and configuration using Kiali, feature toggles for A/B testing using LaunchDarkly or Intuit.

Innovation and Experimentation¶

Baseline: Little to no innovation or experimentation, a risk-averse culture, and a strong focus on maintaining the status quo.

Novice: Occasional innovation and experimentation initiatives, a culture that tolerates some risk-taking, and a moderate focus on exploring new ideas and approaches.

Intermediate: Regular innovation and experimentation initiatives, a culture that encourages risk-taking and learning from failures, and a strong focus on exploring new ideas and approaches.

Advanced: Systematic innovation and experimentation initiatives, a culture that embraces risk-taking and learning from failures, and a strong focus on continually pushing boundaries and seeking disruptive ideas.

Expert: Organisation-wide culture of innovation and experimentation, a strong focus on continuously pushing boundaries and seeking disruptive ideas, and a systematic approach to identifying, testing, and scaling new ideas and approaches.

Agility and Adaptability¶

Baseline: Low agility and adaptability, a strong focus on following established processes and plans, and a resistance to change.

Novice: Some agility and adaptability, occasional changes to processes and plans, and a moderate willingness to embrace change.

Intermediate: Moderate agility and adaptability, regular changes to processes and plans, and a strong willingness to embrace change.

Advanced: High agility and adaptability, continuous evolution of processes and plans, and a proactive approach to embracing change.

Expert: Organisation-wide culture of agility and adaptability, a strong focus on continuously evolving processes and plans, and a proactive approach to embracing change and seising new opportunities.

By working towards continuous improvement in innovation and experimentation, agility and adaptability, organisations can better respond to market changes, customer needs, and new technological advancements. This enables them to stay competitive, deliver value more efficiently, and ultimately achieve better business outcomes. It is essential to create an environment where learning and growth are encouraged, and teams are empowered to take risks, learn from failures, and continuously improve their processes and approaches.

Collaboration and Communication¶

Baseline: Siloed teams, limited cross-team communication, and a lack of transparency in decision making.

Novice: Occasional cross-team communication and collaboration, some transparency in decision making, and basic information sharing.

Intermediate: Regular cross-team communication and collaboration, increased transparency in decision making, and structured information sharing.

Advanced: Strong cross-team communication and collaboration, high transparency in decision making, and a culture of open information sharing.

Expert: Seamless cross-team communication and collaboration, full transparency in decision making, and a strong culture of open information sharing and collective decision making.

Continuous Learning and Skill Development¶

Baseline: Minimal focus on continuous learning and skill development, limited opportunities for professional development, and a lack of support for employee growth.

Novice: Occasional focus on continuous learning and skill development, some opportunities for professional development, and moderate support for employee growth.

Intermediate: Regular focus on continuous learning and skill development, multiple opportunities for professional development, and a strong commitment to employee growth.

Advanced: A strong culture of continuous learning and skill development, extensive opportunities for professional development, and active support for employee growth.

Expert: Organisation-wide culture of continuous learning and skill development, a wide range of professional development opportunities, and a proactive approach to supporting employee growth and development.

By understanding these stages of maturity and working towards continuous improvement in each area, organisations can optimise their software delivery processes, enhance collaboration, and ultimately deliver better products and services. The key is to regularly assess and analyse the current state of maturity and identify areas for improvement, while also fostering a culture of learning and growth within the organisation.

By fostering a culture of collaboration and communication, organisations can break down silos, improve information sharing, and drive better decision-making. In addition, by focusing on continuous learning and skill development, organisations can build a workforce that is adaptable and able to respond to evolving business needs.

By prioritising these four dimensions, organisations can create an environment where continuous improvement is ingrained in the culture, and employees are empowered to innovate, adapt, collaborate, and learn. This approach will enable businesses to stay competitive in an ever-changing market and deliver long-term value to customers, stakeholders, and employees.