IDC Opinion

The digital business model has digitalized processes, services, and products; significantly transforming customer engagement models. Observability continues to garner significant attention from business and IT executives as detailed visibility, precise control, and automated actions become core operating tenets within the digital infrastructure. Business executives now demand critical customer experience knowledge to create and successfully execute growth strategies.  Executives are adopting observability platforms to deliver high performing digital services that drive great customer experiences and revenue growth.  

This wide use and deep impact are reflected in IDC’s view of a comprehensive Observability Platform as a:

“Platform that collects (in real-time) high volume and high cardinality data at scale, and utilizes advanced analytic models across network, application, cybersecurity, cloud and digital experience domains with extensibility that enables packaged solutions for technology, business, and the partner ecosystem.  In addition, the platform is extensible to take advantage of the data and analytics allowing for custom solutions to be developed for additional use cases.”  

An observability platform is an evolution of monitoring capabilities that empowers technology teams to move from being reactive to proactive by unifying, viewing, analyzing, and automating the entire end-to-end top-to-bottom digital infrastructure, encompassing core computing and cloud infrastructure, network, applications, security, and digital experience data to prevent downtime and negative impacts to customers.  It enables technology teams (application, security, networking, platform engineering, and Site Reliability Engineering (SREs) to collaborate with business executives on understanding the customer journey and measuring and managing the moments that matter for revenue growth and customer experiences. It offers a consolidated path across technology, teams, and processes to deliver lower costs, automated actions, and team and process efficiencies that directly corelate to revenue generating services that support customers and business growth.    

75% of the respondents agree that their CEO and business leadership understand that complete visibility and control over infrastructure, network, applications, security, and digital experience, is critical to digital business success. And with so much riding on the digital infrastructure, 47% of the respondents cite their average cost of an hour of downtime at $250,000 or more.  Executives realize that employees, partners, and customers demand high performing services. Business architectures are now technology architectures, and delivering great experiences are a major factor in generating sustainable revenue growth, customers adoption and innovation.  IT organizations are recognizing that the digital infrastructure must be viewed as more of a concert than a collection of instruments. As such, comprehensive observability is fast developing into a core tenet when building out a resilient and responsive digital infrastructure.

The research further suggests (and often quantifies) critical themes that IT and business leadership teams should consider as drivers for Observability platform investments.  These are: 

• Collaboration and improved proactive control points across security, IT operations, and SRE teams offers a pathway to improving and accelerating an organization’s security posture and service performance at the same time. 
• Observability platform adoption offers technology capabilities that unifies traditionally fragmented data onto a single platform, and applies analytic models to improve team and process efficiencies that reduce business and customer risks.  
• The importance of solution extensibility that empowers customers with the ability to ask and solve questions from a unified observability data model, unlocking data value and new use cases that are unique to every organization. 
• The rising influence and collaboration of Security Operations teams with IT Operations and Site Reliability Engineering teams when using Observability platforms and improving an organization security posture.
• Observability provides IT leadership teams with a direct path for business leadership collaboration on customer journey and experience conversations and value streams.  
• Service Partners have opportunities with Observability platforms through new use case development, and high interest in SaaS-delivered services.  

With most existing IT management tools, staff, and practices focused on specific siloed technologies, moving toward a more concerted monitoring and management effort presents many challenges. Successfully applying comprehensive observability practices and capabilities involves people and organizational constructs, processes, and technology. Through unification, a comprehensive observability platform offers IT organizations an opportunity to satisfy complex management requirements. Meeting these management requirements drives operational excellence, empowers business and technology teams to work faster and smarter, and deliver rich, reliable digital experiences to workers, partners, and customers. 

An overall analysis of IDC worldwide survey results focused on the use, challenges, and returns relating to observability yields detailed views into current state, solution expectations, and future strategies (see Figure 1). 

Methodology

This white paper is based on a global IDC web-based survey of organizations (see Appendix). It included 2,062 respondents represented by titles that are IT operations (44%), network operations (17%), IT executives (13%), observability engineers (10%), DevOps (10%), system manager (9%), Cloud operations (7%), AppOps (7%), SecOps (5%), Platform Engineering (5%), and Site Reliability Engineering (5%). Organizations were categorized as small (500–999 employees, 20%), medium sized (1,000–1,999 employees, 30%), large-sized (2000-4,999 employees, 30%) and very-large (5,000 or more employees, 19%). Respondents were asked about the role and value of observability in their organization; how it impacts security, operations, development, and SRE teams (among others) and processes; and business executive involvement and value. The survey spanned core themes across people, process, and technology impact, and the role that observability can play in the future of business outcome realization.

Respondents represented 12 countries: Australia, Brazil, Canada, Germany, France, India, Japan, Mexico, Netherlands, Singapore, the United Kingdom, and the United States. Organizations represented a range of industries spread across financial services, construction, professional services, telecommunications, manufacturing, healthcare, retail, transportation, and others.

This survey was conducted in March 2023 by IDC and was commissioned by Cisco.

Situation Overview

Observability as a practice and set of capabilities requires a platform foundation based on data collection, intelligent analysis, and actionable insights that can be applied and shared in-context across multiple IT and business domains. But how do executives perceive (or define) observability today? The research indicates that 40% of the respondents aligned their understanding of observability as applied to monitoring as “tools and practices that enable the sharing of intelligence and insights across multiple IT teams.” And current and future spending in the segment is strong. 75% said they spend $1 million or more, and of that group, 21% spend $5 million or more. Reflecting the current and potential impact observability is expected to have on both IT and business initiatives, 46% or respondents are planning to increase their related spending over the next two years. 42% plan to stay the same.

Critical Needs: Consolidation, Collaboration, and Proactive Control

What is the reality of observability within the IT organization today? IT organizations are increasingly experiencing tool chain fatigue, resulting in high costs, unneeded complexity, delayed responses, security vulnerabilities, and integration barriers (see Figure 2). 40% of the survey respondents use 2- 10 monitoring/observability tools, 28% 11-20 tools, and 18.5% use 21-40 tools. Incredibly, another 10% indicate use of 41-100 tools! With so many tools in use, it should come as no surprise that 74% of respondents said data collection and correlation is difficult. The many respondent citations relating to needed improvement in staff productivity and team collaboration are certainly driven by the over- abundance of tools gathering data, issuing alarms, providing conflicting insights, and misdirecting staff actions.

However, despite the plethora of tools deployed, critical measurements and coverage gaps persist. For example, with multiple tools to apply and a wide mix of IT domains (e.g., operations, engineering, security, networking, etc.) and lines of business involved, 82% of the survey respondents said it’s difficult to deliver a flawless end-to-end digital experience to internal employees and external business partners; 77% said the same for end customers. If the digital infrastructure must be presented as a concert, the audience experience goes unmeasured and, certainly, undermanaged.

Only 17% of organizations stated they are using an observability solution that meets all their needs, and 25% of organizations believe they have optimized their observability journey. The research indicates the maturation of observability solutions and practices and the movement from reactive monitoring to proactive observability can further align people, process, technology, and automation in unique ways. Figure 3 indicates how IT, and business teams are considering the use of observability to further unify data with collaboration and automation for a positive business impact. IT organizations are certainly looking to achieve greater results with a comprehensive observability approach and platform.

Data unification across critical IT management domains is another pressing requirement. In fact, 53% of organizations stated they need to unify observability across critical IT management domains such as infrastructure, network, applications, cybersecurity, and digital experience. The top three leading drivers of unifying observability data are:

• Improving business and technology use case capabilities using observability data and analytics
• Using observability data to improve teamwork across various IT domains
• Increasing cybersecurity and application security threats.

The Rising Influence of SecOps on Observability

The involvement and rising collaboration from Security Operations (SecOps) stakeholders with observability practices cannot be denied. In fact, increasing collaboration with SecOps teams and adjacent IT teams is a key value proposition for observability (see Figure 4).

As a growing influencer in observability decision making, SecOps teams are demanding greater visibility and control; underscoring the growing complexity of securing application architectures and the infrastructure platforms that support modern, containerized applications. Because of the increased influence held by both developer and security teams today, operations teams must increasingly view their counterparts’ teams as internal “customers” of their services and platforms — building solutions that meet the fast-moving needs called for by oncoming DevSecOps practices.

IT organizations are reacting positively to heightened demand for SecOps involvement in observability. 61% of the respondents agree that the security team will play a larger role in the observability solution selection, use, and collaboration. In fact, SecOps teams ranked second (only ITOps was higher) for who should be responsible for establishing, controlling, and advancing observability efforts within the organization (see Figure 5).

Over the next two years, the survey data also indicates that security influence is expected to grow with observability decisions. 99% of the respondents stated that observability will be more influential to their security decision making when compared to today.

Observability Provides Ample Service Partner Opportunities

For the service partner ecosystem (e.g., managed service providers, outsourcing vendors, and systems integrators), observability provides significant opportunities to deliver value added management services and support. 67% of respondents agree that service partners are an acceptable or even preferred alternative for their observability initiatives (see Figure 8). Often daunting observability challenges and heightening requirements are driving many IT organizations to demand service-based deployment (and payment) options and a platform perspective for observability. In addition, these service partners provide much-needed observability expertise, integration capabilities, and best practices to their service clients.

In addition, Figure 9 indicates the breath of opportunities for service partners to augment specific skillsets that IT organizations prioritize when developing and hiring staff with observability responsibilities. Indicated staff shortages, skill gaps, and constrained teamwork are all pointers to service partner opportunities. Notably, significant areas that service partners can add value for IT organizations include comprehensive observability platform extensions focused on automation, security, data management, and cross-IT engineering and operations.

Future Outlook

In laying out their plans for observability, respondents offered answers directly matching often cited strategic business priorities. Productivity, impact, efficiency, effectiveness, and collaboration are all assigned high placement in observability plans (see Figure 10).

When examining specific observability capabilities to be prioritized now and into the future, respondents reflect a movement to a more proactive management approach. Reactive management capabilities such as identify problems and threats and provide rapid remediation are assigned almost the exact same importance as proactive management functions such as predictive analysis, problem prevention, and digital acceleration (see Figure 11). And again, we see further evidence of the importance of a single source of truth and extensible observability solution – bolstered by support for standardized technologies (e.g., Open Telemetry).

When examining observability practices, future plans are aligned to multiple priorities. There is little difference among the top-ranked impact areas. Once again, high value is placed on cross-IT teamwork, IT automation, unified management, intelligent analytics, and reduced cost and complexity (see Figure 12). Organizations do indeed aspire to accomplish much with their future comprehensive observability platforms and efforts.

Challenges/Opportunities

Opportunities abound for organizations to reap benefits from the deployment of a comprehensive observability platform. Of course, opportunities are frequently made possible because of challenges that disrupt the way things operated in the past. Specific challenges and opportunities associated with a comprehensive observability approach include:

• Customer Experience challenge: Customer Experience is the new growth engine and delivering a flawless and exceptional user-experience is directly associated with positive business outcomes.
• Opportunity: there is a critical need to monitor and track real-time digital experiences for users and applications as a business requirement.
• Security challenge: Security threats are rising while, at the same time, security vulnerabilities are increasing.
• Opportunity: Security is pivoting from a surround-and-defend approach to an approach where security becomes intrinsic to each component of a digital infrastructure – from core systems and services to business data and applications. As such, comprehensive observability must support a strengthening of security postures and practices across all possible attack vectors and cyber targets.
• Cross-domain challenge: Security, Operations, Network, and SRE executives face the need to create a collaborative culture that measures success based on data, using common metrics that drive behaviors, and a blameless culture that empowers teams to fail fast and learn without unnecessary career risks.
• Opportunity: Organizations have a once-in-a-lifetime opportunity to fundamentally rethink how teams are organized around observability platforms, how teams integrate and collaborate, and how decision making is carried out.
• Digital Infrastructure challenge: Organizations are struggling with organizational and technology complexity as their digital infrastructure evolves to support accelerating business demands, intensifying user demands, and an expanding web of IT systems and services – e.g., multiple clouds, zero trust, sustainability requirements and distributed and diverse networks.
• Opportunity: IT organizations can reduce process and tool complexity by standardizing on a comprehensive observability platform that allows for consistent, policy-driven management across business workloads, multicloud environments, hyper-connected infrastructure, and digital experiences.

Survey Demographics

Respondents per country

Primary Industry

Department

Company Size

Knowledge/Authority