Observability for product teams: the what, why, who, and how

Observability isn’t a new concept in the software industry, but there are still so many different interpretations of this term. In many cases, observability ownership appears ambiguous, and it’s often treated as an afterthought.

Have you ever released a new experiment or feature into the wild only to discover that your team can’t determine its success? What if different people have different ideas about what success implies?

This article will dive into what observability brings to product teams and how it can become an automatic way of thinking.

What does observability mean?

Observability was introduced in the 1960s by R. E. Kalman as the ability to understand the internal state of a system by observing its external outputs. It’s a fundamental concept in control theory, systems theory, and computer science—particularly in the context of distributed systems and microservices.

Today, team compositions represent the system's ideal future state (Conway's Law and the Cognitive Load Theory). We've seen an increase in modularity, reflected in smaller product teams owning scoped software modules and features with more explicit boundaries. Observing the system's internal state is no longer sufficient to determine success.

For example, you can have an endpoint feature that operates at 99.999% uptime, but does that mean it's successful? It depends on what you decided to optimize for and how well-received the feature is. 

Let's simplify this engineering-heavy term: the word ‘observability’ comes from the verb ‘observe’, meaning ‘to take notice of something’ and regard it with attention. 

So, let’s start with a more straightforward fact: product teams need to be able to observe their product.

Observability vs. monitoring

Another misconception is telemetry, or monitoring, which is often confused with observability. Monitoring and alerting are tools that work on top of observable properties. 

Observability can be described using a definition coined by Vasco Pinho, former Senior Director of Engineering at Hotjar by Contentsquare: “Observability should be treated as a required property of the system that enables us to measure success, learn, and decide where to invest.” 

On top of observable properties, we can build monitors, service-level objectives (SLOs), HEART (Happiness, Engagement, Adoption, Retention, Task) metric graphs and more, but we need the data to be in place first. Observable properties are proactively implemented to enable monitoring to be built on top.

Here are 2 examples to clarify the difference between observable properties and the tooling used on top:

• Out-of-work example: a person goes out for a jog and as they jog, they keep looking at their smartwatch to regulate their pace based on their heart rate’s bpm

  • Observable properties: body data points, heartbeat, the jogging pace

  • Tools used: smartwatch monitors and alarms for high heart rate

• Software example: a team releases a new API endpoint, and they set up monitors to know when something’s not going well

  • Observable properties: metrics, logs, and traces

  • Tools used: monitors, SLOs, graphs, and alerts

This data will reside in multiple systems because each system focuses on a specific area. One does not replace the other, so you need to make sure you’re using the right tools for the right job.

For example:

• Datadog allows you to determine performance metrics as well as event monitoring for infrastructure and cloud services

• Mixpanel enables you to capture data on how users interact with your digital product 

• Digital experience platforms like Contentsquare give you behavioral analytics and feedback data to help you empathize with and understand your customers

All systems matter and are equally important because they explain a different aspect, or part of the sum, but a feature’s end-user experience is the sum of all the parts.

It’s essential to distinguish between the observable properties and the tooling used on top of it. 

Why does observability matter?

Reason 1: HEART metrics dependency

The first reason revolves around Google’s HEART metrics relying on the foundation of reliability. Why? Because reliability is what leads users to trust your product. Here’s how you can think of trust:

Trust = reliability = consistent behavior over time

This equation means that trust depends on reliability, and reliability is reflected through providing a consistent user experience that meets your users' expectations over time. 

Each time a user engages with a product and it doesn’t behave as expected, it leads to confusion, frustration, doubt, and a dent in trust. Consistency here doesn’t just mean UX consistency across different features of your product. It also means a consistent ability to access your product (no downtime) and consistent user operation performance (requests not taking too long). 

To achieve consistency, you need to proactively learn about and respond to different life events happening all the time. This is what an observability mindset enables.

Reason 2: delivering the correct value, faster

The second reason is an increased ability to deliver value to our users faster by investing in the right place at the right time. When building a minimum viable product (MVP), it's common to cut corners to release them quickly and start collecting user feedback, but this can be broken down a little more. Let’s start with the definition of MVP:

💡 A minimum viable product (MVP) is the first version of a product fit for market. An MVP has core functionality and, coupled with customer feedback, is a learning tool for product teams to release new features and better iterations of the product.

Teams focused on releasing an MVP try their best to avoid lengthy and (ultimately) ‘unnecessary work’, but how do you classify unnecessary work? Does observability fall into this bucket? Let’s dig deeper and pay some attention to the V in MVP: viable. Viable means ‘capable of working successfully; feasible’.

How do you make sure your feature is viable? Through proactively planned observability. Releasing a feature into the wild and expecting customer feedback via 'contact us' forms will get you some feedback—but only from that small subset of loyal users willing to take the additional steps to get the input through. 

With observable data in place and the tooling to help visualize it, teams can learn much more in much less time. Observability is not one of those corners you want to cut during an MVP.

Source

Let’s use a fictional example explaining how speed depends on good observability. Imagine driving a car much faster than you’re used to, like a Lamborghini Huracan with ​​a V10 engine, 0-100 km/h in about 3 seconds. With that much speed, you need to have clear visibility of what is coming up ahead, right? 

For product squads, trying to release faster with little (or no) observability is like driving a Huracan with a blindfold on (please don’t try this at home).

Reason 3: alignment

A third and final reason is alignment. By proactively thinking of what data you need to observe, you also define what success looks like. Observability provides a common language and shared understanding of the product and its performance. 

Through observability, teams can access the same real-time data and insights, allowing them to make data-driven decisions and stay aligned with the product's goals and objectives. 

This alignment helps to ensure everyone’s working towards the same vision and objectives, reduces misunderstandings, and improves overall collaboration and teamwork.

In summary, observability promotes alignment by providing a shared understanding of the product, enabling the team to make data-driven decisions, and fostering collaboration and teamwork towards a common goal. 

Regarding MVPs and observability, it’s essential to strike a good balance and identify the minimum observable properties needed. With this agreement documented, you leave less room for interpretation.

Who owns observability?

Observable data comes from many different sources and is no longer a topic solely driven by engineering. It’s something that all angles of a product need to reflect upon, including

• 📱 Product

• 🎨 Design

• 🛠 Engineering

In a nutshell, the owning product squad owns observability. Let’s look at a scenario to help explain why.

🔎 Case study: observability for product teams

Let’s say a product team finished the last commit of a feature, the reviewer approved the Merge Request, and the continuous integration (CI) pipeline pushed this to production. 

You switch on the feature flag, and your users experience the new feature. After a few hours, you receive user feedback saying they can’t perform fundamental operations, which escalates into an incident. 

The team that worked on the feature gets pulled in to assess the impact. After further analysis, the problem is identified, and a fix is deployed. 

For this example, let’s assume the whole process took 4 hours to detect the incident and another 4 hours to get the fix live, with the entire squad looking into it. 

This means you had 8 hours of user impact, but that’s not all. The team that worked on the feature returned to their sprint board, but they lost half a day, with the entire team focusing on the unplanned incident. 

As a repercussion, some tasks must drop out of the sprint meaning the team won’t meet their goal. The disruption impacted the team owning the feature, mainly because the issue was discovered reactively. 

As you can see, the feature-owning team is the most disrupted in this scenario. The scenario makes it more apparent that product teams are responsible for observability not only due to the disruptions to their planned work but also because they want to see their features succeed and their users happy. 

As explained earlier, observability covers different aspects, therefore all team members can contribute. The easier it is for the team to proactively learn about a new feature’s success, the less impact on their users—and the less reactive disruptions to their weekly goals.

However, it's important to note that observability is a cross-functional responsibility. Everyone in the team should know the importance of observability and participate in building and maintaining its infrastructure.

How should teams go about incorporating observability?

Teams must agree that observability is not optional and for every feature, it should be a required part of the definition of ‘done’. When looking at a new user story, it can only be considered ready for sizing if the definition of success is clear. With this in place, the team can assess the observable properties needed during implementation, what quantitative feedback they want to collect as part of the user experience journey, and what additional tooling they need to monitor this data. 

Every time you design a new feature or make a change to an existing one, you must think of the observable properties, user behaviors, or system behaviors you need to observe to measure the product's success.

From an engineering perspective, you must consider the following: 

• Logging and log levels to record events, errors, and debugging information to make it easier to query and analyze 

• Adding instrumentation to the code to capture performance metrics, such as response times, error rates, and throughput

• Distributed tracing to track requests as they move through different systems to help identify bottlenecks and debug complex problems

• Using a monitoring platform to help monitor this data for anomalies and failures, allowing you to set up alerts and notifications

In your MVP and initial releases, you don’t need all the fancy SLOs, monitors, and graphs. You need the foundational data to help determine success from failure and understand early system performance so that you can learn from it. With a limited set of observable properties and the necessary tooling, you can determine where to iterate and what additional tools are needed.

In summary, to avoid treating observability as a speed bump, you need to plan ahead for it.

FAQs about observability

Observability isn’t just metrics and data. It’s a mindset change that product teams need to drive faster and invest in the right space.