Improving Signal-to-Noise ratio in mobile app monitoring

[gandharva]

In our last discussion, we explored how mobile app reliability goes beyond crash rates and ANRs, looking at various types of failures that impact user experience. But spotting these reliability issues in real-time presents another challenge: how do we separate genuine problems from normal variations in our monitoring data?

When an issue occurs, resolution time isn't just about finding and fixing the problem - it's heavily impacted by app store review times and how quickly users update their apps. Even after deploying a fix, it might take hours before metrics return to normal.

Traditional monitoring approaches often generate more noise than signal. A spike in errors could indicate a critical bug, or it might just be a busy Friday evening. The diversity of mobile ecosystems adds another layer of complexity - each user's device varies in platform version, locale, network conditions, and countless other factors.

Two Approaches for Better Signal Quality

In widely deployed apps, even when most users haven't received a fix, telemetry is constantly streaming in. Some devices will get updates quickly and start reporting data. This creates an opportunity for two approaches to get clear signals from noisy data:

1. Smart Error Ratios: Beyond Raw Numbers

Instead of watching absolute error counts, design low-latency error ratios that use reliable data while accounting for normal traffic fluctuations. This lets you evaluate fixes immediately after deployment, even with partial user adoption.

Think of it like this: If your app serves millions of users and only 20% have updated, a small improvement in overall metrics might actually represent a massive success among updated users. The key is adjusting your perspective to consider the fix's adoption rate.

2. Configuration-Aware Metrics: Understanding Context

The second approach is more structural: design your metrics to include device configuration state as a dimension. This means every metric carries information about the app version, feature flags, and experiment groups in effect when it was generated.

This approach shines in experiment-based deployments. When you roll out changes through controlled experiments, you can easily:

• Compare metrics between control and treatment groups
• Isolate issues to specific configurations
• Measure the true impact of changes
• Avoid false signals from device/network quality variations

Why These Approaches Matter

Here's a common scenario: during staged rollouts, metrics often look great in the first few days but deteriorate after a week. Why? Users with better devices and networks tend to update first. Without proper metric design, you're not comparing versions - you're comparing user populations.

This is where configuration-aware metrics and smart error ratios become crucial. They help you:

• Distinguish between real issues and population differences
• Get reliable signals even with partial rollouts
• Make data-driven decisions about continuing or rolling back changes
• Identify issues before they affect your entire user base

Implementing These Approaches

To make this work:

1. Design your telemetry system to always include:

   • App version
   • Experiment IDs
   • Feature flag states
   • Device capabilities
   • Network conditions

2. Build your analysis tools to:

   • Compare metrics within similar user segments
   • Account for rollout percentages
   • Consider device and network characteristics
   • Track error ratios rather than absolute numbers

3. Remember that every upgrade can introduce side effects that skew your metrics. Think about what happens when a user gets an app update. Their first launch often clears caches and local storage, leading to what we call a "cold start." These cold starts naturally run slower as the app needs to rebuild its caches, reload data from the network, and reinitialize its databases.
   But it's not just about cold starts. Updates might also reset user preferences or trigger data migrations. Users might need to grant new permissions, or their settings might revert to defaults. Each of these changes affects how the app performs and how users interact with it.

These side effects can trigger false alarms in your monitoring systems. When you see increased latency after an update, you need to ask: Is this a real performance regression, or just the natural consequence of users starting fresh? Are users experiencing actual problems, or just the temporary friction of an upgrade?

To handle this complexity, consider implementing a "settling period" in your analysis. Compare metrics before and after this period to distinguish between temporary upgrade effects and real issues. For critical features, you might want to maintain separate dashboards for first launches versus subsequent uses. This separation helps you understand the true impact of your changes without the noise of upgrade effects.

Is it worth the effort?

These approaches require more upfront investment in your monitoring infrastructure. But in mobile app development, where every app store update takes hours or days to reach users, catching issues early is invaluable.

The goal isn't to collect more data - it's to get clearer signals from the data you already have. When you can confidently interpret your metrics, you can make faster, better decisions about your app's health and reliability.

What monitoring approaches have helped you cut through the noise? Share your experiences!

⭐ If you like this post, please checkout Measure. It's an open source tool to monitor mobile apps. It captures crashes, ANRs, navigation events, API requests, and much more to create detailed session timelines that help find patterns and get to the root cause of issues. Check it out here and feel free to star it for updates!