OPS 2: How are you capturing and surfacing metric data that can be used to analyze the usage and consumption trends of individual tenants?

Publish and aggregate tenant-aware metric data from your SaaS environment, making it available to a range of roles within a SaaS organization. This will provide immediate access to key usage and activity trends that are essential both business and technical teams. This data is used to shape architecture strategies, pricing models, product roadmaps, and operational efficiency.

Resources

AWS re:Invent 2017: GPS: SaaS Metrics: The Ultimate View of Tenant Consumption (GPSTEC308)
GPSTEC308-the Ultimate View of Tenant Consumption Slides
Calculating SaaS Cost Per Tenant: A PoC Implementation in an AWS Kubernetes Environment.

Best Practices:

• Capture low fidelity tenant activity metrics: Use packaged frameworks and tools that can capture and surface readliy available application and system insights with minimal instrumentation, injecting tenant context where possible.

• Create a complete view of tenant consumption: The SaaS application is fully instrumented with metrics that capture a range of tenant activity, feature usage, and resource consumption events. These metrics enable product managers, architects, and operations teams to build analytics views of this data to drive technical and business decisions.

• Instrument high-value workflows of the system with tenant-aware metrics: Targeted, high-value areas of the system are instrumented with metrics that provide insights on workflows and use cases that are essential to understanding the customer experience and consumption patterns of these high-value targets. Use analytics tools to analyze and surface operationally significant data.

Improvement Plan

Capture low fidelity tenant activity metrics

Inject web analytics into your web application, capturing tenant activity data that can be used to profile their patterns for accessing and consuming features in your SaaS application.

Instrument the services of your application with tracing mechanisms that captures and records request activity for backend services.

• Use AWS X-Ray or observability frameworks to capture and publish request activity with tenant context.

Aggregate the web analytics request tracking data to create a view of how tenants are consuming front-end and backend services.

• Store aggregated metric data in a warehouse (such as Amazon Redshift or Amazon S3), enabling product managers, operations teams, architects, and business leaders to leverage BI tools (such as Amazon QuickSight) to analyze trends that might inform business or technical strategies.

Create a complete view of tenant consumption

Publish metrics that cover the full spectrum of services and experiences in your application. Introduce metrics and use them to construct a detailed profile of the resources and services that are being consumed by the tenant as well as the performance data summarizes their end-to-end experience.

• Publish composite metrics that correlate to domain constructs. This might require combining common metrics data (storage, compute, latency, etc.) into a single metric that has more meaning to the business.
• Publish business metrics from your SaaS environment that provide insights into trends that are essential to the agility, innovation, and operational success of the organization.
• Capture and publish data that reflects the overall agility of your business. The metrics are less about tenants and more about operational trends.

Common metrics to gather include: cycle time, availability, release frequency, mean time to detection (MTTD), mean time to recovery (MTTR), and the defect escape rate.

Capture data that can measure the efficiency of your tenant onboarding experience, such as the number of tenants onboarded and the durations for provisioning new tenant environments.

Capture data that can drive the customer success experience, including information about time to value for new tenants and metrics that identify renewal candidates that might have descending levels of activity.

Publish data on changes in tenant state that are often driven by business events, specifically when tenants are changing their status (active/inactive) or tier.

Aggregate detailed metrics to a single repository that houses a unified view of all tenant metric data. Allow business and technical teams to use their BI tools to create analytics views of this data that best align with their needs.

For AWS based data aggregation, you can use Amazon Kinesis Data Firehose to publish metrics to Amazon Redshift or Amazon S3.

Instrument high-value workflows of the system with tenant-aware metrics

Instrument key workflows with tenant metrics

• Introduce detailed tenant activity and consumption metrics instrumentation that are essential to the gathering insights and can inform key business or technical strategies.
• Publish detailed metrics in a single repository that houses a unified view of all tenant metric data. Use these metrics to build an enriched view of tenant activity.
  • For AWS based data aggregation, you can use Amazon Kinesis Data Firehose to publish metrics to Amazon Redshift or Amazon S3.
    AWS re:Invent 2017: GPS: SaaS Monitoring - Creating a Unified View of Multi-tenant Health featuring New Relic (GPSTEC309)