COST 1: How do you measure the resource consumption of individual tenants?

Attribute infrastructure consumption to the individual tenants of your system. Surface metrics that can be used to profile the activity and consumption patterns of tenants, gathering enough insight to be able to approximate and calculate the infrastructure footprint of each tenant that is using your SaaS application.

Resources

SaaS metrics deep dive: A look inside multi-tenant analytics
AWS re:Invent 2017: GPS: SaaS Metrics: The Ultimate View of Tenant Consumption
SaaS Analytics and Metrics: Capturing and Surfacing the Data That's Fundamental to Your Success (ARC325)

Best Practices:

• Approximate tenant consumption: A simple metric (for example, number of requests) is used to create an approximate allocation of consumption for each tenant. Manual processes are used to correlate this tenant allocation with your AWS bill to arrive at a rough estimate of the cost per tenant.

• Use tenant consumption insights to shape operational and architectural efficiency: Tenant consumption data is used to provide actionable insights to operational and architecture teams, enabling them to introduce policies and strategies that can enhance to analyze and improve the efficiency of a multi-tenant system.

• Build a rich view of tenant consumption: The consumption of the application’s individual resources is metered and attributed to each tenant. This data is aggregated and used to create a detailed estimate of tenant consumption.

Improvement Plan

Approximate tenant consumption

Approximate consumption based on general tenant activity

• Use tools to capture the number of requests that are being generated by each tenant. Use Amazon CloudWatch, AWS X-Ray, or language-specific frameworks to capture and publish tenant activity. 
  Use Amazon API Gateway Monitoring tools in AWS
• Aggregate the published data for a window of time and approximate tenant consumption based on the general tenant activity.

Calculate an approximation of cost per tenant

Use AWS cost analytics or APN Partner tools to assemble a summary of AWS costs for a given billing period.

Apply approximations of tenant consumption to billing data to calculate a cost per tenant.

Select approximation models that are likely to correlate to those services that represent the key resources of your system (cost and/or efficiency).

Use tenant consumption insights to shape operational and architectural efficiency

Incorporate tenant consumption into operational views

• Ingest tenant data into operational dashboards, providing views of tenant consumption activity.
• Create views of consumption that can be used to assess the system’s ability to efficiently respond to the continually changing loads of your multi-tenant environment.
• Create views of consumption that can identify tenants that are most actively consuming the system’s resources.
• Create views of consumption that can profile how tenants are placing load on the key elements of your architecture (storage services, compute resources, etc.).
• Create views of consumption that can profile how tenants are placing load on individual application services.
• Enable operations teams to view consumption for individual tenants or tenant tiers.

Use tenant consumption data to proactively manage tenant health.

• Create policies that can proactively identify tenants that may be approaching SLA thresholds.
• Create policies that can proactively identify tenants that are having their experience throttled.

Optimize architecture based on tenant consumption data

• Use tenant consumption profiles to refine scaling, tiering, and throttling strategies to create a more efficient multi-tenant operational footprint.

Build a rich view of tenant consumption

Instrument your application with detailed tenant insights

• Identify the resources that need to be metered. Consider how each of these resources correlates to your overall AWS bill and how they contribute to overall expenses. This helps shape the granularity of your resource metering strategy.
• Add metering instrumentation across the stack of your solution, and publish consumption metrics for each resource type. The nature of the consumption metrics varies based on the type of resource being consumed.
  • Identify the dimension of tenant activity that best models tenant consumption for a given resource (microservice, storage, compute, etc.)
  • For API-based services that are driven more by activity, use counts of tenant requests to model tenant consumption.
  • For throughput or CPU-intensive resources, use latency to model tenant consumption.
  • For storage services, model the compute (if applicable) and data footprint of each tenant.
  • For serverless environments, model consumption based on the number of invocations of a given function.

Aggregate and summarize consumption metrics

• Create automation to analyze detailed insights and summarize tenant consumption for the full footprint of your environment.
• Aggregate tenant consumption insights with the AWS data warehouse and aggregation services that best support your needs. Amazon Kinesis Data Firehose, Amazon Redshift, Amazon S3, Amazon Athena, Amazon CloudWatch, and Amazon OpenSearch Service are among the services that can be used to aggregate and analyze your metric data.

Calculate an approximation each tenant’s consumption

• Use aggregated tenant consumption data to construct a summary that attributes a consumption percentage to each tenant of your system.
• Introduce policies/rules that determine how tenant consumption data will be weighted and transformed into a total consumption percentage for each tenant.
• Use AWS cost analytics or APN Partner tools to assemble a summary of AWS costs for a given billing period.
• Apply approximations of tenant consumption to billing data to calculate a cost per tenant.
  Calculating Tenant Costs in SaaS Environments