REL 3: How are you testing the multi-tenant capabilities of your SaaS application?

Introduce automated testing that exercises and validates use cases specific to multi-tenant workloads. Focus on simulating tenant workloads and user experiences that are essential to the stability of your multi-tenant architecture.

Resources

Testing SaaS Solutions on AWS

Best Practices:

• Validate “noisy neighbor” scale and availability: Test various noisy neighbor conditions, assessing the system’s ability to identify and respond to scenarios where a subset of tenants places a disproportionate load on your system. Develop a suite of tests that assess the system’s ability to apply scaling, throttling, and tiering policies for a range of tenant tiers and profiles.

• Validate tenant isolation: Simulate interactions with your system to help ensure that the tenant isolation policies and practices are being successfully applied. Include tests that examine scenarios where a developer’s multi-tenant code could unintentionally cross a tenant boundary.

• Exercise key workflows under multi-tenant load: Identify a range of workflows that might be key to your customer experience, implementing tests that validate your system’s ability to support SLAs under multi-tenant load. Assess the system’s stability as tenants place a mix of loads at varying levels of tenant activity.

• Validate the scale and repeatability of tenant onboarding: Ensure that your tenant onboarding experience can reliably and repeatably onboard tenants with varying patterns and configurations. Verify that the onboarding process continues to meet target SLAs.

• Ensure that tenancy configuration changes are successfully propagated: Validate that the system is correctly applying and propagating changes to tenant billing accounts. Changes to account state, such as status and tier, must be shared between the billing system and your SaaS environment. Tests must validate that synchronization of this state is successfully processed.

Improvement Plan

Validate “noisy neighbor” scale and availability

Create tests that simulate various noisy neighbor tenant scenarios for various tenant personas, validating that availability policies are successfully applied to different tenant profiles and tiers.

Validate that the noisy neighbor tests exercise the throttling and scaling policies of your system, ensuring that individual tenants won’t impact the availability of your system.

Validate that the SLAs associated with tenant tiers are enforced by the SLA and scaling policies of your system.

For systems that have public APIs, validate that tenants cannot saturate the API and impact the availability of your system.

Validate tenant isolation

Create tests to verify that the application is enforcing the tenant isolation policies of your system. These tests should search for potential opportunities to subvert the isolation model and ensure that misbehaved users cannot cross a tenant boundary.

Inject tenant tokens that attempt to simulate a SaaS identity.

For applications that are using shared libraries or frameworks to enforce isolation across the system, introduce tests to exercise these frameworks and validate that they accurately apply tenant isolation policies.

Create tests that attempt to change the tenant context by injecting a new tenant identifier. Verify that the injection is blocked from crossing a tenant boundary.

Exercise key workflows under multi-tenant load

Create tests that validate that key workflows of your system respond effectively to the continually evolving load profile of your multi-tenant environment.

Simulate multi-tenant load patterns for key system workflows, ensuring that tenant experience and SLAs are not impacted during different scaling states.

Validate that the SLAs continue to be met even as the system is scaling up or down to meet the demands of a shifting multi-tenant load.

• Validate the system’s availability with a tenant count that represents a meaningful load on the system.
• For systems that include billing integration, create tests that exercise the billing provisioning experience and its ability to successfully provision new billing accounts in the billing system.
• For system that rely on third-party integrations that are configured as part of onboarding, create tests that validate that the provisioning experience successfully configures and provisions these integrations.

Validate the scale and repeatability of tenant onboarding

Enable operations to configure tenant alerts and alarms

Create tests to validate that the automation of onboarding is successfully provisioning new tenants with the appropriate infrastructure, configuration, isolation policies, identity footprint, and so on.

For systems with specific onboarding SLAs, introduce tests that verify that the onboarding of various tenant configurations will meet your target onboarding SLA requirements.

For systems that require tenant validation as part of onboarding, create tests that exercise this validation process and ensure that messaging and first-time entry of the tenant function as required.

Ensure that tenancy configuration changes are successfully propagated

Create tests to validate that changes in a tenant’s state are properly applied to the running system.

Validate that the disabling and reenabling of tenants is successfully enforced as tenants attempt to access your system.

Validate that changes to a tenant’s tier is successfully detected and applied to the system, enabling or disabling access to capabilities associated with the tier boundary.

Introduce tests that ensure that the limits and policies associated with each tenant tier are being enforced by the system.

Validate that deleted tenants are correctly processed by the system (decommissioned, data archived, etc.)