Cost and Performance Scaling for Data Warehouse for Launch

While cloud data warehouses are capable of reaching immense scale, they should be adopted as early as possible. This allows them to evolve in tandem with growth of business process and volume. Our client was facing an unusual situation: They relied on a data warehouse to serve multiple functions, but anticipated a 100x surge in user volume accompanying the launch of a multimillion-dollar partnership. They needed assurance that their data and analytics function would scale smoothly and cost-effectively.

Motivation

The client was anticipating up to 100x user growth accompanying the launch of their multimillion dollar strategic partnership, but forecasts showed unacceptable cloud computing costs for their data warehouse and ETL. Additionally, they didn't want to be distracted by unexpected outages when they most needed visibility into the performance of the launch. With the launch date already set, the clock was ticking to ensure uninterrupted and cost effective service.

Approach

Extrasensory¹ began with an overview of the business and collected considerations from each of the functions that consumed data and analytics. After reviewing the existing solution, it became clear that datasets, queries, and reports were not well separated between functions, so an inappropriately high level of accuracy and performance had been assumed across many features. Additionally, many aspects of the solution were acceptable as "minimum viable", but needed to be reinforced to handle scale. Fortunately, the technology selections were more than adequate, so we were able to ensure robust scaling without replacing any components as follows:

• Inventory of consumption patterns across functions, the intentions attached to them, and implied tolerances.
• Understanding of existing user personas, their use profiles, the marketing strategy behind the launch, and how new users might contrast.
• Optimization of all reporting, dashboard, and maintenance queries and ETL making use of existing features intended for massive scale.
• Monitoring and adjustment of solution through the critical launch phase.
• Organization and documentation of all supported uses according to function and key trade offs made in implementation.

Results

Extrasensory completed all needed upgrades with time to spare. On launch day, users began pouring in, and the newly upgraded analytics solution worked as designed. ETL services autoscaled to meet demand in minutes, dashboards stayed updated and correct, and real-time analytics were uninterrupted. The client stayed focused on their rapidly developing business and maximizing the value of their new users.