APPROACH
The approach to address the client’s challenge included:
- Setting up a central Cloud-Data warehouse as a single source of truth
- Laying the foundational data-architecture which classified requests to zones based on repeatability and usage
- Introducing a metadata driven- UI based governance framework to capture data-requests
- Building re-usable frameworks to enable the platform to serve personas with data requests, platform usage requests and insights requests
- Harnessing the power of Data-Bricks on Azure for workload based dynamic, auto-scalable ingestion layer creation
KEY BENEFITS
- Azure native solution easily integrable with Rest-API driven governance framework on Collibra for organization wide transparency
- Handles, changing records, unified streaming, and in-built data-quality layer using the delta-lake feature of data-bricks
- Loosely coupled frameworks for logging/Alerting & monitoring which are cloud-agnostic and can be re-configured and used as need be
RESULTS
- The time to on-board data decreased to a 24-hour cycle
- Overall storage and compute cost slashed by 30%
- One platform – For Bring your own data, Platform & data services and Insights services