How ProCogia Reduced Azure Storage Costs by 95% for a Marine & Logistics Client

Introduction

Storage costs in cloud environments can quickly become unmanageable. For a Marine and Logistics client, ProCogia implemented a Delta Lake vacuuming strategy on Azure, reducing storage from 40TB to 500GB—a 95% cost reduction. Here’s how we did it.

Challenge  

ProCogia’s client faced significant expenses related to their Azure resource utilization, with storage costs being a primary concern. While they were aware of the substantial amount of data stored in their Azure Blob Storage, they were keen to explore strategies to reduce these costs. Our task was to identify and implement effective solutions to optimize their storage and lower expenses.

Approach  

To address the client’s storage cost concerns, we leveraged Delta Lake’s vacuum operation on their Blob Storage. This process systematically removed obsolete and unreferenced data files from Delta tables, which had accumulated due to versioning and transactional updates. The following steps need to be followed to perform vacuum operation  

• Identify the Azure Blob Storage container containing the Delta tables and configure a retention period of 30 days. This retention period was specified by the client to ensure that only files older than 30 days are removed during the vacuum operation. The retention duration can be customized based on specific requirements to balance data availability and cost optimization.  

• Develop a pyspark script in Databricks to perform the vacuum operation.  

• Create a mount point on the Blob Storage account by specifying the container name and securely accessing the storage account key through Azure Key Vault. Mount points in Databricks enable seamless access to files stored in the Blob Storage account. The following script can be used as a reference for setting up the mount point.