Now more than ever, in the era of smart working, the future of data analysis involves the adoption of cloud-native solutions.
In the past decade, data warehouses have been the mainstay of the analysis of business data: administrative, marketing, and sales departments have based their budgets, planning, and reporting on these centralised data platforms fed by all the information systems.
With the advent of cloud-native data analysis solutions offered by Google, Microsoft, and AWS, more and more companies are embarking on the path of modernising their data warehouses. A path towards what all the analysts are calling Augmented Business Intelligence.
We discuss this phenomenon with three experts from our Datwave centre of excellence on cloud technologies and data management.
What Does It Mean to Modernise Your Data Warehouse?
Marco Pesarini, Partner at Datwave, explains what it means to modernise your data warehouse and why our customers are considering these solutions with interest.
“Modernising your data warehouse means bringing your data analysis platform to cloud-native functions, fully managed and offered in pay-per-use mode. A household analogy would be what we did with email servers, which were replaced by cloud services like Office 365 or Gmail. No more hardware or licences, which often weigh down the IT budget; no more operating costs or disruptions arising from your data centre; data, reports, and analyses are always available through the network and with a cost model that is based purely on consumption.”
Google BigQuery, Azure Synapse, and AWS RedShift are the turnkey cloud solutions for building a data platform, without worrying about the need for a complicated IT architecture to support it.
Cost Savings and Beyond: The Benefits of Modernisation
Many of our customers are proceeding to modernise their data warehouses for cost reasons: our benchmarks report savings of up to 50% in the Total Cost of Ownership (TCO) compared to traditional solutions. The lion’s share of this reduction is attributed to savings in terms of software licences that are replaced by leaner pay-per-use models that can improve the migration project’s payback time to below 18 months.
But the benefits of modernisation go far beyond savings: customers who start exploring the topic realise how a modernised data warehouse solution allows analysis that isn’t possible with traditional data warehouses.
Real-Time Data Management
Cloud solutions enable much greater real-time management of reporting, thereby surpassing the “delayed” data warehouse model, where data is often 24 hours old and many reports are updated on a weekly or monthly basis. Cloud-native solutions adopt big data uploading tools which offer a near real-time supply from operational systems to the data warehouse, providing – to give just one example – fresh accounting data for the day.
Handling Unstructured Data with Ease
Cloud-native data warehouses are also more open to managing unstructured data originating from outside the company. For example, think of data from IoT sources and social media, for a company that wants to enrich its marketing analyses with sentiments from the network or with data collected from connected objects in the field. This data often originates in the cloud and comes in highly significant volumes and variety, so managing it directly on the cloud helps drastically reduce transfer costs and enables flexible resource management.
Integration of Machine Learning and AI
Last but not least, cloud data warehouse solutions natively integrate the most advanced machine learning and artificial intelligence features on the market from Google, Microsoft, and Amazon. These features can be inserted directly into the company’s reporting to transform the old data warehouse reports into advanced analysis, deduction, and forecasting tools.
“All this is offered by the cloud, accessible with just a few clicks from our PCs, tablets, or smartphones, from wherever we are. This is why we’re talking about an epochal transformation.”
Challenges in Modernising Your Data Warehouse
Eros Frigerio, Manager and Solution Architect at Datwave, explains the main challenges our customers face in modernising their data warehouses and the best approach to follow.
“As my colleague Marco explained, choosing to modernise your data warehouse in the cloud brings about numerous benefits, but at the same time it entails a series of challenges to tackle when designing and implementing the solution.”
Main Challenges:
- Meeting Security and Compliance Requirements: In the world of the public cloud, you have to pay close attention to the security and compliance requirements (e.g., the GDPR). We need to find solutions that provide the best compromise between operational agility and compliance.
- Adopting a Cloud-Native Approach: Data models and data ingestion and transformation pipelines must be optimised for the cloud. Approaching the cloud with overly traditional solutions can be counterproductive.
- Implementing Data Governance: The configuration and implementation of cloud services require a data governance framework to control the data along its transformation path and ensure expected technical and business quality levels.
Datwave’s Approach to Data Warehouse Modernisation
To ensure and support the evolution of the cornerstones of the new data warehouse, our approach consists of three key phases which examine four areas of intervention: the choice of cloud access strategy, the design of the reference architecture, data modelling, and roll-out of analytics solutions.
- Assessment Phase: This step analyses the current situation to evaluate technical and economic feasibility, organisational maturity, and data availability to identify the best strategic direction.
- Conceptualisation Phase of Foundation: Creating the main founding pillars of the new data warehouse, especially in terms of architecture, data model, and implementation guidelines.
- Implementation Phase: Execute, measure, and improve. This involves activating and configuring the cloud architecture components, developing the new data model, and implementing the analytical and reporting use cases.
Technologies Enabling Modern Data Warehousing
Luca Natali, Associate Data Engineer at Datwave, explains the main technologies enabling the development of a modernised data warehouse platform.
The main technologies for modern data warehouses are cloud-native services that offer complete management from raw data to visualisation dashboards. These services can be grouped into five basic categories:
- Raw Data Layer: Object-storage services for structured, semi-structured, and unstructured data. Examples: Amazon S3, Google Cloud Storage, Azure Blob Storage.
- Data Integration: Managed and scalable processing tools for distributed data processing. Examples: Amazon Glue, Azure Data Factory, Google Dataflow.
- Data Warehousing: The primary component where data is saved in the table format. Examples: Amazon Redshift, Google BigQuery, Azure Synapse Analytics.
- Visualisation/Reporting: Services for developing reports and data visualisation dashboards. Examples: Amazon QuickSight, Microsoft Power BI, Google Data Studio.
Examples include: Amazon QuickSight, Microsoft Power BI, Google Data Studio.
“Even for customers who have already brought their data warehouse to the cloud, modernisation can offer further benefits on top of those already obtained with the first migration of the data warehouse.”
Modernising your data warehouse with Datwave ensures you gain greater flexibility, platform reliability, and a significant reduction in operational tasks, making your data management chain—from ingestion to reporting—more efficient and robust.
