Data Virtualisation and why has it Evolved?

Data virtualisation enables businesses to access, manage, incorporate, and aggregate data from different sources independent from its physical location or format in real-time.

As per The Data Management Association International (DAMA) and Data Management Body of Knowledge (DMBOK), "Data virtualisation enables distributed databases, as well as several heterogeneous data stores, to be accessed and viewed as a single database. Rather than physically performing ETL (Extract, Transform, and Load) on data with transformation engines, data virtualisation offers performing data extraction, transformation, and integration virtually."

Why Data Virtualisation Evolved?

With the fast-changing business world, information has become an essential production factor. Data-driven decision-making is a tool to withstand the growing competition around global industries and markets. Exploiting the power of business intelligence (BI) or analytics and automating workflows is one way for businesses to generate new revenue while reducing costs by enhancing the efficiency of their daily processes'.

Today, enterprise data is preserved in different locations and comes in various, fast evolved forms like:

• Social media or website data such as Facebook, Twitter or Google Analytics
• Relational and non-relational databases such as MySQL, Amazon Redshift or MongoDB
• CRM or ERP data such as SAP, Oracle or Microsoft Dynamics
• Flat files such as XML, CSV or JSON
• Cloud or software-as-a-service applications such as Netsuite, Salesforce or Mailchimp
• Data lakes and enterprise data warehouses
• Big data

Businesses have been facing increasing volumes of data accompanied by growing data variety and velocity. This often leads to challenges like obtaining trustworthy data quality, time efficiency in data management, and self-service capacities for data users. Conquering these challenges efficiently and effectively became critical for modern enterprises' success.

Data virtualisation helps businesses to deal with these challenges using the full potential of their data. Data virtualisation's primary concept is to break free from the requirement of knowing every technical detail of the data like its exact physical location or its root format. It enables integration and aggregation of data from disparate physical sources and diverse formats within one view without moving the data into central storage. As all data remains in the source systems, data virtualisation builds a virtual/logical layer to enable real-time accessibility with the possible manipulation and transformation of data in virtual views. This virtual layer permits data management that is simpler and more efficient. Data virtualisation tools usually make data accessible with SQL, REST, or other standard data query methods, regardless of the source's file format. It further simplifies data management efforts; however, this depends on its solution and still isn't standardised.

How Data Virtualisation Works

The centrepiece of a data virtualisation application is the virtual or semantic layer, enabling data or companies to manipulate, join, and calculate data independently from their format, sources, and involved metadata appear in one single user interface. This virtual layer further permits to organise data in different virtual schemes and views. Business users can easily enrich the raw data from the source systems with their business logic and ready the data for analytics, reporting, or automation processes.

The virtual layer ideally also covers data governance and metadata exploration capacities. It is crucial to note that every tool does not include this functionality. For instance, with a sophisticated user-based permission management, the virtual layer builds a single truth source across the entire organisation in a fully compliant and secure manner. Authorised users can access all relevant data that way from one single point in one tool. Consequently, the creation of data silos is avoided.

Data virtualisation generally does not retain data from the source systems. It stores metadata to feed the virtual views and allows the creation of individual incorporation logic. The critical aspect of data virtualisation is delivering data in real-time to any front-end or application like business intelligence tools, microservices, or custom programs. It works by fetching the data in real-time from the underlying source systems.