The major hurdle for any machine learning model to succeed, in the majority of cases, is the lack of adequate data. If the model fails even for an instance, for sure the team would be looking for a new set of data that is compatible with the context. One particular type of data that can help ML engineers is dark data. One may have many questions like, what is dark data? How is it different from big data? Big data is data that is easily accessible because it is visible and unlike dark data which is hidden. Though we can get hold of zillions of data sets that are available, one cannot say that it completes the information set.

All the information that a person can dig out on Google or Bing search constitutes only 5% of the total data. The rest of the 95 % of the data, the unindexed data exists in the deep web. Financial records, medical and legal documents, government and organization specific data repositories are some examples. And as data like pictures and videos add to this data, it becomes a complex amalgamation of unstructured data. So, what is stopping companies from using dark data? The problem goes beyond data visibility as more and more companies are into putting their data behind the firewall.  Unstructured data presents a variety of challenges pertaining to data management, SLAs around data discovery and classification, and the lack of tools and infrastructure to process the data. Apart from that, there is a certain inertia in the business management towards exploring dark data as it can jeopardize the normal business process putting the team at unease which is used to working with traditional data.

Is there a solution at all? Or will it be a wild goose chase forever? With companies harboring their data on public cloud platforms like Amazon, Google, and Microsoft, and service offerings such as PaaS and SaaS, the journey towards dark data analytics is going to get smooth.