Businesses use automation software to visit various pages online and download the publicly available data in them. It is then processed to create various business intelligence insights. Oxylabs is a premium proxy and public web data acquisition solution provider, enabling companies of all sizes to utilize the power of big data. Analytics Insight has engaged in an exclusive interview with Julius Černiauskas, CEO of Oxylabs.

Kindly brief us about the company, its specialization, and the services that your company offers.

Oxylabs is a premium proxy and public web data acquisition solution provider, enabling companies of all sizes to utilize the power of big data. In simple terms, we provide all the necessary tools and solutions for businesses looking to extract publicly available data on a large scale.

A significant part of our daily processes is to support web scraping. Businesses use automation software to visit various pages online and download the publicly available data in them. It is then processed to create various business intelligence insights.

Our product portfolio includes both complete out-of-the-box solutions such as the Scraper APIs and supporting infrastructure such as proxies. With the former, our clients only have to send requests to our endpoints, and they get the data sent back to them. With proxies, they have to develop automation software themselves as we only provide the know-how and the supporting blocks.

Kindly mention some of the major challenges the company has faced till now.

Providing web scraping solutions is always fraught with challenges. One part of the equation is that the industry is relatively new. Being a pioneer is an amazing feeling, but it makes challenges so much more difficult as no one can help you out. You have to figure out how to solve issues all on your own. Being a front-runner means creating the pace and practices. It means a lot of other companies are looking up to us and following the models we create.

Another part is the technical complexity of the process. We pride ourselves on helping clients achieve an uninterrupted flow of data. Maintaining infrastructure, creating scraping code, and everything else is extremely resource-intensive. With the added novelty of the industry, we are always facing avant-garde challenges.

Finally, web scraping hasn't established itself in the public consciousness yet. There's scarcely any legislation across the world and the overall perception can be seen as somewhat negative.  While our legal team has been doing tremendous work keeping us on the right side of the law, they can't change global perception. Yet, at the same time, web scraping is becoming an essential part of digital businesses.

How do you see the company and the industry in the future ahead?

Another aspect of not being established in the public consciousness is that there's little awareness of the possibilities. As we are the leader of ethical solutions and tools in the industry, we are working towards bringing ethical web scraping out of the darkness and turning it into something that is used for the common good.

We have developed partnerships with universities, NGOs, and other organizations by giving them access to our solutions or otherwise supporting their goals. We help them by providing tools, sharing technical expertise, creating tools to tackle specific problems, and much more regularly.

One of our most successful projects has been developing an AI-driven scraper that identifies harmful and illegal content on the internet, aiding the Lithuanian government institutions in their work. We see web scraping becoming something not only businesses use to further profit incentives, but something that works for the common good of all.

We've been working on other initiatives that would push web scraping further into the public consciousness. Oxylabs hosts an annual conference, OxyCon, and we do a lot to share our knowledge on the legal and technical side of things with the outside world.

How is your company helping customers deliver relevant business outcomes through the adoption of the company's technology innovations?

Innovation is an essential part of Oxylabs. We invest heavily in R&D, primarily in the field of artificial intelligence and machine learning, which sets us apart from our competitors. As I've mentioned before, web scraping is a technically complicated process, involving many different parts that can be optimized almost to no end.

A part of such optimizations is parsing. Generally, when you retrieve data from the web, it's messy and unreadable. Parsers, and other automated software, make the same documents easier to understand and process. Unfortunately, writing them is extremely costly and time-consuming.

We have invested a lot of time in innovating our field through machine learning and now we have an industry-leading Adaptive Parser. It removes the requirement of writing parsers for eCommerce product pages, allowing us to significantly cut costs and provide higher quality data to our clients.

AI is projected to be the next market. How is AI contributing to the making of your products and services?

AI is at the forefront of our innovation push. We have established an AI/ML Advisory Board with 5 leading business (e.g., from Stripe) and academia (e.g. from MIT, NASA) experts with whom we maintain constant collaboration. They help us in driving the progress of AI in web scraping further than we could do ourselves.

We believe AI will play a significant role in the coming years by making parts of web scraping more accessible than ever before. Web scraping can be split into two large chunks: the discovery phase and the collection phase.

The discovery phase is accepting a task from an interested party. A business may be interested in the prices of specific countries within a set region. I see AI aiding significantly in the discovery phase by either allowing vague queries to be easily acted upon or by enabling better finding processes.

Secondly, the collection phase is, as we already see, greatly enhanced by the development of machine learning models. We can take out the most costly processes and completely automate them through the use of artificial intelligence. While there may be doubts about AI helping the discovery phase, I have a resolute belief that it will tremendously change the face of data collection.

How can businesses efficiently extract the value from data, without increasing cost and complexity?

A large part of optimizing data practices is narrowing down collection practices. There's enough data to fulfill any goal and to support any hypothesis, and it's surprisingly easy to over-collect information.

Most of the software we use in daily operations such as tracking tools, CRMs, etc., provide some sort of data collection feature. As a result, there's plenty of data to go around at all times. What businesses should do is create clear goals for the data they already hold.

I prefer to think of data as something that should have intention. Asking oneself "why are we collecting that data" is a surefire way to get down to all the necessary details. If you have to find a complicated or long-winded explanation on why it is necessary, maybe it isn't all that after all.

What are some of the challenges faced by the industry today?

Web scraping remains a process employed by a select few while it can be of tremendous benefit to all. At Oxylabs, we're pushing for ethical web scraping to be recognized as a common good and aim to make it accessible to non-developers.

As such, the two challenges I see most pressing is public legitimacy and accessibility. Legitimacy has been fully achieved in the business world, but it has to spread further as web scraping can do much greater things. I believe complete public legitimacy will be achieved over time through developing partnerships with government institutions and academia. Accessibility is on us – we strive to innovate in making web scraping easier to use and more intuitive.