Driving a New Era of Cloud-Enabled Big Data Analytics

Big Data analytics and cloud computing are the two technologies that are currently top of mind for organizations. The adoption of big data analytical capabilities using cloud delivery models offers the promise of providing valuable insights that create competitive advantage, spark new innovations and transforms industries significantly. Chandra Ambadipudi, CEO of Clairvoyant in an exclusive interaction with Analytics Insight shares how these technologies will bring a new era of cloud-enabled real-time analytics.

Analytics Insight: Kindly brief us about Clairvoyant, its specialization and the services that your company offers. 

Chandra: Clairvoyant was founded in December 2012 and is headquartered in Chandler, Arizona. It started as a product analytics company for the higher education sector, working with universities to understand student recruitment and retention, especially as universities moved to an online learning model, which provides rich signals generated from student involvement and engagement in virtual learning environments.

Today, the company specializes in building big data solutions for enterprises and provides technology consulting and services globally. Clairvoyant helps organizations build innovative products and solutions using big data, analytics, and the cloud. With deep vertical knowledge combined with expertise on multiple, enterprise-grade big data platforms, the Clairvoyant team creates and supports purpose-built solutions that address problems unique to specific industries such as financial services and healthcare.

Clairvoyant's team consists of 350+ experienced big data professionals with backgrounds in design, software engineering, analytics, and data science. Recently, leveraging the core competency of solution engineering, Clairvoyant expanded its footprint in the data security space and launched Kogni, a data security product that enables companies to discover sensitive data in enterprise data sources (both cloud-based and on-premises), to secure data as it is ingested, and to continuously monitor data sources for possible breach and policy violations.

Apart from Kogni, the core services Clairvoyant offers are solution engineering, managed services, product design, application & product development and system integration across multiple big data platforms and the leading Hadoop distributions.

Analytics Insight: Please share your journey in the analytics industry and what are some of the analytics solutions that you have previously worked on?

Chandra: I have roots and extensive experience in the higher education space, working on technology innovation across the entire student lifecycle. Before starting Clairvoyant, I was Vice President of Engineering at Apollo Education Group, which is the parent company of the University of Phoenix.

During my tenure at Apollo, I led a team of 300 engineers, business analysts and project managers, and was responsible for all aspects of the enterprise systems that the University of Phoenix and other Apollo subsidiaries used. I also led the technical team that implemented Apollo's first predictive analytics project code-named "P-wave" that focused on student retention.

I was responsible for various enterprise-class systems that included Enrollment and Admissions, student scheduling, attendance management, the SIS (Student Information System) at the center of UoP operations, Call Center operations and the first version of the mobile platform used by University of Phoenix.

Analytics Insight: With what mission was Clairvoyant set up? In short, tell us about your journey since the inception of the company?

Chandra: Clairvoyant's founding vision was to be a leader in building predictive analytics products and working on the cutting edge of big data technology. Given the founding team's deep experience and expertise in software engineering, big data and technology, this was a natural choice for us and a seamless move from the higher education technology space. A clearly defined business use case helped crystallize the initial plan.

With our background and deep experience in the higher education vertical, we identified a business need amongst both large and small academic institutions to improve student retention. Based on the work we had done at Apollo, we realized that creating a predictive analytics driven product with custom algorithms and data science modeling tools would address this business need. This was the genesis of Blue Canary, the first SaaS-based product built by Clairvoyant, and we created it with this specific use case.

Blue Canary's key component was a custom-built Data Science Modeling tool (DataBrew) that enabled automation of predictive model building. This significantly reduced the time it took to build models from days and weeks to hours and allowed us to easily scale the product across 1000+ customers.

In November 2015, BlackBoard (owned by private equity) acquired Blue Canary. This was during a time where academic institutions were rapidly adopting technology that enabled them to improve learning and performance. BlackBoard was specifically seeking technology that could power the next generation of their BlackBoard Learning Analytics and Predict platforms. As part of the transaction, the Clairvoyant team was involved in the integration with multiple institutions across multiple data sources with different databases and APIs.

Analytics Insight: Could you highlight your company's recent innovations in the AI/ML/analytics space?

Chandra: Based on our experience at Clairvoyant helping enterprises build big data systems that pulled both structured and unstructured data into big data platforms, we realized big data came with an increased risk of data breaches. Additionally, big data expands the compliance requirements to cover the risks associated with centralizing large volumes of data.

In today's enterprise, knowledge and management of sensitive enterprise data is often siloed across departments and IT infrastructures. All of these issues created the challenges around monitoring data security in a centralized manner – especially as the location of sensitive data storage constantly changes.

Many enterprises focus exclusively on perimeter security. Unfortunately, as demonstrated by an endless stream of high profile data breaches at major companies, the question of firewall breach is no longer if, but when. We developed Kogni to help solve all of these issues with tools that enable companies to discover sensitive data in enterprise data sources (cloud-based and on-premise), secure data as it is ingested, and continuously monitor data sources for possible breach and policy violations.

Kogni is built upon three core functionalities and supporting capabilities:

Discover: Kogni scans enterprise data sources, leveraging machine learning and computer vision to identify sensitive data stored in text and images. The platform supports a broad range of data sources including Hadoop, NoSQL, S3, and RDBMS. It uses purpose-built classifiers to identify a variety of sensitive data such as credit card, social security numbers, and more.

Secure: Kogni transparently secures data as it is ingested into Hadoop with zero code change and little performance overhead. The platform supports masking, encryption, redaction, and tokenization of sensitive data based on simple configuration and plugins for Sqoop, Spark, Kylo, Nifi, Streamsets, and many more.

Monitor: Kogni continuously monitors data sources and user behavior for anomalies and triggers detailed alerts on sensitive data proliferation and policy violations.

Analytics Insight: Which industry verticals are you currently focusing on? And what is your go to market strategy for the same?

Chandra: Clairvoyant is focused on the financial services and healthcare verticals. The go-to-market strategy in these verticals is driven first by identifying key industry problems based on our deep vertical expertise in these two verticals. The second part of the strategy is leveraging the core skill sets that we have: data engineering, data science and product development, and cloud engineering and the management of large big data systems in an enterprise environment to build cutting-edge big data solutions.

Over the past few years, we have built a significant knowledge base and use cases in these two verticals working with industry leaders. This knowledge base is the entry points into helping large enterprises identify their own use cases. After building big data systems for these two verticals, we realized that attention to sensitive data held in Hadoop-like systems was low and needed a custom solution to address basic needs like auditing and compliance.

Clairvoyant's deep understanding in data hubs and data lakes from a data security perspective and our skills to crawl into the data footprint, highlighting how data interacts with people, IT systems and business processes and putting this information into a data catalog is what led us to build Kogni. Kogni helps secured and monitor the data footprint that we understand very well and is a key part of our go-to-market strategy for the two data sensitive verticals of financial services and healthcare.

Today Clairvoyant is focused on building financial services solutions for well-known and mid-sized banks, taking advantage of Kogni. We are also developing a healthcare services practice for genomics companies that are developing personalized medicine offerings.

Analytics Insight: Kindly share your point of view on the current scenario of big data analytics and its future. 

Chandra: Most companies have realized that big data projects are hard and many get bogged down under the weight of a huge amount of unstructured data. Adoption of Hadoop has helped companies address this issue and demonstrate ROI from their big data analytics projects. But Hadoop has its issues. It offers a steep learning curve and can strain available IT resources.

Scaling Hadoop can also be a challenge because it requires investment in physical infrastructure, which can further strain those resources. That's why many companies are moving to cloud-based solutions. That's a good solution for companies that want a more agile way to scale, but there are still challenges related to ensuring performance, accessibility, reliability and scalability of data. Data security and an ever-growing list of industry regulations also complicate cloud-based Hadoop adoption.

In the near future, AI, VR/AR and IoT will generate more data than ever before. Companies today need scalable solutions to prepare for this onslaught.

Analytics Insight: How are disruptive technologies like big data analytics, AI/Machine Learning impacting today's innovation?

Chandra: This is an unprecedented time of change in which many technologies are coming together at once. For instance, although there have been huge strides in the role of big data analytics in recent years, we are just seeing the dawn of the global deployment of IoT, which will greatly increase the amount and type of data businesses can use. This is shifting the focus from maintaining business functions to warding off potential disruptions and discovering new consumer touch points.

AI and machine learning, meanwhile, are also shifting the focus to meeting consumers' needs to anticipating them. But rather than technology, it is consumers themselves that are the primary disruptors. We all get used to new technologies and touchpoints quickly and expect businesses to catch up. Consumers — both traditional and B2B — are intolerant of lags in service, mistakes and will not let you waste their time. The onus is on all of us then to learn to use these tools quickly and effectively.

Analytics Insight: The industry is seeing a rising importance of business and technology enablers like virtualization, convergence, and cloud. How do you see these emerging technologies impacting your business sector?

Chandra: The focus for all tech and business activity has shifted to mobile. Mobile Internet use surpassed desktop Internet use in 2016 and Google now gets more searches from mobile than desktop. Mobile technology is also evolving to crunch and analyze data locally rather than send it back to the cloud. This is leading to a more distributed computing environment.

Virtualization means that software can also be moved closer to the edge. The fact that many IoT devices need to act in real time and can't afford a cloud-imposed lag, means that we are entering a new era in which the cloud will no longer be as dominant and more computing will occur at the edge. For businesses, this means that it will be possible to execute more robust computing on IoT devices, especially with the emerging 5G network. The next era will be one of extreme personalization for consumers and extreme efficiency for businesses.

Analytics Insight: What is your leadership mantra?

Chandra: My focus and mantra since founding Clairvoyant has been to foster a culture of empowerment and innovation where the company stays at the leading edge of the technology curve, and most importantly – focuses on ensuring the success of our customers.

Analytics Insight: How do you see your company and the industry in the future ahead?

Chandra: Clairvoyant's vision is to be a "customer outcomes focused company driven by engineering excellence in the Big Data space" with plans to expand into new markets in Europe and Asia. Clairvoyant will continue on expanding their services on the Hadoop platform, and help organizations with the implementation of AI-driven products. With Kogni, we are increasing our capabilities in anomaly detection, improving our AI engine to help catalog sensitive data in more efficient ways, and finally, provide ease of compliance with regulatory frameworks such as GDPR.