Introducing the Concept of Augmented Reality (AR) based Ground-Zero-Analytics – Data Visualization in Real Environments

Analytics as a concept is not new and has been there since the evolution of human being. However, in the past 20 plus years, there has been a systematic approach to Analytics based on data stored in a structured format, in computers. This is the concept of data-driven analytics. Initially, it was in the form of simple reports on operational data. Later with the advent of Storage technologies, there arose the concepts of historical storage called Data Warehouses and Data Marts. This gave rise to more advanced forms of Analytics like OLAP or Multi-Dimensional Analytics, Predictive Analytics and Prescriptive Analytics. Even the reporting has evolved based on these Historical Data, to show more complex Representations in the form of Dashboards, Graphs, Visual representation of data, leading to sophisticated Analytics tools called Visualization tools.

While this progress has been happening in the world of Analytics, there is a parallel, related and interesting development happening in the world of Image Processing. This is the usage of Cameras to build AR (Augmented Reality) applications. Better Telecom Bandwidths, Powerful Mobile cameras, AR techniques have made it possible now to View, Point, Detect and Transmit an Identified object's coordinates and other information to the Backend Systems for further processing.

There is also another technology called the Image Analytics, which can leverage the image captured by the AR application to do the relevant Image Analytics and pass it back to the AR application.

This article attempts to bring a Convergence of the above three Technologies Viz.: Visualization Technologies (backed by Huge Historical Databases or Big Data), Image Analytics and AR Technologies, to bring out the Next Generation Class of Analytics called as "Ground-Zero" Analytics.

Ground-Zero Analytics is basically trying to see the Descriptive, Predictive and Prescriptive Analytics, in Real-time at the Location where it is Happening, using Mobile Cameras, AR Technologies and Visualization Technologies.

Data and Big Data

Data has been around in computer systems for more than a few decades in the form of Files, Databases – either in Structured or Un-Structured forms. However, it is only recently that the data management has encompassed Huge Volumes, Variety and Velocity of data, to deliver faster processing to the downstream Reporting and Visualization applications.

The data is stored in files, Databases, Data Lakes, Data Warehouses, Subject oriented Data Marts, Analytical Databases – all with the single goal of providing valuable information to the downstream Reporting and Visualization applications – through which Business can view their data in multiple dimensions across time, place, products, people.

Data Visualizations

Data Visualization is the most Intuitive approach to depicting the Processed data to the Business Users. This uses various visualization techniques like Graphs, Plots, Tables and many more to show various sets of related data, in a single view to the Business User, so he or she can make an informed decision about their business scenarios. The current Data Visualizations tools and technologies talk about a Business Unit or Business area in the form of Data or Visuals and it is up to the Business User to have the domain knowledge of their Business or a photographic memory of their Business area, or branch or office or Warehouse or Stores.

Augmented Reality of the Domain in which we do Data Visualizations

Augmented Reality is the concept of the user seeing the reality upon which embedded information is also shown. This is not difficult to imagine. While watching a News Channel, we see the actual video footage (say of the Cricket Field) and on top of it we see Data being shown (maybe of the flight path of the ball). This is the simplest form of Augmented reality.

However, the AR (Augmented reality) in the context of Data Visualizations can get a bit more complex and much more Dynamic in nature. While the camera shows the Image of a particular domain, the domain itself is marked with specific points (either in a Marker or Markerless mode), such that when a particular Point in the Domain is in View of the Camera, the AR system can detect the Specific Point and then become aware of what that Specific Point is.

Example: When the camera is pointed toward a Refinery pipeline, the specific point on the pipeline can tell the AR system that this is a "Crude Oil" carrying pipeline toward the Main Refinery. It will also know based on the GPS coordinates of the "Crude Oil" carrying pipeline.

With this basic information about the "Crude Oil" carrying pipeline, like the Actual Image from the Camera, its GPS coordinates, and the Marker Inputs, being passed onto the AR system, there is now scope for Mashing this AR information with its related Analytics information, to derive newer forms of Analytics. This is the core concept of this article and is called "Ground-Zero-Analytics"

Introducing the concept of the AR-based-Ground-Zero Analytics

AR-based "Ground-Zero" analytics is the concept of using the AR-based information like Image, GPS coordinates, Marker information as inputs to a Backend Data Visualization Engine in realtime. The Backend Visualization Engine would then pull the relevant information about the "Marker" in context, build the necessary Charts, Graphs, Historical Summaries and then show in real-time these, Analytical Insights on the "Image" being looked at by the camera, with Zero Latency.

So, in our example, while the Camera is looking at the "Crude Oil" pipeline, at the same time the AR system would pass on information like the Image, GPS coordinates, Marker Inputs to the Data Visualization systems and Image processing systems.

The Data Visualization systems would pull out the "Crude Oil" pipeline-related information, from the databases, like Date of Installation, last Service or Inspection Date, Expected Lifetime of the Pipeline, Any History of Leakages, Breakages, Fire, etc for this "Crude Oil" pipeline. The Data Visualization system would then do the specified analysis and then build the necessary Dashboard, Graphs, Charts, or Additional Textual information – and then show it on the Screen as an Overlay on the Image being shown in the Camera – all in realtime.

In addition to the Data Visualizations, the AR systems will also pass the image of the "Crude Oil" pipeline to an Image Processing engine, to detecting any Faults, Cracks, etc. This would then be passed to the Analytical Systems and finally recommend "Preventive Maintenance" instructions on the Screen, again all in realtime.

Hence by converging AR technologies with Image Processing/Recognition and Data Visualizations, we can take the Analytics from a Backroom activity to "Ground-Zero" where the Report or Dashboard is being spoken or analyzed about, in realtime. This is the concept of the "AR-based Ground-Zero" analytics.

Technical Approach

Shown below is a high-level Technical Approach to implementing an AR-based "Ground-Zero" analytics:

The AR system would expose their AR data to an AR Receiver which would then curate this data and then pass it to the Consuming Systems like Data Management, Visualization and Image Processing Applications.

These Consuming applications would then pass on their relevant Analytical insights to the Visualization Outputs, Text Outputs, Image Processing Outputs and Analytics Output Modules (as shown in the above diagram).

These Analytical Outputs would then be passed onto the AR Transmitter Custom Software Module, which would then pass on their relevant information back to the AR Scenes, which can then display the same as additional Data or Insights or Intelligence on top of the current Scene in view of the Display Screen.

Benefits of AR based-Ground-Zero Analytics

AR-based Ground-Zero Analytics is the real-Time Avatar of the Offline Analytics that we are used to seeing on our Laptops and Mobiles in the form of reports, Dashboards and Visualizations, for the past few decades. While these serve our purpose fairly well, we still have to map (in our mind), the Real-World Scene/Domain about which this particular reports or dashboard or Visualization is talking about.

However, with an AR-based "Ground-Zero" analytics, we can implement the concept of "Get your Insights when you See it and Where you see it", in realtime.

This can be very useful in situations where latency is crucial and we cannot wait for the offline BI and Analytics systems to do their analysis and then enable us to do after-the-fact analysis. An opportunity for Sales or Preventive Care could be lost.

AR-based "Ground-Zero" Analytics can be the future of all reporting and Visualizations and if merged with the Digital-Viewing-Glasses, then we can be a walking-Encyclopaedia in the True-Sense.

Use Cases of AR-based-Ground-Zero Analytics

• AR-based Ground-Zero Analytics can be useful in the Operational Scenarios like Inspectional Visits to Oil refineries, or Office Installations or any Factory Installations. We can point our AR-enabled Mobile Camera on a particular Object in our View (which can have pre-determined Markers or using Markerless Hit-tests) and then use the information received by the AR applications, to pull out the relevant additional Information from the backend systems like Data warehouses, Visualization Software, Image processing software and then give relevant additional data or Instant Insights or Recommendations to the Field Officers or the Disaster Recovery Personnel.
• AR-based Ground-Zero Analytics can be very useful in Defence Scenarios where a Soldier or Navy personnel can point at the Object-in-View to know more information about that object be it the terrain information, the wild fruit in the forest whether it can be consumed, Analysing enemy weapons or landmines, Enemy Troop movement or formations, Enemy Vehicles deployed, Soldier Injury analysis and suggested first aid (e.g. timely amputation of a Hand or Leg can save our Soldier if the appropriate information is provided using the AR-based Ground-Zero Analytics system.
• AR-based Ground-Zero Analytics can be used in the Tourism industry to create AR-Guides which can show additional information, Visualizations of the Tourist Place or Monument, or Palace or Priceless Artifact in the Museum that a Visitor is viewing. This will make the Tourist's visit more enriching. The AR-based Ground-Zero Analytics Travel companion can also give additional information about a particular shop, lake, local food speciality as seen in the local markets etc
• AR-based Ground-Zero Analytics can be used in Server Management, especially in a Cloud Environment where there are tons of Server Farms. An AR-based Ground-Zero Analytics can give instantaneous information (in the form of Graphs, Tables, Text) to tell when was the last time a particular Server failed or had gone through maintenance or part replacement details within that Server and what was the cost and who was the service provider and what was the downtime and impact to the users of that Server. All this Insightful information can be obtained in real-time, just by Pointing our AR-enabled Mobile camera on that particular Server.
• AR-based Ground-Zero Analytics can be a huge aid to the Insurance Investigators who can point at the part of a Vehicle Damaged in Accident and then try to get additional information about that part, its cost, availability with Suppliers, Service costs, Driver behaviors, Repair options, Insights based on other similar claims etc before computing the Insurance Claims, using the real-time information that is available on the Screens against each part of the Vehicle.
• We can use AR-based Ground-Zero Analytics in the Global Education Industry, so be it in the Physical Classroom or in the Online Classes on a computer screen, we can get additional information about the topic being taught, by pointing our mobile camera onto the Writing board or the Computer Screen.
• AR-based Ground-Zero Analytics can be used in the highly critical and large Security Industry, where we can get more information about a person or employee or vendor or customer who is entering or leaving our Office or Factory premises, just by pointing a mobile camera at them or their ID card and then the AR-based Ground-Zero Analytics system will pull all the recent history of that person, thus enabling the Security Personnel with additional information about the person, than what the person might be revealing. This also applies to National Security at our Borders, Immigration Offices, Airports, Local Police Surveillance, Customs, etc.
• AR-based Ground-Zero Analytics can be used in the Billion-dollar Online Shopping Industry by the Customers, so they can get additional Text, Analytical, Audio and Video information about a product on their Mobile Camera before they make an Online Purchase. Thus, Online shopping is now more data-driven and intelligence-based, than what the Vendor wants to tell the customer.
• In a country like India, where we still rely heavily on Small and Medium size shops for all our purchases, it would be an Innovative Concept to give each Shop an AR-Marker, so when a customer points this AR-based Ground-Zero Analytics application at the Shop's Marker, the customer gets information about all the products that are sold in that shop, the quantities, the price range, as per their Budget. This enables the concept of Hybrid-Shopping Model ( or an Intelligent Window-Shopping Model), where a Customer is not Online, but physically in front of the Store and yet does not enter the Store yet, as he/she does Window Shopping using the AR-based Ground-Zero Analytics Application on their mobile phone – simply by pointing the Mobile camera at that Shop's AR Marker-point.

There are hundreds of similar Use Cases as it is applicable to any Business Domain and any Industry like Newspaper, Advertising, Flyers, Books, Reference Books and Journals, Libraries, Detective Agencies, RTO offices, Traffic Violations, Legal Opinions, Medical Inference, Archaeology, Mining Industry, Port Management, Shipping Container Analytics and hundreds of other applications in all Domains of our lives and hence this concept of Ground-Zero Analytics is Ubiquitous in nature and only limited by our imagination, innovation and inquisitiveness.

Conclusion

In this article, an attempt has been made to using three existing technologies like Augmented Reality (AR), Data Visualizations (backed by Data warehouses and Predictive/Prescriptive Analytics) and Image Processing, to come out with a Next-Generation Reporting Concept of Realtime "Ground-Zero" Analytics.  Ground-Zero Analytics is not a complicated concept, it just involves the integration of Outputs/Inputs from AR, Data Visualization and Image Processing engines, to generate Augmented Analytics on top of the Realtime Scene being Viewed. This would enable the users to have Realtime relevant and additional data about their object in view, and thus make intelligent and Data-Driven decisions in real-time.

Ground-Zero Analytics could soon be the new normal for all Operational Analytics, which can co-exist with the traditional Reports, Dashboards and Visualizations which would be used for Offline after-the-fact analysis on a detailed level over a historical period of time.

Ground-Zero Analytics is basically trying to see the Descriptive, Predictive and Prescriptive Analytics of an Object-in-View in Real-time at the Location where it is Happening, using Mobile Cameras, AR Technologies, Visualization Technologies and Image Analytics – thus making it a solid platform for the Intelligent Viewing Glass Assistant of the Near-Future.

About the Author

The author, Harikrishna S Aravapalli, is a Software professional with around 20+ years of IT experience in the areas of Data, Data warehousing, Business Intelligence, Predictive Analytics, Social Medial Analytics, Visualization in various roles like Developer, Technical Lead, BI Architect, Manager and a Solution Architect. He is currently working as a Senior Solution Architect in the areas of Data, Analytics and Advanced Analytics in a major IT company in their Data & Analytics practice. His email id is krishpalli123@gmail.com