Gartner reveals the top trends shaping the future of DSML

Data science and machine learning (DSML) are rapidly growing and evolving fields that leverage the power of data to generate insights, predictions, and decisions. As data becomes more central to artificial intelligence (AI), especially with the rise of generative AI, DSML is facing new opportunities and challenges that will shape its future.

Gartner, a leading research and advisory company, has identified the top trends that are impacting the future of DSML based on a global survey of more than 2,000 executives, experts, and users of DSML tools, as well as a comprehensive review of academic literature, industry reports, and media articles. The main trends highlighted by Gartner are:

 Cloud Data Ecosystems

Data ecosystems are moving from self-contained software or blended deployments to full cloud-native solutions that offer greater scalability, flexibility, and integration. By 2024, Gartner expects 50% of new system deployments in the cloud will be based on cohesive cloud data ecosystems rather than manually integrated point solutions. Organizations should evaluate data ecosystems based on their ability to resolve distributed data challenges and access and integrate with data sources outside their immediate environment.

Edge AI

Edge AI is data processing at the point of creation at the edge, near IoT endpoints, rather than in centralized servers or clouds. This enables real-time insights, pattern detection, and data privacy. Edge AI also improves AI model development, orchestration, integration, and deployment. Gartner predicts that more than 55% of all data analysis by deep neural networks will occur at the point of capture in an edge system by 2025, up from less than 10% in 2021. Organizations should identify the applications, AI training, and inferencing required to move to edge environments.

Responsible AI

Responsible AI is the ethical and social dimension of AI that covers aspects such as value, risk, trust, transparency, accountability, regulation, and governance. Responsible AI aims to make AI a positive force rather than a threat to society and itself. Gartner predicts that the concentration of pre-trained AI models among 1% of AI vendors by 2025 will make responsible AI a societal concern. Organizations should adopt a risk-proportional approach to deliver AI value and take caution when applying solutions and models. They should also seek vendor assurances to manage their risk and compliance obligations.

Data-Centric AI

Data-centric AI is a shift from model and code-centric approaches to a focus on data quality and availability to build better AI systems. Data-centric AI solutions include AI-specific data management, synthetic data generation, and data labeling technologies that aim to overcome data challenges such as accessibility, volume, privacy, security, complexity, and scope. The use of generative AI to create synthetic data is rapidly growing, with Gartner predicting that by 2024, 60% of data for AI will be synthetic.

Accelerated AI Investment

Organizations deploying solutions and industries seeking to expand through AI technology and AI-based enterprises will continue accelerating investment in AI. Gartner forecasts that by the end of 2026, more than $10 billion will have been invested in AI firms that rely on foundation models, which are substantial AI models trained on enormous volumes of data.

In a recent survey by Gartner of over 2,500 corporate leaders, 45% said that the recent hoopla around ChatGPT had led them to raise their AI efforts. 70% of respondents claimed that their organization uses generative AI for study and exploration, while 19% use it for pilot or production.

These trends indicate that DSML is becoming more democratized, dynamic, and data-centric as it adapts to users' and stakeholders' changing needs and expectations. DSML also holds great potential for innovation, productivity, and value creation across various sectors, regions, and functions. However, DSML poses significant technical, ethical, and social challenges that must be addressed to ensure its safe, responsible, and beneficial use. Gartner urges leaders and decision-makers to embrace DSML as a strategic priority and invest in its development and adoption while ensuring its alignment with human values and societal goals.