Artrendex’s ArtPI Uses Artificial Intelligence to Revolutionize Art Discovery

ArtPI (artpi.co) is a new interface or API driven by artificial intelligence that's poised to transform the way art gets discovered, displayed, and sold. It promises to transform art discovery the way Shazam transformed music discovery.

ArtPI detects features and patterns using overarching visual style characteristics based in art theory and history. Built from the ground up specifically for art and drawing on centuries of artworks, ArtPI can find visually similar works, label styles and eras, recognize subject matter and artist, and find connections within a large collection.

For museums and other institutions, this means users can search their collection visually, exploring artworks in a more organic and intuitive way by looking for particular features. Curators can have the entire collection at their fingertips, finding unexpected ties and similarities that may inspire new ways to display or interpret works. For galleries, dealers, and auction houses, ArtPI can be used in conjunction with social media to uncover trends or predict market conditions.

"AI is often used to label images and objects in them, and ArtPI can also find works by subject matter," explains Ahmed Elgammal, creator of ArtPI, Professor of Computer Science at Rutgers University, and founder of AI art startup Artrendex. "But you don't always just want to find painting of dogs or putti. Sometimes you want a particular style of lighting or line. We've trained our algorithms to identify these elements and principles of art, based on foundational art historians' approaches. By working with the visual elements instead of content labels or other metadata like artists or era, you can find art you didn't know existed and see its connections with other works."

ArtPI works even when images of an artwork are distorted, allowing for less-than-ideal source images or shots from a variety of angles. The AI model has been trained to correct for distortion, distilling a slanted or poorly lit image into information that can be compared to other works. This does away with the need for complicated photographic documentation, letting ArtPI work in a variety of less formal circumstances. Now, for example, you can see what work from a major exhibit won the hearts of Instagramers, even if they took a quick snapshot from the side.

AI can spot visual patterns, often better than humans. ArtPI employs neural networks to train its algorithms on images, using more than 250,000 artworks for its initial training phase. The more works encountered by the algorithm, the more nuanced the results. "Humans grasp context quickly, but AI is great at homing in on details and doing so for millions of examples in seconds or fraction of second," says Elgammal. "Every search and every image added to ArtPI will contribute to its refinement, increasing its value over time."

Philadelphia's Barnes Foundation was one of the first institutions to adapt ArtPI. The Barnes first analyzed its extensive collection, which includes a great number of impressionist works. The first round of AI-powered analysis by different computer vision platforms generated some unexpected results: In one instance, the content of museum's Renoirs were labeled as stuffed animals. However, once the museum's works were inputted to ArtPI, some remarkable connections emerged.

"We thought this part of our process would be the hardest, yet it proved the easiest," recalls Shelley Bernstein, the digital consultant on the Barnes project. "The visual similarities were so good, they were almost too good." The problem: The Barnes collection had a wonderfully idiosyncratic way of displaying works together that originated with the collection's founder. To recreate this approach online, the Barnes and ArtPI crafted a custom way to display results that better mirrors the real-life museum experience.

"Exploring art, be it for educational or valuation purposes, can be done in many ways, and AI is just another tool, another frame that helps us find new insights into the art we're working with," says Elgammal. "Being aware and optimized for the concepts of art, ArtPI is designed to be the infrastructure recognition engine for museums and the art market, in other words, the 'Shazam for Art.' I believe visual search and suggested connection can both democratize the discovery process for laypeople and renew the excitement curators, galleries, and auction houses feel as they consider the works in their domain."