The analysis of the millions of works that are part of the cultural and artistic heritage is a task that seems impossible for humans, but not for supercomputers. However, before machines can do it, they have to learn how.
The European project Saint George on a Bike, coordinated by the Barcelona Supercomputing Center – Centro Nacional de Supercomputación (BSC-CNS) in collaboration with the Europeana Foundation, started in 2019 with the aim of using the supercomputer MareNostrum4 to train artificial intelligence (AI) models to better interpret European cultural heritage, as well as to spread awareness of its richness, value, and the importance of preserving it.
A campaign has been launched on a citizen science portal to collect annotations that will help computational models interpret 5,000 European paintings.
For this purpose, the project seeks to generate automatic descriptions of hundreds of thousands of images from various cultural heritage repositories using natural language processing algorithms and deep learning.
Now, in a second phase of the project, researchers have launched a collective collaboration campaign or crowdsourcing on Zooniverse, a citizen science portal, to collect thousands of manual annotations to better train these AI models. The campaign is completely open and anyone can participate in the analysis of 5,000 European paintings.
For the first time, artificial intelligence provides descriptions of cultural heritage images with the widest coverage of themes, objects, and iconographic relationships, taking into account the temporal context of the works’ creation and the composition rules of different periods and scenes of sacred iconography between the 12th and 18th centuries.
“This ambitious project interprets images for the first time according to their context, and seeks to provide machines with a certain common sense, which is one of the major barriers of AI today,” says Joaquim Moré, a researcher at BSC and computational linguistics expert on the project.
“For example, when the system initially identifies a motorcycle in a 15th-century painting of Saint George, it corrects and identifies the object that is more plausible for the time, which is the horse. Later on, this adaptation will be made to the cultural context. So, in the Japanese cultural context, what in Europe we would call a knight, would be a samurai,” he explains.
Thanks to this initiative, paintings can be analyzed better, unseen images or compositions can be detected, relationships between their elements can be discovered, and their symbolism can be interpreted, aiding in the development of cultural and dissemination initiatives, such as virtual exhibitions with related paintings from around the world.
Cultural, educational, and academic applications
“Our project will allow quick access to cultural information that can be used not only for cultural and social purposes, but also in other sectors like education and tourism, and possibly also be helpful for historians or anthropologists,” says Maria-Cristina Marinescu, a researcher at BSC and coordinator of Saint George on a Bike.
“Furthermore,” she concludes, “society as a whole can also benefit from better public services, such as improved accessibility to websites for visually impaired people, narratives that can expose social injustice, or integration through collective cultural heritage to help create a more tolerant European identity.”
via: MiMub in Spanish
