This post presents part of my new chapter, Augmented Reality in Museums: Enhancing Accessibility and Alternative Content to Official Rhetoric Through Digital Technologies, published in the book Information, mediation, and culture: theories, methods and research 1, edited by Letramento publisher and PPGCI (Graduate in Information Science at UFMG University), with resources from CAPES foundation.
In the chapter, in addition to the experience of Augmented Reality (AR) that I carried out at MASP (one of the most important art museums in Brazil), presented below, I discuss notorious cases of digital technology use in ethical activism in museums, involving themes such as decolonisation and digital repatriation.
Download the free book’s PDF at the end of this post (Portuguese). The book can also be purchased in print in Brazil.
Augmented Reality is digital technology that allows you to combine real and virtual objects in an existing environment, viewed through some electronic device, such as a cell phone or tablet screen. By aligning surrounding objects with virtual ones, AR allows interaction with them in real time as the user moves the device.
AR differs from Virtual Reality (VR) as VR does not add elements of the user’s face-to-face environment to the experience, promoting the user’s immersion in an entirely virtual environment.
Videos of My Augmented Reality Experiences
In the video below, on the left, I saved the phone screen showing my AR experience on MASP2. In it, the HP Reveal3 app recognises the painting. Then, overlap the painting’s image with the photograph of its respective label provided by the institution on the back of the artwork.
This automatic triggering, from the very appearance of the objects, allows digital content to be inserted into a museum without any interference or physical addition to the space, as QR Code labels require, for example. Thus, AR interventions can be official or not, providing a range of possibilities for visitors to “rewrite” the museum discourse.
With this same automatic shooting feature, on the right, the other video shows an entertainment I made for my students in the Museums Studies course at UFMG.
Also, using the HP Reveal app, I augmented artwork by well-known artists such as Salvador Dalí, Magritte, Arcimboldo, and Mondrian. Similar to the MASP experience, the app recognises the painting and presents the student with its respective animated experience, chosen by me from the app’s library and linked to that work using my @webmuseu profile. I also taught students how to create their profile and their augmented reality experiences in class.
Unfortunately, the HP Reveal app has been discontinued. So now I’m redirecting my AR investigations to the Metaverse 4 free app.
Research reveals that the use of AR in Brazilian museums can be considered incipient, as are the postgraduate studies developed on the subject in the Information Science field. 5
We understand that AR can contribute not only to museum communication but also to inclusion and accessibility, acting as assistive technology. 6
Therefore, I hope this new book chapter contributes to expanding the use of augmented reality in our country’s information and culture units.
Introduction
In the last national census, around 45 million Brazilians declared themselves to have at least one of the listed disabilities, representing approximately one-fifth of the population.
In addition, at some point in our lives, all of us will present some disability, temporary or permanent, as is the case of advanced ages mobility issues. These numbers reinforce the fact that accessibility is closely related to the social function of museums:
“Exhibitions are complex presentations that convey concepts, showcase objects, and excite the senses. However, as museums recognize the diversity within their audiences, they realize that exhibitions must do more: exhibitions must teach to different learning styles, respond to issues of cultural and gender equity, and offer multiple levels of information. The resulting changes in exhibitions have made these presentations more understandable, enjoyable, and connected to visitors’ lives. Accessible design must be a part of this new philosophy of exhibition development because people with disabilities are a part of museums’ diverse audience. Discovering exciting, attractive ways to make exhibitions accessible will most directly serve people with disabilities and older adults.” (Smithsonian guidelines for accessible exhibition design, 2021, p.2)
Augmented reality apps, in this context, act as assistive technologies, that is, resources and services that expand the autonomy, inclusion, and functional capacities of people with disabilities.
Although it is a broader concept, within the scope of this text, we understand Augmented Reality (AR) as an interactive experience provided by digital technologies, whose real-world information is amplified by others generated using electronic devices, such as images, videos, links, and animations.
Thus, digital elements can overlap or be added to the real ones captured, for example, by cell phone cameras. Unlike Virtual Reality (VR), in which there is an entire replacement of the real by immersive simulations, in augmented reality, we undertake a dialogue between the face-to-face and content mediated by equipment and its apps (smartphones, tablets, etc.).
In this chapter, we will present an exploratory study with augmented reality experiences in one of Brazil’s most important art institutions — the São Paulo Museum of Art (MASP). The goal is to enhance the accessibility of its main collection.
In the case of MASP, the labels on the back of the paintings in the long-term exhibition – Picture Gallery in Transformation – overlapped on the paintings using the augmented reality application HP Reveal, as we will see in detail below. This experience was called MASP Labels by Webmuseu.
Crystal Easels at MASP Museum: A Sea of Masterpieces Floating in Space
MASP houses the most important collection of European art in the southern hemisphere, totalling more than ten thousand works that include African, Asian, and American art.
It is a private, non-profit museum founded by Assis Chateaubriand in 1947. In 1968, it began to occupy the headquarters designed by Lina Bo Bardi, becoming an architectural landmark on Paulista Avenue (the most important in São Paulo). MASP is also the meeting point for political, artistic, and cultural manifestations.
“The works are exhibited on glass easels—made of a glass plate fitted in a concrete block. The easels are arranged in rows in the large gallery space, with no divisions, located on the second floor of the museum. Removing the artworks from the walls and displaying them on these easels allow a closer encounter between the public and the artworks. The visitor is invited to walk through the gallery in the midst of the artworks like a forest of pictures that seems to float in space. The open, fluid, and permeable gallery space offers multiple possibilities of access and readings of the works, eliminating predetermined hierarchies and scripts.” (MASP, 2022, website)
In a controversial rescue, the crystal easels (glass plates fitted to concrete blocks) were replaced in 1996 and redone in 2015. In defence of this return to the past, MASP reiterated the political character of the open gallery designed by Lina, which would reduce hierarchies, predetermined paths, and sacralization. The glass support also allows access to the frames’ back details, rarely seen in conventional exhibitions.
The label located on the back provides first contact with the artwork without information and texts. This is perhaps the most exciting aspect of the current design. The very location of these paintings – in one of the most important museums in the world – is already a sufficiently inductive context.
Given this complex scenario, prudence called for us to defer our conclusions until an in-person visit. At the time, in 2018, we found that Lina Bo Bardi and MASP made the right decision, in our view: the sea of masterpieces suspended in space provides an unprecedented aesthetic and museum experience.
The Dilemmas With Crystal Easels
Despite the best intentions of this translucent expography, when staying for a long time at MASP, some flaws jumped to the eye.
Accessibility Barriers Caused by Labels on the Artworks’ Back
Older people, wheelchair users, pregnant women, visitors with children in their arms, or with reduced mobility often asked their companions to make the journey to the labels for them, returning only with the basic information retained by memory. For this reason, the museum’s labels, which have relevant content, did not reach a significant part of the public.
The flyers with maps, made available by MASP, had summarised information about the works (names and dates). Following it required much attention to avoid getting lost in the list, as they are only numbered texts without images. On our visit in 2019, the flyers ran out at the beginning of the day and were not replaced due to technical problems. On the other hand, the pocket guide presents a selection of sixty items: less than a third of the total paintings exposed in the period.
Visitor fatigue
People with reduced mobility and also the general public sometimes consider the constant walking tiring. Some easels are arranged in five or six side by side, making the route long. In our post about this experiment, one visitor commented:
[…] I am a “normal” person, but with an inflamed knee on the day of the visit and without the glasses that I only use to read at home, I confess that I had a lot of difficulties because I really wanted to read, even though I knew the work and the history. Anyway, what a beautiful project (thinking about a reality that is not yours), innovative (I’ve never seen anything like it), functional (who wouldn’t like to read the information calmly or even at home?), and vast capacity to generate media […]“Backward” Visit Choosing Which Works to Enjoy by the Labels
Rushed visitors, perhaps short-time business travellers, seemed to carry out the visit starting from the end, choosing what to see through the labels on the back of the paintings. This inverted visit results in opposite outcome to those desired by the exhibition proposal, reinforcing the fetish of the great masters, keeping the visitor in their “artistic bubble,” and limiting discoveries.
Overcrowding and Selfies
As the museum starts to fill up, which often happens at this prestigious institution, walking between the easels becomes slower, taking more time and requiring more attention to avoid bumping into the works. It is also necessary to wait for visitors to take their selfies and photos in front of the paintings. It is not uncommon for this to be a long process, as people wait for other passers-by to get out of focus, as the crystal easels reveal everything and everyone around them.
This transparency can also hinder the enjoyment of the painting, especially for people with attention difficulties.
Reading the Label Simultaneously With the Observation of the Artwork
Visitors who study the art may prefer to observe the painting as they read the label. The texts present descriptions of the scenes, artistic techniques, and symbols retained in the composition, among other details that demand the simultaneous scrutiny of the painting.
I caught myself photographing some labels, then analysing the content in front of the artwork on the cell phone screen, which gave rise to the idea for this virtual intervention at MASP.
MASP Labels by Webmuseu: Augmenting MASP Texts with HP Reveal
My proposal for a collaborative intervention at MASP consisted of inserting labels over the artwork using the free augmented reality application HP Reveal. The texts were photographed and added to the app, linking them with their respective paintings. The painting worked as an automatic trigger for the superposition of the label on the smartphone screen.
This was a significant advantage of HP Reveal compared to many other apps: capturing anything special (e.g., QR Code) to visualise the experience was unnecessary. Once the visitor downloaded the application and selected the profile @Webmuseu, where I made the MASP collection publicly available, the user simply pointed the cell phone camera at the desired painting, and the text automatically appeared on the screen.
The sculptures and other objects were not included in the HP Reveal online collection because we understand that their three-dimensional quality invites the user to enjoy their surroundings, which, in this process, will naturally access the label.
Another important aspect of this experience is that it did not substantially interfere with the museography conception of MASP and Lina Bo Bardi, in our understanding. Users could still choose which path they wanted to take in the exhibition. They also remained in contact with the artwork without prior information.
The only essential change is that this was done through the app instead of going to the back of the easel to read the text. In this sense, eye contact with the back of the frame would not happen. In other digital bases, such as a website or in a specific museum app, photographs of the back could also be made available, allowing visitors to choose whether to view it up close, thus going around the easel or not.
By using artwork as the trigger of the experience itself, augmented reality, in this case, does not condition the path. This is also important because this exhibition aims at semi-permanence, and is constantly revisited for modifications and adjustments. It also receives external works that can be included in the app whenever they are loaned to the museum within the international exchange proposal that began in 2018 with Tate.
In 2019, the institution invited by MASP was the Museum of Contemporary Art Chicago (MCA), whose photography and sharing permission policy is similar to that of MASP, as described in its Frequently Asked Questions (FAQ). MCA’s works were placed along the crystal easels in chronological order, starting with the most recent.
“Although there is a focus on figurative art, which reflects the MASP collection, some works by MCA flirt with abstraction — such as those by Gonzalez Torres and Levine, as well as others by MASP, such as those by Lygia Clark and Reuben Valentine. Exchanges with other museums allow MASP to bring different works to coexist with those in the museum’s collection, suppressing, albeit temporarily, some of our gaps, as well as establishing other possibilities for dialogue with the collection.” (MASP, 2019, website, our translation).
At this moment, it is necessary to point out some obstacles users could encounter while using the tool. Although free, HP Reveal needed to be downloaded, which implied a memory consumption of around 400 megabytes. The museum has free Wi-Fi, so it would not be necessary for the visitor to spend internet credits.
In Brazil, only 32% of museums provided Wi-Fi to the public in 2019. Despite being very simple to operate, HP Reveal was not so intuitive for most people precisely because Brazilians are not so familiar with advanced digital technologies such as augmented reality.
Simpler smartphones, with lower quality cameras, could have difficulties capturing some paintings whose contrast was not so evident. As part of our usability assessments of the case, we tested all the AR experiences at MASP on an iPhone with a powerful camera and an old Sony android cell phone, whose camera was damaged and blurry. Even so, it was possible to capture all the experiences in the paintings, although some took longer to be recognised by HP Reveal.
Accessibility in HP Reveal is not universal since the text appears on the app as an image. Due to the interactive nature of the experience, the label did not have excellent resolution. People with low vision or blindness and deaf people who can’t read Portuguese or English could not be considered included.
Other Possibilities for Digital Accessibility Resources
With universal design as our guide, we considered some possibilities for inserting MASP content (pictures of paintings and their labels) on the following platforms. These resources can be implemented by the MASP or other researchers in the area, with authorisation from the institution, leaving it here as an open suggestion to all interested in collaborating with the museums.
WordPress Website with Alt Text and Sign Language
WordPress consists of a Content Management System (CMS). It’s free software for the Web, which accounts for about 40% of websites published online today.
Among the numerous advantages of WordPress are the Semantic Web and Search Engine Optimization (SEO), that is, its ability to inform search mechanisms and screen readers, such as NVDA, the various websites’ parts: menus, titles, images, texts, keywords, etc. This expands accessibility in every way, from people with disabilities to information retrieval. We address these issues in the Wise Stones project guidebook, in which we also apply content management processes and web writing techniques.
With WordPress, it is also possible to add alternative descriptions (alt text) in the media, which consist of metadata not visible to users in general but read by NVDA and the like, with relevant information for people who are blind or have low vision.
Unlike conventional subtitles, written for sighted people, alt texts describe the scene focusing on those who cannot view them. On a website, therefore, it would be possible to add the MASP paintings, the labels made available by the museum, as well as exclusive content produced for people with visual impairments (alt text). This online collection could be developed using the resources of the Tainacan.org digital repository platform (Brazilian themes and plugins for WordPress).
Finally, it would be possible to insert audio and videos with subtitles in Libras – the Brazilian Sign Language. Translation plugins, such as VLibras, would allow deaf people who do not know Portuguese to read the content.
For those who can see, photographs of the back of the paintings, with the frame details exposed by the crystal easels, would help visitors choose whether they want to go around the easel to observe these artwork’s details.
Artificial Intelligence Associated with Augmented Reality
Perhaps the most ambitious possibility is the association of augmented reality with Artificial Intelligence (AI).
An example of AI use in museums is the project “The voice of art”, carried out at the Pinacoteca in São Paulo. Through Watson, AI technology from IBM, visitors to this museum could “talk” to some artworks, asking questions to the app, which was previously fed and updated with a wealth of information.
Artificial intelligence would allow the visitor full and free interaction with the content, adapted to their profile through algorithms. They can respond to a brief questionnaire, so the algorithm would determine which database to access, considering age, language, education level, manifested interests, and even particular needs, such as cognitive impairment, autism, or dyslexia.
Throughout the experience, the app could also adjust to accents and regionalisms, considering that there are several vocabularies in Brazil. Maybe there we are talking about a universal design.
However, the adoption of such technologies requires a lot of resources, both human and financial. Finally, they require assessing the risks associated with algorithms, such as the “bubbles”, the propagation of prejudices by those who programmed the code, and the reinforcement of confirmation bias.
Reflections
According to the TIC Cultura (ICT Culture) 2020 report, only 32% of Brazilian museums had their own website, with another 24% hosted on third-party websites. About half (56%) were on social networks, and 25% made digital collections available to the public on the Internet. In Brazil, therefore, the use of digital resources and the online presence of museums is still quite precarious.
Within this context, interventions such as MASP Labels by Webmuseu, which are not temporary spontaneous experiences, but permanent projects based on free software, can collaborate with museum institutions, expanding their potential for inclusion in a fast, free and sustainable way. The various assistive technologies, including augmented reality, enhance accessibility in art museums, especially when faced with a radical exhibition design, such as the glass easels at MASP.
We also understand that museums need to move, more and more, towards opening their collections and making their content widely and freely available on the Web. In this space, protagonism is democratised by the countless free resources currently available in cyberspace.
Finally, augmented reality apps, storytelling, audio guides, and information management on the Web make it possible to give voice to the excluded minorities and offer alternative content to the official rhetoric. These contents can be collaborative interventions (like our experiment at MASP) or critical ones, like the Feminist Museum Hack and MoMAR, discussed in my book chapter.
In the case of hacktivism, reflections on the curatorship elitism, sexism, structural racism manifested in the labels’ language, Eurocentric perspective, and other topics involving decolonial studies are placed inside museums.
They also contribute to the virtual repatriation of the collections that, inhabiting international museums so often, can arrive in their lands of origin, at least through the Web.
The opposite is also true: through digital, we can intervene in museums around the planet with unprecedented collaborative content coming from those who, imbued with a new conscience, do not need to silently agree with the colonisation of their discourse.
Notes
1 – The experiment was carried out in the HP Reveal app (see note 3) and is no longer available. However, a few months after our experience, the MASP museum launched an app containing the same augmented reality feature described here.
2 – Informação, mediação e cultura: teorias, métodos e pesquisas (Information, mediation, and culture: theories, methods and research, Portuguese e-book and printed version). Editors: Fabrício Silveira, Maria Guiomar Frota, Rodrigo Moreno Marques. Belo Horizonte city. Publishers: Literacy, PPGCI-UFMG. Financial support: CAPES foundation. 2022.
3 – HP Reveal was a free app that allowed users to create profiles, give likes, and share content through augmented reality experiences. Unfortunately, HP discontinued the app, and, at the time of publication of this chapter, in 2022, it is no longer available. We consider this a significant loss, even though other free augmented reality applications exist, such as Metaverse (see note 4 below).
4 – The Metaverse app, in addition to augmented reality experiences, allows the creation of quizzes and games. The app is free, and its experiences can be triggered by QR Code or geographic location (GPS).
5 – Read more in the article “O uso de Realidade Aumentada no contexto dos museus: o portfólio brasileiro de teses e dissertações até 2017” (The use of Augmented Reality in the context of museums: the Brazilian portfolio of theses and dissertations up to 2017) by Graciela Menezes, William Vianna, and Márcio Matias. Journal Em Questão, Porto Alegre, v. 25, no. 3, p. 246-268, Sep./Dec. 2019.
6 – Assistive technologies are inclusive technologies that provide people with disabilities with greater autonomy and quality of life.
Acknowledgements: To PPGCI Graduate Program colleagues for the opportunity to publish this book together. To MASP for being one of the most important art museums in Brazil and the world, providing researchers with space for experimentation and knowledge.
Image: Coffee and tablet with the book cover (Kaboompics.com, Pexels).
Please help me improve my English by submitting your suggestions through this contact form.
