Something is brought together: Building bridges between Computer Science and the Arts
In September 2020, a collaborative project between visual artist Lauren Moffatt and V-Sense, a team of Computer Science researchers at Trinity College Dublin, was nominated for the prestigious VR Art Prize, for a piece entitled Image Technology Echoes.It is an “immersive experimental fiction” in which an older man and a younger women exchange stilted dialogue about the work they are looking at. The results are due to be announced soon, but this is just one example of collaboration between the V-SENSE team and artists or literary scholars.
Artists and producers of creative works often have a certain scepticism of the dominance of technology. But what if technology could provide another way to create stories and make art? What if the reader of a novel, or the viewer of a play, could step inside the story and embody a character, become a part of the text?
Led by Professor Aljosa Smolic, Trinity’s Professor of Creative Technologies, V-SENSE, according to their website, are keen to partner with both academics from the Arts and Humanities and with artists to create visual effects for audiences using augmented reality and virtual reality. Virtual reality (VR) refers to the creation of a fully immersive environment, transmitted to the viewer through a headset. Augmented reality (AR) is where objects or environments in the real world are enhanced by computer-generated details.
Aljosa emphasises the importance of creativity to their work. He previously worked for Disney and views technology as an asset to creative production and vice versa. He contends that it is important for colleagues from Computer Science and Arts backgrounds to meet and to have an open mind. If this happens, sometimes “something resonates, something is brought together, someone has an idea.”
An example is MR Play, a collaboration between Néill O’Dwyer and the V-SENSE team, and Dr. Nicholas Johnson from the Department of Drama. Nick Johnson is the co-founder of the Trinity Centre for Beckett Studies, and Néill O’Dwyer is part of both the School of Computer Science and Statistics and the School of Creative Arts. This work resulted in “research experiments at the cross-roads of the performing arts and computer science,” and showed Beckett’s masterpiece, Play, written in 1963, in a completely new light.
MR Play used both VR and AR techniques by positioning the audience as a 3rd person viewer and by immersing the audience in the world of the play. The premise was that “by placing the viewer at the centre of the storytelling process, they are more appropriately assimilated to the virtual world and are henceforth empowered to explore, discover and decode the story, as opposed to passively watching and listening.” To this thesis, Nick Johnson provided an understanding of “the complexities, technicalities and nuances of staging Beckett productions,” without which audiences (particularly those familiar with Beckett and the play) would not be convinced.
The importance of collaboration
Aljosa acknowledges that “the bridge between artists and computer science is a wide one, but it is essential that we connect.” He said creative people are often “the end users of what we are doing” and without their input “we could invent stuff which is not useful at all.” Aljosa says that without the input of AH academics and creative minds to their work, they would not be able to achieve what they wish to achieve.
There are limitations to their work based on, among other things, a lack of space to experiment (the V-SENSE team are based in Stack B). But “the limitations developed into an advantage,” Aljosa says, because the technology can now be used under less-than-ideal conditions. It can be affordable, for everyone, rather than exclusive, professional, and specialised.
The Trinity spin-out company, Volograms, seeks to democratise this technology and make it widely available. Funded through Atlantic Bridge, Enterprise Ireland and Suir Valley Ventures, the focus of the company is more commercial than the “blue-sky, idea-driven research” of the V-SENSE team. Aljosa, who did not take a role in the company but acts as an advisor, describes the idea of “teaching a computer to learn from examples.” From there, a computer could produce the work itself, “without us programming it.” He says that the computer processing power and data available on this has expanded widely in recent years and they “can do things now that were not possible before.” Looking to the future, this could mean that conversations through, for example, Zoom, are carried out in 3D.
The V-Sense team have worked on other creative collaborations in Trinity, including putting Jonathan Swift into the Old Library.
They used what they call “volumetric video” technology, which allows for “an interactive narrative whereby a visitor can either engage in VR tour of the Long Room [from anywhere in the world] or ‘augment’ their real visit [on site] by using an AR app that can run on a mobile phone/tablet or an AR head-mounted display.” Swift addresses the audience directly, providing the viewer with a sense of what Swift might have been like, personifying him and his characteristics.
Another project by the V-SENSE team was Mixed Reality Ulysses,where the viewer actually embodies the character of Stephen Dedalus from Joyce’s ground-breaking novel and engages in conversation with Buck Mulligan on the roof of the Martello Tower at Sandycove.
The audience can view this, through virtual reality technology, or actually go to the real-life location and experience it through augmented reality, using a phone or headset. In the latter instance, according to the V-SENSE website, “the staging (mise-en-scène) must be very carefully considered because the viewing environment must give meaning and context to the action taking place.”
An advisor on this issue was Dr Sam Slote from the School of English, an expert in Joyce Studies. He provided advice and guidance to Aljosa, Néill O’Dwyer, and the team on the Ulysses project. He had seen the collaboration with Nick Johnson and recognised what the technology could be capable of. The benefit, he said, is the option to “present a more immersive visual experience” to the viewer. But there are still challenges in the technology’s capacity to depict this.
“Reading a text is very different to seeing a text,” Dr Slote says. “The text doesn’t present everything. There are pointed gaps in representation.” When using Joyce for this kind of VR/AR work, there is a danger that the scientists “could over-elucidate which would then distort.” Dr Slote says that it was useful for him, as an academic, to think in a practical way about what should and should not be represented when experimenting with Joyce using this technology.
He gave the example, of the line from the first chapter of Ulysses, “Telemachus”, in which Buck Mulligan (depicted above) is on the roof of the Martello Tower with Stephen Dedalus: “They halted, looking towards the blunt cape of Bray Head.” Dr Slote pointed out that, in a VR representation of this text, this does not mean that Bray Head is visible. It means that the characters look in the direction of Bray Head, not necessarily that they are seeing it. It is important, Dr Slote says, that the narrative perspective in this representation is limited by what Stephen can see.
A full production of Ulysses is in the pipeline, spear-headed by Néill. This might tie into next year’s centennial anniversary of the publication of Ulysses. Dr Slote says that the technology will need to cope with the complexity of the text, including Bloom’s wanderings around Dublin city centre, and scenes in which various characters are featured simultaneously. He is keen to see what might be possible.
There is “great potential,” according to Aljosa, in creating connections between Computer Science and the Arts. “Trinity is very strong – world-leading – in the Arts and Humanities,” and there is a “growing understanding in Industry that this is needed to be successful.”
Article by Dr Kate Smyth, Consultancy Development Officer, CONSULT Trinity, Trinity Research and Innovation.
For more information on academic consultancy visit CONSULT Trinity
All images courtesy of V-Sense