Our development as a group of artists working with technology has been made possible by, to our knowledge, the longest and deepest collaboration between an artists’ group and a university in the world. Working closely together, we have created 12 new artworks – from VR warfare installations, to interrogations at the Venice Biennale about political violence.
This year we’ve been celebrating the 20th anniversary of collaboration between Blast Theory and the University of Nottingham’s Mixed Reality Lab. Matt Adams spoke to Steve Benford, Professor of Collaborative Computing in the Mixed Reality Laboratory at the University of Nottingham.
Q: First of all Steve, how did you meet Blast Theory and what were your first impressions?
Steve (MRL): We met through Andrew Chetty, at the Nottingham Trent residency. Water spray, pre-Desert Rain. I must have had conversations with Andrew at the Now Festival and he said I should really come down and meet you. First impressions were, you folks know what you’re doing. I already had some sense of your approach and the excitement of what you were doing.
Matt (Blast Theory): I can remember very clearly meeting you after the demonstrator that we did. We were at Nottingham Trent for a week looking at experimenting with the effect of projecting into water spray. I can remember having a discussion with you and you kindly inviting us to the Mixed Reality Lab (MRL). I remember it very vividly because it was the first time I’d ever been in computer science lab. Even just being in an environment with lots of high-powered computers was completely novel to me. I think you had silicon graphics machines: you were working with fairly state-of-the-art virtual technology. Then you set me and Ju loose on your massive VR environment and left us to it for an hour. We had that sense of there being a really thrilling set of possibilities, both in terms of the skills and knowledge you had and the scale of your operation. At that point we were still sharing a single computer between the five people in Blast Theory and it was an era where having your own computer was something quite exciting.
Q: Steve, how would you describe Blast Theory?
Steve: If I were doing a hyperbole for the marketing, I would say world leading and cutting-edge in terms of mixed reality performance and showing the possibilities of what it can be. At a more personal level, it’s that combination of great creativity but also, and you may not see it this way, but a fairly ruthless organizational ability to manage that creative process, to move between moments of being really open and plain, and moments of drilling down and really focusing. I think that became clear during all the workshops we did, and it’s not something I have seen elsewhere in the same way.
Q: Matt, how would you describe the MRL?
Matt: I would describe the MRL as an interdisciplinary team at a very deep level, with a profound understanding of what it means to be committed to being interdisciplinary; as opposed to just combining one or two disciplines. The MRL not only combines multiple disciplines but then thinks about, for example, a software problem and how a different discipline, like a social science, might contribute and vice versa. I always find that exciting. Also, Steve you have a real passionate interest in creativity, and the fact you’re a musician and that many of the people who you work with are artists and creators as well as software developers, coders and programmers, that makes a massive difference. So the reason that the research that you’ve done has stayed really relevant and that the collaboration that we’ve had has stayed so important to us, is to do with that mutual process of listening very closely to one another and an interest in learning from one another.
Blast Theory likes to feel as though we have an interdisciplinary approach, and Nick Tandavanitj has a big part in this. While he’s very modest about it, he has enough technical skills and knowledge to engage in discussion with the people who are dealing with the software and hardware. I think Nick has been a fantastic bridge for conversations where Ju and I would both admit that we don’t fully understand what’s being discussed.
It’s also important to say that you personally have mentored me and Blast Theory a lot over the years. You’ve helped us learn a tremendous amount about research as a process. The resources that the University of Nottingham has and the funding relationships we’ve been able to forge are a critical part of it too. The relationship has enabled us to work at a level we would never have been able to work at, and it’s given us access to technology we never would have had access to. It’s also enabled us to explore and play with technologies like GPS, WIFI networks and location-based technologies. Virtual reality (VR), mixed reality (MR) and augmented reality (AR) are things we wouldn’t have had the opportunity to experiment with, or learn from, had it not been for our relationship and that opportunity has been pivotal in our development artistically.
Steve: Again it’s absolutely a two-way street. The eRENA project was where we had been doing creative work like Out Of This World at The Green Room, alongside International Symposium of Electronic Arts (ISEA). Technically it worked well given the complexity of what we were trying to do with head mounts and multi-user VR. But placed in front of the audience and then standing up and asking what they made of it they responded with incredible hostility about any content or meaning in the work – and they were quite right as well. That’s very rare in our field – you get beaten up in written reviews but you won’t stand in front of an audience and take bullets like that. Without the collaboration with Blast Theory we wouldn’t have been able to work practically in an artistic performative context.
Matt: So is there an element to which the relationship is useful for you in terms of taking your research out of the lab?
Steve: When I frame to people why we think it’s great to work with Blast Theory and people like Blast Theory, there are three primary reasons: one is because we do actually care about the area, we want to help make cultural works and so collaborating is important. Secondly, there is the new ideas, creativity and challenge to technology that working with you brings. It makes our research collectively more innovative. Thirdly, the world of theatre is a laboratory for being able to engage public audiences with technologies in a quick and iterative way. So we care about the area, it’s incredibly creative and it means we get to do unusual things with the opportunity to engage with people.
Q: What is mixed reality and what does it have to do with art?
Steve: There’s a lecture course! My favourite definition of mixed reality is Milgram’s and Kishino’s from 1994: Mixed reality is a spectrum of different kinds of experience, is how I would paraphrase it. It ranges from you thinking you are in the everyday world and new things have emerged around you, to thinking you’ve been transported to some entirely different world that is VR, and lots of other possibilities in-between.
Another definition of mixed reality is that it’s much more a journey through those possibilities. It’s not just in VR or AR, but you’re taken on a journey from where you are through a mixture of physical and digital things to land back somewhere else, and that is what our work together has taught me. In a lecture course I give, we spend a long time talking about the spectrum. But at some point I say, let’s throw that away and talk about journeys and how you move in and out of different modes and moments of engaging with the digital and physical in different ways.
Finally, I think Milgram and Kishino’s definition I gave at the beginning was about essentially here ness or there ness – am I here with the stuff coming to me or am I going there? However, I think there’s another way of beginning to think about it. Mixed reality is about creating different sensory alignments. Sometimes it’s about what you’re hearing, seeing, touching and feeling and how they do or don’t line up in different ways. In particular when they don’t line up, that’s when you get really interesting effects.
Matt: Great. The definition I would add as a layperson is, I feel it’s an incredibly prescient way of thinking about the modern world, because it suggests that your experience and a software layer are co joined in some way all the time. Originally, that was a very narrow technological thing and it took enormous processing power to create a mixed reality where you are both in a real space as well as in some kind of virtual or digital space. But now that’s everyone’s daily experience all day long.
Mixed reality is then a lens into what it means to walk down a road with something in your pocket that is connected to the Internet and is calling information into your pocket without you doing anything about it. It’s affecting how you walk down the street, where you walk, how you define yourself and what experiences and relationships are possible for you. It feels like we’ve been on this incredible journey of thinking about it as quite a narrow technical word to the permeation of the electronic space into our perception and how we think about the world. To me, being able to make works in mixed reality has enabled us to work with some of those ideas to try and find entry points into thinking about some of those questions.
Steve: I agree, I think it’s a great answer and a tricky way of seeing it. Mixed reality gives artists the tools or the lens to absolutely reflect on where society is and what modern life is like and how essentially everything is a mixed reality. Again, I think that’s something that of all the people we’ve worked with, your work does more than anyone else’s. It says, actually let’s use our work as a lens to think about surveillance, for example, and really interrogate that.
Matt: What else do you think artists and universities can learn from each other? From my point of view, what fascinated me is learning about a research process and what a research process looks like. In the arts, people talk about research all the time but it typically means going to an archive or a library to find out, for example, about cooking in the Middle Ages, for the purposes of then bringing that knowledge back into a piece of work or a fiction. It’s gathering information – it’s not posing questions, rigorously breaking those questions down into fungible chunks, then looking to answer those questions, bring that knowledge back, and share it in a peer reviewed world. For the first three or four years that we were working together, I was constantly learning and being tripped up by my ignorance of that process and what it means when your job is to produce really strong research papers, not just a great experience for a bunch of people at an arts festival – when that is purely a stepping-stone towards the work of real value.
I come out of a theatre background and it’s a tits and teeth world where as long as people love it on the night, you’ve done well. We learnt a tremendous amount from trying to understand and unpick the ontological implications of what it means to be asking questions that you don’t know the answer to, and then rigorously trying to answer them. Then there’s the issue around software development processes, and how you manage an interdisciplinary team of people who are working together around a certain goal. I certainly feel working with the MRL in the first few years we were just soaking up all that information about how you work and how you structure a team.
As an individual Steve, you have a very distinctive style of management. You’re very open and permissive in terms of how people go about their day and what they do. Yet you also have a tremendous clarity of focus around when that looser work is starting to arrive at something meaningful and significant. You have an ability to be very divergent early on in a process and allow lots of things to breath, and then you have a convergent impulse that you apply to sharpen things down towards a question. Then the ancillary to that is you brought us into relationships with other researchers and industrial partners like SONY, NOKIA, the BBC and British Telecom who we then learnt huge amounts from. Without the University of Nottingham as the broker, we would never have been able to join those kinds of relationships.
Steve: Interesting. Again, there’s very much a mirror picture. From you we learnt about practice-led research. What we do, certainly early on, was very different methodologically from a lot of the research our peers were doing in computer science, and even in human-computer interaction. Over the years, lots of other design-led methods have come to the fore, and we’ve had to think what we specifically do with performance in light of that. That’s a whole journey we wouldn’t have gone on without being able to work with you in a practice-led way.
It’s really interesting what you said about divergence and convergence because that’s something I completely see in your process. What always impresses me is the ability for you to allow that divergence and convergence in your own process – to have moments where things do diverge and go off into different areas and it seems unclear, deliberately unclear, as to what might be done and what might come out. But then you have moments of focus to bring it together and also a willingness and ability to reflect, which is a very research-oriented thing to do. Possibly why we fit together really well is because those processes dovetail nicely.
Impact of Technology
Q: Increasing our understanding of how human behaviour is shaped by technology is at the heart of your 20-year relationship, what’s the single most surprising thing you’ve learnt?
Steve: Single – that’s the killer!
Matt: One thing I’m really struck by is the relationship between software, which we think of as structured, deterministic and mathematically governed, and the perception and desires of the people using it, and how this incredible interplay happens between those two things. A good example is in Can You See Me Now? Where we’re using GPS reported location. We’ve got performers running around the streets and GPS is placing them into a virtual model so you can see an avatar of a figure running around.
At times the GPS is wrong and you can jump from one street to a street 200 meters away in a split second, then five minutes later it will flip you back again. Hundreds of millions of dollars of satellite hardware is reporting my position as being in one place, and yet I’m on a walkie-talkie perfectly capable of pretending I’m in some other place.
So there’s a bizarre melding between the raw computer science and these perceptual tricks – essentiality I’m a performer manipulating the situation. However, often the public themselves invest in the idea that this technology is futuristic and therefore they overlook their own evidence which shows it’s broken. They are folding that brokenness back into their experience and making it whole and coherent again; to me that run through a lot of the work we have done.
Steve: Yeah and I think that was definitely one of the big results research-wise. Specifically, we used Can You See Me Now? in a paper to articulate notions of seams and Matthew Chalmer’s ‘seamfullness’ which was a big finding.
Matt: Can you explain in a sentence or two what you mean by seams and ‘seamfullness’?
Steve: Seams are the gaps or joins in technology where they are knitted together and where things fall down the cracks, for example the holes in GPS and the way GPS connects to WIFI or doesn’t. There are all these kinds of seams in the weave of the material matter. Then there is the notion of simple design, of not just trying to hide those seams but sometimes celebrating or exploiting them. There are different strategies for how you deal with it, but you have to at least recognise that the digital material has got its own qualities and its own seams.
Matt: I would say the classic example of that are the bars on your phone telling you how much connection you’ve got. In Apple’s ideal world they would be completely non-existent, they would never show that to you. But, of course they have to make that seam visible because it’s so much a part of your lived experience as to, why is my phone not working now? Oh I have naught bars or one bar – it’s the seam we use every day.
What keeps the working relationship so strong and ambitious over all this time? That’s a big assumption that it’s both strong and ambitious there Steve!
Steve: Let’s live with that shall we! Strong – it’s got to be respect hasn’t it? You really have to respect each other’s ability, knowledge, way of doing things and goals. That’s absolutely at the core of it. It has to feel – in that regard – like a meeting of equals.
Matt: I would say that. I think also the depth of history we have together is an incredibly fertile soil to draw on. The fact that we have been through so many iterations of work together, so many forms of research and artistic production means there’s lots of shared understanding. There’s lots of ways in which the depth of knowledge of each other’s processes and what we do is really applicable in terms of how we work now.
In the early years, I remember feeling extremely nervous and exposed when the creative process went in a slightly different direction, worrying that we weren’t doing what we said we were going to do. Now when we’re a bit stuck or we’re off on a tangent and it’s not quite clear where we’re heading, I feel comfortable with that because I know you trust that something will arrive out of that process and we’ll get through. I think that 20-year history is rich material to draw on.
Steve: One of things that’s so much a part of shaping that is our involvement with various European projects on the way. If there’s one thing that being involved in a European project teaches you is that no one ever got anywhere by doing what they said they’d do.
You’ve just got to do what you want to do, live with the consequences and be prepared to stand up and take whatever comes your way. With that said, the other obvious thing that makes the relationship strong is always having a common enemy really.
Q: The kinds of interactive experiences you’ve created together using elements of MR, VR and extended reality (XR) are today very much part of the mainstream. What do you think the next step is in terms of technology-driven interactive experience? Or what are you interested in exploring further?
Steve: There’s a couple of ways to come at that. Where is the technology going? I’m not sure that’s the way you ultimately want to approach it, but I suppose you can think about it. There’s clearly an ever deeper physiological, and now a cognitive and mental, connection between humans and machines – so that’s a technical boundary that’s full of challenge and possibility. Certainly, in recent years we’ve been very interested in things outside the body that move around as carriers of information and stories. I kind of worry whether MR, VR and XR are really mainstream, I’m not sure they are. A big challenge to that question is about whether people playing with those technologies are thinking about them in the right ways.
Matt: Yeah, I think that’s true. It’s important to say this is at least a third wave of VR being fashionable and the thing that will change the future of the world and there are probably others I’m not aware of that came before my time. Certainly, the late 1980’s and early 90s was a VR moment, the late 90s was another and we’re in one now. But at each of those junctures the idea that VR would change the world has seemed obvious, and yet it hasn’t yet come to pass.
Because the work we make is interactive, the on going question for us is how does art and culture change in a world where the audience – however broadly you understand them – is interacting with the cultural artefact – whatever that might be. The 20th century was broadly an age of industrial production in culture: in film, radio, television, popular music, and now we’re into a two way street in terms of interaction. And I think in the 21st century we’re in the very infancy of understanding what that two way street looks like and what the possibilities are there.
Q: Projects such as 2015’s Karen seem all the more pertinent today in light of revelations about Facebook’s use of personal data. Do you think this knowledge will change the way we behave online?
Steve: Yes and no. The problem with the personal data issue is it’s really hard to answer in the abstract and so much of the time it feels like that’s what the debate is about. It’s about; they’ve got your personal data, are you bothered or aren’t you bothered? How can I possibly answer that question? What personal data, for what purpose and what might happen?
I think the interesting thing about works like Karen and some other works we’ve done, are that they make the debate more concrete. It is in the moment, in those choices you make, that the ramifications become real. It takes particular incidences of provocation to change people’s behavior online. I guess either those are, if you like, real things or maybe they can be foregrounded through artistic works that make you think and put you in that situation. But it is artistic works that will help drive that, rather than abstract debates and reflections.
Behind the scenes on Karen
Matt: Yeah. I think the thing I would say is when we started work on Karen, with the National Theatre of Wales in 2012, Facebook was the Wild West and it was quite clear that data was being heavily manipulated and misused which was one of the reasons we got interested in it. We were aware of software tools that were written to scrape data out of Facebook and initially that was extremely simple to do. Facebook have gradually tightened that up.
On the broader question, my perception is we have this tension in our culture between individual agency and accepting that we live in a society in which the collective is the most meaningful place in which to conduct major decisions. Hither to, we’ve been in a world where we are all personally responsible for sorting out our data – if you don’t like it don’t do it, if you don’t want to don’t share it and be sure you’ve got the right sort of data hygiene principles – that you’ve got proper passwords. All responsibility is pushed onto the individual, as if you yourself can deal with all of these problems. Of course, what happens to data and who owns it is actually a society-level problem.
I hope that we are moving in a direction where ultimately data will be seen as a personal asset that you own and have control over, but I think we are a fair way away from that. I broadly end up in the same position as you, which is to say that the needle has been moved a very small amount, but there is still a long, long way to go.
Q: Musical artist Brian Eno is currently collaborating with software designer Peter Chilvers to explore the use of video gaming technology in music. Together, you’ve collaborated with theatre companies and presented work in major visual art festivals. Is a music collaboration a next step for you? Who would your dream musical collaborator be?
Steve: Musical collaboration would be fantastic. We are now year four of a five-year exploration of a kind of adaptive, semantic music platform project. This has transformed quite a lot of thinking in the lab about the nature and importance of music. I think that using music to enhance, change, distort, re-map your everyday experience is really powerful and possibly something we have not done enough of.
Just a really small example, but more recently we tried out some things with Jocelyn Spence in Nottingham around gifting and making gift experiences, and there were things about replaying sounds from the city back through headphones at other times when you were in the city. It was really, really powerful because it’s just so easy with sound to blur those boundaries. So technically it’s really compelling and practical doing stuff that focuses more on music and adaptation. It would be something great to do.
Ideal musical collaborator? Let’s see, I don’t know. All my ideas would be completely inappropriate! Probably Béla Fleck the banjo player, but I don’t think he would necessarily be a suitable collaborator.
Matt: I’ve always had a fantasy of doing a big musical collaboration, it feels super exciting. In fact Andrew Chetty, back when we were first meeting each other in the mid 90’s, said “Who’s the band you want to work with?”. I can remember then saying, oh yeah DJ shadow, we’ll collaborate with him, which would be amazing. But strangely – in terms of our work – those big high-concept kind of things have never really gone anywhere with us. Ultimately the work comes back to me, Ju and Nick in a room together, scratching away at something quite small and extrapolating from that towards a piece of work. We’ve found it quite hard to start with a big-ticket thing like a major collaboration. All our works have been collaborations, not just with MRL, but with countless other creative people – artists, writers, performers, performance makers and so on. But I hope one day we’ll find a way to make it happen.
Steve: I would say doing something more with music and sound would be great and maybe would bring us onto our next collaboration together. It would really work for us. But actually I don’t think the way to do it is to sign up a big music artist. It’s not how I would approach it. I would think about the experiences and work with people who are much more at a cutting creative edge.
Matt: Yeah okay great. Is there anything else you want to say about our next future collaboration with each other?
Steve: Yeah, obviously we had an interesting moment where our current collaboration is very much on the research end of the spectrum, which is an on going relationship we’ve also had over the years. I’m really looking forward to watching the gifting work play out and roll out. But obviously, it would also be great to think about what a performance collaboration would be, and one that would be driven by making a work would be really interesting.
Matt: The work we’re doing about virtual museums now, which we’ve been working on last year, this year and we’ll work on next year, is fascinating and it’s a very granular, fine grained examination of how people walk through a museum and what kind of experiences you can create for them there. But looking further ahead I think we want to do large-scale collaborations again and for me at the moment interactive cinema, or immersive cinema, is very interesting. The boundary between live event and cinematic presentation and thinking about an audience in a cinema being able to interact with the film in real time, those are the things that I find very exciting and a massive challenge. That’s something I’m excited to explore more.
Steve: I think something that interrogates the notion of liveness in a really deep and interesting way, as part of that would also be very interesting. Moving onto new works would certainty be a primary goal but re-visiting an old one would be great. Certainly good to mention both Chris Greenhalgh and Martin Flintham with that, and I think we all feel enthusiastic about it. Desert Rain is always the obvious one to contemplate whether it’s possible. For us at MRL, to articulate the challenges and ultimately the solutions to re-creating a piece of artistic work like that would actually make a very interesting research study.
Q: What was Desert Rain like for you as an experience?
Steve: That was a game changer, for the scale, the theatrical staging and the fact that, although the technology of the rain curtain and multi-user VR were interesting, they were embedded into something much bigger. I remember that was an eye opener, it was really exciting to be there.
Matt: Yeah, from my point of view we broke a lot of ground in a number of ways. One of the reasons for that was because the project was delayed by a year through a partner dropping out quite late and actually that was an absolute silver lining. We were already quite advanced in our thinking, and then it got put back a year and we had the opportunity to raise additional money and spend more time developing it. We had a residency at the ZKM in Germany where we were able to spend two weeks on nothing but how the projection hit the water spray and experimenting with different projectors, different lenses, different angles of projection and light levels – lots of technical work there. The big artistic shift for us was it was the first time we made a game. We suddenly realised in the development of that work, oh we can make this into a game and at the same time use it to address issues like how many people were killed in the 1991 Gulf War, and how the hundreds of thousands of casualties were lost and no one really knows to the nearest 10,000 how many people were killed.
It’s an unknown number. So there was something there about this incredibly mechanical warfare where there’s precision weapons, with all of the jargon, and yet there is no precision about how many people were killed by them. This distortion of how technology affected our attitude towards killing we see still going on very much today. It’s still a remarkably live issue in terms of the work we are making this year about commemorating the end of the WW1. And considering warfare today, there’s the issue of how much people in Britain feel responsible for the civilians who were killed in Syria by British munitions – that is still partly to do with the technical apparatus of warfare.
Steve: I think that was the thing about Desert Rain – it was difficult and an eye-opener to try to understand what the work was about and realise the importance of that. Again, in terms of that first stage of the research journey, it was the first time we’d been in that position. That was the start of the journey and that was a very notable thing.
Q: Can You See Me Now?, how do you remember that?
Steve: The whole bridging realities idea was really novel and probably the first time it had been done; certainly done properly. Making work that toured and what it involved was really interesting and our first time doing that. Also, the sense of how powerful that audio connection was to a remote space, how live that felt, the first time I experienced that it was really exciting.
Matt: Yeah, and that comes back to what I was saying earlier about how the theatricality connects with the technology and how they support one another. I often feel like there’s a credibility threshold that you have to get across with projects like this that are really technically complex. Where it’s breaking over and over again and the experience is irreparably damaged by that, then you just reach this threshold where you get over the top of it and people experience it, and then they allow for some of the seamfulness, some of the gaps in the technology and suddenly they aren’t a problem. I remember that very vividly.
I think the thing I remember from Can You See Me Now? is we would never have made it without our relationship with you. You were looking at GPS, you understood what GPS was when I had no idea really what it was or how it worked. Then you were able to buy some GPS receivers and we just did really simple experiments of: I’ll walk around Nottingham city centre, lets see if it tracks us on a map as we walk. At the time that was a [impressive] technology. Then us trying to think about creating the simplest way of testing this in terms of experience design, and coming up with this very simple chase game. It’s a playground game of one person chasing another in a way, except one person is on a street and one of them is in a virtual representation of that street.
Q: What about Uncle Roy All Around You?
Steve: Incredible memories. It felt like a really hot time in the summer, being in that really crowded office in the Institute of Contemporary Arts (ICA). I still walk past Uncle Roy’s office quite often. Every time I walk past that office, that door, I am back.
Matt: Me too.
Steve: I’m back to that experience, which really means something to me – it was compelling! Then, of course, as a project it really changed our thinking about blurring the boundaries between real and fictional, physical and digital, and the opportunities, risks and challenges involved. That was a big game changer and probably most shaped the trajectory of stuff. If I teach trajectories now, that is the one example I would pull out that had it all. I think it’s really interesting.
Matt: Yeah. One footnote, but which I think is really important, is of course the moment you brought in people from British Telecom to help work on the project, among a number of other partners. Amanda Gower and John Sutton coming in really helped and they threw themselves at that project at a critical moment. Obviously, there were many key people from the MRL – Martin Flintham most particularly, who without him we really couldn’t have made most of these projects. He’s been such a key collaborator.
But from the artist side it was the moment at which we became fascinated in trust, in virtual space, in mixed reality space and what do we mean when we talk about communities online. It felt like a very exciting idea to take this risk of saying will you enter a contract to look out for a stranger for the next year based on you experience in this game today? We had no sense of whether that would work, what that would mean or how people would interpret that. There was a real sense of the unknown and we were curious either way really. I don’t think we necessarily felt it would be a big proof of concept if loads of people looked out for one another. But I do know anecdotally that some very firm relationships were formed through those anonymous pairings.
I think I Like Frank, which is the next work, that we made in South Australia together, was a key development of that idea and, of course, it was the first time we ever worked with 3G. It’s at a moment where 3G were a very new technology worldwide. We were lucky enough through our Thinkers In Residence invitation in Adelaide to spend time together, MRL and Blast Theory, on the ground in Adelaide experimenting in a 3G test bed at a time when 3G didn’t exist in the UK. It was extremely exciting to understand what 3G might mean, and for the first time in 2004 to work with mobile phones connected to the internet. It was thrilling and fascinating to play with some of the same ideas but in a totally different part of the world and with a totally different set of technical challenges.
Matt: Lets talk about Day Of The Figurines. I suppose just quickly what I would say about that is, it’s the first time we turn away from cutting-edge tech. For a while we are working with the most leading edge technology we can and we are doing a lot of software and sometimes hardware development to create these works. Can You See Me Now? and Uncle Roy All Around You existed before WIFI really existed as a domestic technology, so we were setting up our own WIFI networks. In Day Of The Figurines we decided to work with SMS, and this is one of those moments I can remember really clearly where you trusted us as we took quite a left turn. The narrative around our collaboration is we are working with the latest technologies to make groundbreaking artworks, and then suddenly we pop up and say we want to do the next thing with text messaging.
But we were excited to examine what you could do with a universal technology and to think about that as a way of putting culture somewhere it could have never been before – in your pocket all day long. Day Of The Figurines is a game that lasts 24 days so what is exciting this game is part of my life for the next month, and when my phone beeps it might be my mother, it might be partner, and it might be this game nagging me or inviting me to do something.
One of the most meaningful experiences for me as a player of Day Of The Figurines was being in a queue at an airport for a flight, and every 30 seconds a message came back from another player who I had randomly bumped into in the game. She and I were communicating backwards and forwards for 20 minutes, 30 minutes in this incredibly rapid-fire way and then she moved on to another part of the game and so did I and I never saw that player again. But we had this very strong moment of connection within that game.
Steve: I have a very vivid memory of engaging with the game. I remember being there in Berlin, seeing the table, seeing it starting up. I also remember playing it in Sardinia on the beach. I think the key lesson from that piece of work, that followed through to Ulrike And Eamon Compliant and other works, is instructions – voice instructions, it matters so much. It’s so much a part of your work and it’s so much the thing that makes the connection. I’m probably moving more to Ulrike and Eamon [Compliant] territory here, but how in such a small piece of text you can convey setting, instruction, mood and framing: and that is an art. That’s certainly something we’ve published about and certainly something I tell my students about now. You could throw away all the GPS and you could make a brilliant experience if you only understood how to give people instructions.
Day Of The Figurines, 2006.
Matt: It’s definitely true, and it’s a theatrical skill. It’s also something about reciprocity, which is how we as artists try and put ourselves into the work, to put some kind of vulnerability into the work. Ju is absolutely the queen of this, she does this incredibly well and I think Rider Spoke is a great example. The opening text of Rider Spoke was largely written by Ju and recorded by her, and she sets the tone beautifully. The grammar is slightly technically incorrect, she uses tense and subjectivity in a very slippery way where often her sentences are moving tense within a sentence. And there is a kind of poetics there where she is playing with a sense of it being absolutely her voice, and being a voice that feels fallible. It’s not always an instructional or authorial voice, although we do sometimes use an assertive and instructional tone. But then nearly always there is some play of vulnerability and openness within that voice and, as you say, it sets a mood but it also invites the public to [experience] a different tone. Rider Spoke for me is all about that.
Also, the quite poor hand-drawn interface that I did. Both Ju and Nick have been to art school, both of them actually have the skills to draw where I have no drawing skills whatsoever, but in that instant it made sense for me to do those drawings. The poor quality of those hand drawings became a distinctive part of the aesthetic and we all realised that something where the perspective is poor, and the scale of differing objects isn’t appropriate really felt right for that project.
Steve: Again, a really interesting thing to work out was how those things combined with the range of movement of being on a bicycle and being in a city at night – all those other things that came with it – and how the gentle physicality creates that moment. So much of your work and so much of my interest in Human Computer Interaction (HCI) has become about creating that sublime moment, whatever that is. How do you lead people to that point where the world shifts a bit? That’s what it’s all about.
Matt: I think with Rider Spoke, as with a number our works, you are out on a street alone. It’s true of Uncle Roy All Around You, I Like Frank, Ulrike and Eamon Compliant; you are alone on the streets and that’s a big part of it. It’s the sense of being slightly isolated, slightly stripped of your moorings.
In Uncle Roy All Around You we asked you to give us your wallet, your phone, all of your possessions before you go out onto the street. You are denuded of all the tools you might normally use to support yourself out in the city and I think that’s a really key part of creating a kind of openness. It puts you in a position where you are alone, on a street corner, at a certain moment and sometimes you are saying something you would never have said out loud before.
And that’s when it’s really working. It’s fantastic to listen to the recordings from Rider Spoke and to hear that going on. Sometimes you can hear people in tears, sometimes people sharing secrets, sometimes people sharing scurrilous and outrageous stories. But you can hear some degree of real openness and precise intimate experience going on.
Let’s delve into You Get Me, the project with the Royal Opera House. Again, Martin Flintham played a massive, pivotal role in helping realise that work, and as did the MRL more generally. We worked with people from Mile End park in the East End of London to create a game where people in the Royal Opera House are invited – if not forced – to listen carefully to what those teenagers and people in their early 20s are saying. Then they had to ask them questions that are appropriate and well attuned to those people, and to build trust with them to progress in the game. It’s perhaps the furthest we have ever been in terms of a fusion between a conversation with a stranger and gameplay. You put in your own mobile number, they ring you and you are one-to-one with them on the phone. That’s the ultimate reward of the game – it’s a personal conversation. I don’t know if you have any memories of that work?
Steve: Yeah, I think you are absolutely right to flag up the enrolment of Martin and also Chris and the platform work was so consistent throughout the project.
Matt: Chris Greenhalgh, of course. Then we worked together on Flypad, an 11 player mixed reality experience made bespoke for the Public Museum in West Bromwich. It’s a museum with a tortured history, of which we became a part. I can remember Flypad being shown on the 10 O’Clock news on the BBC as part of a damning report on it’s use of public funds. It was always a difficult project.
Steve: But Flypad played out for quite a long time.
Matt: For many years.
Steve: It was incredible and a really complex piece of design – working out how interaction fits within the architecture of the space and the walkways in that really complicated building.
Matt: For me what was exciting about it was that you fly your avatar around the atrium of the museum, and we had modelled the virtual world onto the physical atrium so that you could bump into the balustrades or the structure of the museum with your avatar. Your aim was to keep your avatar flying and to go into “holds” with other avatars and when two avatars bumped into each other and held on they swapped some body parts. The avatars were a mixture of Mexican wrestlers, characters from Peking Opera, and parachutists – very bizarre and extreme body shapes.
It’s a fantastic thing to design works for museums, especially permanent works where it’s got to be there for at least 10 years – it has to be suitable for any age and people have to be able to understand it immediately. So we were doing tests of Flypad with seven year olds and eight year olds. In fact, we worked with school children in West Bromwich to help design some of the avatars, and it’s a very exciting discipline to take something technically complex and make it readily understandable to a seven year old.
Steve: It felt to me remarkably successful as a piece of work, in spite of the context.
Q: And finally, Ulrike and Eamon Compliant?
Matt: It’s a work we first made in Venice together at the Venice Biennale, again Martin Flintham playing a key role. It’s a work for voice calling on phones, so again it’s part of us using everyday technology and we made a number of works together just using voice-calling technology. It’s again us turning away from technologically cutting-edge solutions to really try to think about experientially appropriate solutions. The MRL helped us build a very sophisticated system of orchestration and control, where a number of performers were hidden around the streets of Venice, watching each member of the audience as they go through the experience and triggering content for them at precisely the moment that they arrive onto the middle of a bridge, or sit on a bench. So we were orchestrating with incredible precision the experience of each individual visitor and that’s something the Venice Biennale made possible because it’s such a unique context. It was very exciting to do.
Steve: That was the one experience that really foregrounded for us using discomfort carefully and deliberately. It was evident in other works looking back, and I’m sure for you part and parcel of the practice, but for us as interaction designers that was the moment we went ‘okay there’s something going on here that we need to try and articulate back to our community’ – using discomfort can be good. But, on the other hand, how can you tell that story in an appropriate way? That thread of discomfort and ambiguity, Ulrike and Eamon [Compliant] really crystallised that for me. It was the realisation.