chapter

Sound and Temporality

Jordi Viader Guerrero 00:02 For chapter one, we’re welcoming Carolina Valente Pinto. She’s a researcher at the Institute of Network Cultures. Carolina was the co-leader of the Living Archives group of Going Hybrid. And from the Participatory Livecasting group, we’re welcoming Lilian Stolk, the director of The Hmm, a platform for internet culture and an emoji expert. runs a small background information link about the living archive group here: https://networkcultures.org/goinghybrid/2022/09/21/a-living-archive-is-also-a-dying-archive She cares about understanding the internet and has a special focus on internet languages and the rise of image-based communication. Furthermore, we have Heerko van der Kooij with us on stage. He is a creative technologist working with Hackers & Designers. He gives workshops on the development of hardware and software. Welcome everybody! Seppi Fun fact, everyone participating in the event as a speaker is present today on-site, a kind of mini audience. We’re all trying to be super silent not to disturb the recording!

Esther Hammelburg 00:43 When we started this research project, one of the starting points was my research on liveness. Esther Hammelburg’s PhD thesis was published as a book titled Being There Live!, which you can find on her website: https://www.estherhammelburg.nl At the time, I had just finished my PhD research into what ‘liveness’ means in a cultural setting. When we discuss livenesss, people tend to focus on the present, the ‘now’, and how everything is evolving at this moment. But when you look a bit more closely, it’s not only about the ‘now’. Liveness in fact bridges both temporalities and spaces. For intance, in live experiences we often experience a ‘now’ which was anticipated beforehand. student at the back what about non-real time and non-real space? Our minds thrive in those We look forward to an experience of liveness, we think about this live event that’s coming up. Also, we tend to make videovideos and still imagephotos of performancelive events, because we want to actively create future memories in the ‘now’. In other words, we’re creating a future past in the present. A habit researched by Hammelburg is how visitors post pictures and videos of concerts and festival on Instagram to fulfil the ‘liveness’ of being here, now. Continuing to exist online, these photos and videos extend the liveness in both spatial and temporal senses. The experience of liveness is also profoundly tied to issues of memory and remembering. How do we keep our memories? How do we share our memories? How do we archive them, and what do we do with these archives? These are some elements for the discussion in this chapter. This brings me to my first question: How do you look at hybridity as a phenomenon in the here and now, and how does it relate to the ‘afterlife’? The hybrid happens ‘now’, but does it need to be recorded? Also, I am curious about the connection between time and audioaudio. Who would like to comment?

Here, There and Everytime

Carolina Valente Pinto 03:05 Thank you for that introduction. I’m here representing the Living Archives group and our project MUMORIES. runs link to the MUMORIES project here: https://mumories.hackersanddesigners.nl/Listening The aspects of time and hybridity came up in Living Archives most clearly when we collected people’s memories of MU Hybrid House in Eindhoven. We tried to find out what people get from experiencing an exhibition or an event, and what they remember. Interestingly, it turned out that remembering cultural events often brings back memories from before the event itself. Hybridity is exactly this: mixing different elements into one. The Living Archive is a way to reflect on the past, present, and future, and even translating the past into possible futures. So, yes, questions of the role of the present in this afterlife is particularly present when ‘going hybrid’. In the ‘now’ of events, we have something in common: we’re all in the same space. But in what remains after, different key moments come up.

Heerko van der Kooi 05:12 In the Participatory Livecasting group, our research was very much focused on real-time. Specifically, we tried to give the online audience agency or presence in the on-site space. We were thinking about the ‘in the moment’ agency, which is fundamentally not something you can recreate or play back. We were not too concerned about that aspect.

Lilian Stolk 05:50 But I do think we can reflect on the double notion of ‘time and space’. Space was a prominent element in our research. In a hybrid event, as we saw it, you have spacethe physical space, and many online spaces or people joining online from their own house. In the livecasting group, we built ‘Emoji Proxies & Ghost Messengers’, a tool that could make the online audience more present on-site during an event. Using commands in the livestream chat, online audiences can influence something in the space, like triggering an on-site objectlight or scenta scent diffuser in a spacespace. We aimed to make the relationship between offline and on-site audiences a bit more equal. This is difficult to achieve, especially when there is a main stage, which is the objectphysical stage. As long as we think of on-site programming as primary, it’s very difficult to give online audiences a more equal type of agency. So we tried to decenter the main stage. mushy the virtual space is merging in the background, can it be considered non-real space? For instance, we tested the scent diffuser during an event that was focused on audioaudio. Madame Bertha alexa this is not FaceTime app how did I get here So there was a main stage, but there were no visuals on a beamer. It was a listening event. Klara Debeljak Ultra hyper-focus on the present moment, years of preparation and research leading up to the current moment, with weeks of post-production dealing with the now. How authentic is this now then? This worked well. mushy I think you’re in the right place, welcome!

Hybrid Senses

Esther 08:53 I have a follow-up question for Lilian. student at the back I guess so mushy, it is a background made of numbers rather than actual landscapes Your event was focused on audio as opposed to a screen and a physical space. student at the back welcome Madame Bertha Do you think that matters?

Lilian 09:17 I think so. In this case, it was equalizing the relationship between online and on-site audiences. Maybe because there is one more prominent sense.

Jordi 09:55 I want to continue with the topic of audioaudio. Maybe, to make audiences online and on site more equal, we also need to change the affordances and the space in which the on-site event is taking place. We cannot continue doing the same kind of offline events if we really want to change access and agency, right? And I feel there’s something about privileging hearing over vision, that challenges how we do events now. Carolina, can you tell us more about the importance of listening and how to work with the audience and audio, from the experience of the MUMORIES experiment?

Carolina 10:48 MUORIES is a website and it was also presented as an interactive installation at MU during the exhibition time this summer, in which we were celebrating their 25th anniversary. Even though it was about this specific institutional history, we wanted to open it up to people who had never been there before and therefore didn’t have memories of MU. This led us to play with the idea of speculative memories. What do you think it’s going to be like? What do you think it’s going to smell like? Speculative memories were a bridge to connect practices of audiooral history to these institutional memories.

Carolina 11:37 With everything that’s going on, like the war in Gaza, I think it’s important to reflect on the act of listening. Who is listening to whom? In that sense, MUMORIES which was developed in the context of an art exhibition space, opened up wider possibilities of archiving different types of records (other types of knowledge beond textfacts or numbers) in a way that’s less procedural. mushy the camera on the left is being moved to make a better focus in the physical space seppi Fun fact 2: this chat is not just a live chat, but also the annotation tool of Going Hybrid’s final publication. It will be included in the margins of the print book Screentime Airtime Facetime. Chatting today is co-authoring the book! As you can maybe tell, in our group, we shared a feminist understanding of memory and archiving with oral histories. We have included in this publication a part of the audioradio show ‘Talkshop’ that our group did last month in Screentime Airtime Facetime. See the intermission ‘Talkshop Mumories: Living Archives’. The program reflects on MUMORIES and the memories, on listening to each other, on the privilege that comes with speaking and listening.

Lilian 13:05 Audio was not a prominent aspect of the research in the Participatory Livecasting group, but it is present. As The Hmm, we’ve been updating our websitelivestream website, moving it away from the primary focus on videomoving image and introducing alternate viewing modes. It’s possible to get different video qualities, with or without live captions. It’s also possible to select and audio only mode.

Esther 13:58 Watching implies a power structure. Some people can watch something and some people can take the stage and be watched. Is this different when giving more attention to audiolistening?

Lilian 14:18 It can be more equal. I also remember that we had an event taking place at four locations. See: https://thehmm.nl/event/the-hmm-4-locations. You inspired us to do that, Esther, when you said that a good example of a hybrid experience is a football game that people watch together on a screen but also in a spacestadium. Read the interview with Esther Hammelburg by Lilian Stolk, ‘“We’re Here Together Now” Is the Crux of Every Event‘, The Hmm, 9 August 2021, https://thehmm.nl/we-are-here-together-now. Even though they’re watching a screen, they’re dressed up and watching together, which gives the feeling that they’re part of this live game. We tried to translate that into a cultural event that took place in four different places at the same time. There were two on-site speakers at every location. The rest of us watched a livestream. There was no single, big stage, because the whole event was decentralized. But then, we got feedback from the online audience saying that they missed seeing who was on-site in the different locations.

Decentering Attention

Esther 16:16 As an organizer, you have the power to make people visible or invisible, and you need to be aware of this. I’m also interested in the archives part. Did you focus only audioaudio-related or did you also play with the layering of physical and virtual spaces?

Carolina 16:42 Already early on we had the idea of making a website, which is accessible at home, and an exhibition intallation with a tablet. It’s still possible to browse MUMORIES and even to contribute. Go to: https://mumories.hackersanddesigners.nl/Interaction But it wasn’t easy to integrate our multimedia experiment into an exhibition. During the opening of the exhibition, we tried to create a comfortable setting, like a spaceliving room. student at the back a living archive obviously needs a living room We had a sofa with 25 years of MU catalogs on it, inviting memories to be triggered. We also created a layered audioscape. People on-site could audiolisten to what was happening around them live, but at the same time, recorded memories were being played on the speakers. mushy front camera is being moved too, we want some good close-ups for our hosts :)

Lilian 18:18 For some years already, seeing a rise of the audiopodcast: a medium that is focused solely on audio. But, lately, we’ve come to realize that the podcast is a perfect hybrid medium because you’re always listening to podcasts while doing something else.

Jordi 18:49 I was thinking exactly the same. This automatically decenters the main stage. When you have a audiopodcast or maybe a videoYouTube video that you don’t necessarily need to watch, you’re able to do a bunch of other things and suddenly spaceyour house becomes your own stage. It’s funny because, in the physical space of MU, you wanted to recreate a living room, right? seppi Fellow audience, how are you feeling? Are we blasting into your living rooms? Do you want/have agency? But for me, since I’ve never been to MU, the experience of MUMORIES was the other way around. I was in my living room listening to memories that were created in this space I’ve never been to, and I don’t know what it looks like. It was like a dream or something from the distant past, suddenly acquiring a presence in my living room. The point of decentering the stage, of course, is to give agency to the audience, wherever they are. My next question is about this. Madame Bertha I am enjoying sitting but being there at the same time we live in the future darlings full stop no i meant . Do you have any other examples, comments, reflections, or thoughts about audience agency?

Heerko 20:08 It starts in the tool development. We didn’t develop the ideas for this toolkit by ourselves, but we organized two workshops where we asked people: what could be the ways of an online audience to have agency in this event, in spacephysical space? Then, we developed the camera that we were playing with just now and the scent diffuser we descussed before. So, from a development point of view, these tools are not made by us as prescriptive solutions to a problem, but rather by the audience itself or by the organizers of events. seppi On-site, it does feel like a living room a little bit. It’s warm in this space, people are tip-toeing around behind the camera They aim not a grand ‘solution’, but a tiny intervention that gives the online audience some sort of presence on-site.

Lilian 21:18 It was difficult to determine what kind of meaningful contributions an online audience can have in a spacespace. We often experience that it became a bit gimmicky. Klara Debeljak How to avoid audience participation and participatory events being gimmicky is a big question for sure… They could turn on a light, but how is it affecting the physical event? We also did experiments where we did it the other way around, so that the on-site audience could influence the videolivestream. For instance, by pressing a button they could let the livestream screen turn around. But that didn’t really work because it was enabling the online audience even less because the live stream view became worse. Together with MU and Affect Lab, The Hmm has been collecting many practices of hybrid culture beyond the gimmick into a publicly available repository called ‘Toolkit for the In-Between’: https://networkcultures.org/goinghybrid/2023/10/21/the-hmm-mu-and-affect-lab-launch-toolkit-for-the-in-between

Heerko 22:28 We did these workshops twice, and especially the second time we really consciously took a lot of objectcraft materials and really made it hands-on, where people could invent little objectweird puppets that could be moved by the online audience. dunkirk Back to listening to: I feel that if we focus on listening we shift the attention from the people who have something to say to the people who decide whether or not to pay attention. Listening involves goodwill and perhaps a form of power. It wasn’t so serious. I always say that there hasn’t been a big solution, but more playful paths. It reminded me of the early web where you could assemble your website using copy/paste, and here you could copy/paste little weird interactions.

Moving through Stories

Esther 24:18 There is one question from the chat: what about non-real-time and non-real spacespace?

Carolina 24:39 I guess this question is about imagination, speculation, and fiction. Hybridity can be a mix of real and fictional. In the Living Archives group, we tried to get people to imagine ‘how it would be like’. When we think about the different temporalities and especially the afterlife, there’s a lot you can add to the experience that happened during the program. For instance, this particular publication we’re experiencing now will then become a websiteweb and textprint publication. We can add things to it. We can tell a different story. Different mediums can add different elements.

Esther 25:36 We tend to think of events as limited in time and certain. But in living archives and growing publications, we’re continuing to develop storylines or stories. The event is not the end point of a story, it’s one of the elements within a story development that could continue further with a audiopodcast or with online comments afterward… seppi At INC, we’re experimenting with an audio-visual podcast & pop-up event format called The Void. I think it’s very interesting to explore these hybridities: https://networkcultures.org/void. Indeed, people can engage more on their own terms - whether that’s on the bike, while cooking, or in high concentration

Lilian 26:15 Two months ago we organized a data center tour with The Hmm. You can find the program of the Data Center Tour here: https://thehmm.nl/event/the-hmms-data-centre-tour This one was a five-hour trip, and we didn’t like the idea of such a long livestream. We instead invited podcast makers to be there and record all the speakers They also spoke with some visitors and recorded audiosounds of the data center and combined it all into a podcast episode. This different format really added value, that could not have been created if it was just a videolivestream, because it was also adding the layer of the participation of the visitors and the space with the sounds of the spacedata center. We have to remember that a livestream recording is not really recording the event… This audio report, because it also included the audience and also included the space, was a way better translation of the event. Listen back to the Data Center Tour here: https://live.thehmm.nl/the-hmm-s-data-centre-tour

Esther 27:38 Last question from the audience. When is a hybrid experience successful or good?

Lilian 27:53 Good question. I would say when there’s interaction, especially interaction between online and on-site audiences. During Going Hybrid, Lilian wrote about the interaction between on-site and online audiences in a blog-post called ‘Participatory Livecasting’: https://networkcultures.org/goinghybrid/2022/11/17/participatory-livecasting

Carolina 28:18 I would also say, winking to the next chapter, when it is accessible to both audiences and spaces. student at the back Here the interaction was made: they took my comment on non-real-time and non-real space seriously (I am not confident on the existence of these non-real realities) Klara Debeljak Success!

Esther 28:30 A successful hybrid experience includes different times, different spaces, different people. On that note, we’re wrapping up this chapter. Thank you very much.

About this text

Jordi Viader Guerrero (MX/ES) is a practice-based researcher in philosophy of technology and media, currently doing a PhD at TU Delft on the reimagining of algorithmic practices for democratic engagement. His research and practice are chiefly focused on using design and theory to articulate digital technologies within wider cultural, political, and epistemic logics.

Esther Hammelburg is a senior lecturer at the Amsterdam University of Applied Sciences and holds a PhD from the University of Amsterdam. She researched ‘liveness’: a historically evolving practice of establishing instances of ‘now here together’ through media practices which align physical and mediated environments.

Carolina Valente Pinto is a researcher at the Institute of Network Cultures and team member of The Hmm. She was the project coordinator of Going Hybrid, co-leader of the Living Archives group, and co-editor of Screentime Airtime Facetime.

Lilian Stolk is the director of The Hmm, a platform for internet culture, and an emoji expert. She was a participant of the Participatory Livecasting group of Going Hybrid.

Heerko van der Kooij is a creative technologist working with Hackers & Designers. He was also a member of the Participatory Livecasting group.

This chapter-conversation was originally recorded during the a live event Screentime Airtime Facetime, which you can watch back here: https://vimeo.com/908212301.