1 of 23

Exploring Latency, Jitter, and Packet Loss as Audiovisual Aesthetics in a Gesture-Enabled, Browser-Based Networked Live Coding Platform

Quargs Greene

2 of 23

The name “Trussal” originates from a combination of the words “truss” and “algorave”.

3 of 23

Overview

  • Real-time performance within a video conference
  • Participants play one at a time and can spawn bots
  • Effects, scheduling influenced by network conditions
  • Everyone’s media goes through a centralized scheduler

4 of 23

Background

  • The Hub is regarded as pioneering networked music ensemble
  • Further investigation of network as aesthetic with Nrev, Global Drum Circle, Global Net Orchestra
  • Significant body of research in telematic dance
  • Collaborative live coding using CRDT in Flok, Gibber
  • Very little live coding accessibility research, no one has combined fake time approach, live coding

5 of 23

Why Trussal?

  • Circumventing the social construct of charisma and stage presence
  • Developing online electroacoustic performance practices as a medium
  • Exploring customizable avenues for accessibility
  • Bringing algorave online
  • Connecting open source infrastructure

6 of 23

7 of 23

Features

  • Room effects include, reverb, noise, and more
  • Live coding with your face
  • Create bots according to your code (without an LLM!)
  • Live code text, meeting reactions, audio, video

8 of 23

A little tour

9 of 23

Text live coding

10 of 23

Meeting reaction live coding

11 of 23

Gesture live coding

12 of 23

Hands-free live coding

13 of 23

Room effects

14 of 23

Adjusting network-based modulation

15 of 23

Photo credit: Quargs Greene

16 of 23

Current Implementation

  • Microservices
  • Jitsi Meet
  • MediaPipe
  • Web Audio API

17 of 23

Current Implementation

​

  • CRDT, Y.js
  • JPattern, a domain-specific pattern language
  • XMPP, O2lite, HTTPS
  • Consistent hashing

18 of 23

Testing & Evaluation

  • WAVE extension
  • Focus groups
  • Large ensemble simulation

19 of 23

Video feed processing

Photo credit: Jennifer Deafenbaugh

20 of 23

Limitations & Challenges

  • Jitsi Meet optimizes speech over music
  • Figuring out what makes using hands-optional technologies easier
  • Unforeseen Strudel issue
  • Optimization
  • No database

21 of 23

Limitations & Challenges

  • JPattern grammar
  • Jitsi Meet Docker image
  • Downtime and ISP problems
  • Cumbersome for artists with no coding experience
  • Lack of authentication

22 of 23

Punchcards gone wild

Photo credit: Quargs Greene

Nine meeting participants

Photo credit: Quargs Greene

Trussal x Chromebook Studies by Will Rinkoff

Photo credit: Quargs Greene

23 of 23

Scan to Find Me Elsewhere

Los Angeles

Projector Swarm

Quargs

Trussal GitHub Repo