Bad news first: latency in online music lessons is inevitable.
The good news: Caleb Dolister is a web developer, drummer, composer and writer from Queens, New York. He published the following article titled “Why can’t musicians jam with each other online without latency or other issues?” in April 2020. Because we couldn’t have put it better, we asked Caleb to allow us to publish his text here in our SIRIUS Blog in German. Without much latency he agreed. We then translated it back into English. You will find the link to the original english article at the end of the following text.
“Why can’t musicians jam with each other online without latency or other issues?” based on Caleb Dolister’s article.
It’s 2020, broadband is everywhere. Networks are fast. We can join an MMO (massively multiplayer online) gaming server and support hundreds of players with low latency, but a couple of musicians can’t sync up and jam together?
The idea of getting together online to make music is not new. Developers have been working on it for years, and tech-savvy musicians have been enthusiastically downloading apps or buying and building hardware trying to make it happen. There are some solutions, but latency is inevitable, even with the most professional systems.
It seems that musicians everywhere are looking for answers to the question:
Why doesn’t jamming online work, despite a high-bandwidth connection and today’s technology?
The brain’s fantastic ability to predict movement during latency
Latency has plagued online players since the days of dial-up. I grew up playing online games in a small mountain town in Northern California in the late 90s, and remember feeling like I had a bombproof connection when my community’s dial-up network delivered 250ms.
Why? Honestly, I didn’t know any better. There was no broadband. Rural “gamers” like me learned to play with latency by guessing what other players would do and compensating by analyzing movement. It was less about seeing and reacting and more about seeing and predicting.
Acoustics
In fact, we already deal with latency in music all the time.
Music and sound are transmitted at different rates depending on altitude, temperature, humidity, frequency, and other factors. In general, we can assume that sound travels through the air at about 18 ms per 6 meters.
Think about how far away you usually are from other musicians when performing live. When playing acoustic music on a large stage (without monitors), the latency has to be compensated already at distances up to about 4.5 – 6 meters. Anyone who has played on a large stage without amplification, or in an orchestra or marching band, can confirm that it becomes difficult to play at greater distances.
However, acceptable latency for recording is lower, usually around 10-12 ms, although humans can perceive latency of around 5 ms. The challenge in the studio is that the notes you play will reach your ear more slowly than they would by acoustic means, due to encoding and hardware latency.
Measurement of latency and time in music
For all readers who don’t know how time is calculated in music:
Speed is interpreted as the number of beats per minute (bpm), which is called tempo. 60 bpm = 1 beat per second, 120 bpm = 2 beats per second, and so on. If the tempo of a song is 120 bpm, this corresponds to 500 ms between beats (1 second = 1000 ms, 0.5 seconds = 500 ms). At a distance of 6 meters, there is a natural latency of 18ms, so 500ms/120bpm feels like 518ms/115bpm.
In layman’s terms, it feels like the other player is playing slower, even if they are not. To compensate for this natural latency, larger bands are led by conductors so that there is a visual representation of the time. Natural latency can be managed in a given condition because the musician can compensate for it if it is even. In a larger setting with multiple players and latencies, this becomes impossible, especially if those latencies change.
Multiple latencies
Consider a marching band with well over 200 musicians spread across an entire American football field. The players on the edge of the field may be separated by great distances, while some are only a few feet apart, but the band is always moving. At any given time, these players hear the person next to them at about 5 ms, the person 3 meters away at about 9 ms, those 6 meters away at 18 ms, and finally the players 60 meters away at 180 ms. Without a conductor, this is a disaster.
When latency is a challenge, dealing with multiple changing latencies — becomes a crisis.
Online jamming is, unfortunately, just like being in a marching band in different parts of the field.
Each player has different latency. Depending on where they are in relation to the shared server. That latency changes due to network congestion.
The Deal Breaker:
Traffic & Variations
For musicians who are unfamiliar with network pings and jitter, here’s a little experiment: do a ping test on
8.8.8.8. This IP address is the primary DNS server for Google DNS and one of the most consistent servers on the Internet.
You will notice that the results can fluctuate with each packet sent. Sometimes the time can vary by +5ms… -10ms… -1ms… +7ms…, or the packet is dropped altogether. It is rarely the same.
This is caused by network traffic and is a bigger problem in a music environment because every online session has these fluctuations. Musical timing is not only delayed, but also in flux. There are no solutions to predict and balance network traffic.
We can learn to deal with latency in the range of 20-30 ms, maybe even 50 ms or more if it is absolutely reliable. However, public networks have a lot of traffic, so even the best connections fluctuate, packets get dropped, delayed, reset, etc. Musicians are sensitive enough to feel a 5 ms change, so these latency fluctuations are annoying, especially because they are impossible to predict.
I have seen musicians playing duo videos, even with latency.
I would guess that they’re not really playing written music, and it looks like it would work best with rhythm giver on one side and a solo on the other. It seems to work, but only until it’s time to play parts together. The addition of a third voice or instrument is where it finally fails.
And for those who think 5G connectivity is just around the corner — even though these can theoretically support consistent transfer speeds of 1-5ms, networks will still be affected by fluctuations caused by network traffic. And you still have your local environment and home network latency.
Most audio platforms generate latency through digital audio encoding. Video platforms need to encode both audio and video. And if you don’t have a wired connection to the Internet, there’s a lot of latency between your audio source and your WiFi router.
But: can it work even with the most “ideal” setup, a direct fiber connection between two hosts?
Conclusion
According to mathematician and physicist Philippe Kahn, there is still one major problem that prevents musicians from achieving a real-time experience:
Einstein’s theory of relativity, according to which nothing can travel faster than the speed of light.
In addition to mathematics, one of Philippe Kahn’s many passions is his lifelong interest in classical music and jazz.
“No matter how efficient the network and equipment, latency is inevitable. Therefore, the problem of real-time remote transmission of music boils down to the question, “What latency is acceptable?” In my personal opinion, 10 ms is the minimum for all styles of music. The less, the better. But there will always be some latency. You can’t beat Einstein and the laws of physics, except in science fiction books where we travel through time, which is very funny!”
Philippe Kahn
The speed of light has a latency of about 5 ms over a distance of 1500 km. A network connection cannot be faster than this time under perfect conditions.
LoLa
In one case, I found out about LoLa. This seems to be the best solution to allow musicians to play remotely. Unfortunately, it is not a home user solution and requires very specialized hardware and a high performance 1GB clear path connection between sites. The hardware helps reduce latency, and the clear-path data connection helps avoid spikes caused by network traffic. The project is installed in many places thanks to universities and research labs around the world, allowing musicians to rehearse and in some cases perform.
Jack Trip
Another promising option (audio only) is JackTrip.
Thanks to a mention by NPR, JackTrip got my attention and seems to work best over shorter distances, 800 km maximum. Jacktrip shortens the time it takes to encode your audio signal into packets that can be sent over the Internet. Significantly!
Unfortunately, most of us don’t have perfect network connections. We struggle with network traffic and have to try to balance the fluctuations between multiple musicians trying to connect with different latencies. Achieving a real-time feel online won’t work for music, although our brains can learn to compensate and predict.
But that doesn’t mean we can’t go online and have some fun.
Find out more about Caleb und latency per se
