Send audio outside of Max

[valiafetisov]

Hi Tomoya! Thank you for your work!

Problem

I couldn't make this external work with pulseaudio RTP receiver and found no information online, although the problem seems quite common.

More specifically, I tried to use mc.rtpsend~ to send audio to a Raspberry PI on the same network as Max. I expect that pulseaudio module-rtp-recv should be able to receive the stream and sink it to the speaker.

I can't find alternatives, but my experience with audio streaming is very limited.

Potential solution

It will be amazing if you can provide a documentation of what exactly happening under the hood. Or, event better, if you would give an example of receiving audio using pulseaudio RTP, Roc or anything else that would work on raspberry/any other linux machine.

[valiafetisov]

More debugging information:

If I use max object

mc.rtpsend~ @address 127.0.0.1 @port 30000 @use_rtsp 0 @channels 2 @active 1

I'm able to receive the data via

ffplay -i rtp://127.0.0.1:30000

But it doesn't play it as audio, but displays it (which looks like spectrogram to me):

Full output:

% ffplay -i rtp://127.0.0.1:30000
ffplay version 4.4 Copyright (c) 2003-2021 the FFmpeg developers
  built with Apple clang version 12.0.5 (clang-1205.0.22.9)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/4.4_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[rtp @ 0x131816400] Guessing on RTP content - if not received properly you need an SDP file describing it
Input #0, rtp, from 'rtp://127.0.0.1:30000':
  Duration: N/A, start: 0.000000, bitrate: 1411 kb/s
  Stream #0:0: Audio: pcm_s16be, 44100 Hz, 2 channels, s16, 1411 kb/s
  10.47 M-A: -0.000 fd=   0 aq=    0KB vq=    0KB sq=    0B f=0/0   

Do you know what am I doing wrong?
Do you maybe have correct SDP file what would help playing it?

[tomoyanonymous]

Hi Valia, thanks for reporting this.

（In the first place, the attribute @use_rtsp 0 is kept for backward compatibility and not well maintained: it's basically unstable.)

You cannot simply receive the stream by accessing rtp://127.0.0.1:30000 because stream parameters like SR and a number of channels are not provided from the sender in this option(RTSP protocol provides them in the default option).

However, you can use it by creating SDP file manually, as the receiver doing internally.

rtpsendreceive/source/rtpsendreceive_lib/rtpsrbase.cpp

Lines 99 to 120 in 3cd8232

	std::string RtpInOption::makeDummySdp() {
	std::string sdp_content =
	R"(SDP:
	v=0
	o=- 0 0 IN IP4 $address$
	s=DUMMY SDP
	c=IN IP4 $address$
	t=0 0
	a=tool:libavformat 58.29.100
	m=audio $port$ RTP/AVP 97
	b=AS:$bitrate$
	a=rtpmap:97 L16/$samplerate$/$channels$)";
	std::vector<std::pair<std::string, std::string>> replace_pairs = {{"\\$port\\$", std::to_string(url.port)},
	{"\\$address\\$", url.address},
	{"\\$channels\\$", std::to_string(channels)},
	{"\\$bitrate\\$", std::to_string((samplerate / 1000) * 16 * channels)},
	{"\\$samplerate\\$", std::to_string(samplerate)}};
	std::for_each(replace_pairs.begin(), replace_pairs.end(), [&](auto pair) {
	sdp_content = std::regex_replace(sdp_content, std::regex(pair.first), pair.second);
	});
	return sdp_content;
	}

The receiver creates a dummy sdp file by replacing the parameters closed with $$. For example, with your setting, SDP file will be like below.

SDP:  
v=0
o=- 0 0 IN IP4 127.0.0.1
s=DUMMY SDP  
c=IN IP4 127.0.0.1 t=0 0  
a=tool:libavformat 58.29.100  
m=audio 30000 RTP/AVP 97  
b=AS:1411.2
a=rtpmap:97 L16/44100/2

(Note that bitrate will be (samplerate / 1000) * 16 * channels)

After saving this file as live.sdp, you will be able to receive the stream on RasPi with

ffplay -re -i path/to/live.sdp

Again raw rtp mode is quite unstable, if you want to just stream from Max on macOS or Window to RaspberryPi, using NetJack might be an easier option.

https://github.com/jackaudio/jackaudio.github.com/wiki/WalkThrough_User_NetJack2

[tomoyanonymous]

Or, If you want to just receive the stream and play it back to a speaker on RasPi: use RTSP!

change the attributes of the object to

mc.rtpsend~ @port 30000 @use_rtsp 1 @channels 2

and receive on raspi with

ffplay -rtsp_flags listen rtsp://127.0.0.1:30000

(Of course, you need to specify the correct IP address when streaming between 2 computers)

[valiafetisov]

Thanks! I was able to connect to the rtsp stream using the exact commands you've provided in the last message, but I still can't hear the sound, not sure why. I can see similar image as in my comment above but can't hear the sound. Normal ffplay path/to/a/file.wav works well (both playing audio and displaying spectrogram as expected), so it's not the issue with the output device. It's also not a network problem, as I am sending the data only locally right now.

Do you have a clue of what may be the problem?

By the way, rtpsendreceive/test_patchs also do not work together with the externals from Release v0.3.0. So at this point I can't confirm that the externals works for me. Let me know if you need some debugging information.

The other option I consider is to use sonobus but it will require installing another program and will complicate autostart.

[tomoyanonymous]

Which OS are you working on now for what version of Max? (Currently, I have not tested on Big Sur.)

Also, check your signal vector size on Audio Settings. The buffer size of each packet currently depends on Max's audio setting. In my environment(Catalina, Macbook Pro 16inch 2019), a smaller size than 128 produces an audio glitch shortly later its start and finally sound stops. 512~ is recommended.

The test patch is... I usually use self_loop_test.maxpat.
The basic usage is

1. Turn on metro on left-center
2. set a number of channels by setting numbox on right-top
3. turn on dsp
   it sends a signal for a single channel which is periodically switched.
   If this test patch does not work, it also will not work with ffmpeg...

[valiafetisov]

Which OS are you working on now for what version of Max?

I've tested on both 11.1 M1 (with rosetta) and 10.15 intel. Both with latest Max 8. I've just tested again, following your instruction, but it didn't work on both systems. Interestingly, ffplay is able to receive and display rtsp stream, but not play it, so it is surely sending the audio, but something else is off. Let me know if you need some specific information to debug it.

[tomoyanonymous]

Hmm... Could you check actual packets sent/received with Wireshark or something?
RTSP uses 2 or more network ports when you specified 30000 in Max: 30000 to RTSP handshake and other ports used for actual streaming depending on the situation decided on the handshake.

You can inspect actual RTSP request/response on ffplay by using -v trace option.
The response contains something like below:

[rtsp @ 0x7fb70280de00] WARNING: Path /live.sdp differs from expected
[rtsp @ 0x7fb70280de00] Parsing[7]: CSeq: 1
[rtsp @ 0x7fb70280de00] Parsing[25]: User-Agent: Lavf58.45.100
[rtsp @ 0x7fb70280de00] Sending response:
RTSP/1.0 200 OK
CSeq: 1
Server: Lavf58.45.100
Public: ANNOUNCE, PAUSE, SETUP, TEARDOWN, RECORD

[rtsp @ 0x7fb70280de00] WARNING: Path /live.sdp differs from expected
[rtsp @ 0x7fb70280de00] Updating control URI to udp://127.0.0.1:30000/live.sdp
[rtsp @ 0x7fb70280de00] Parsing[29]: Content-Type: application/sdp
[rtsp @ 0x7fb70280de00] Parsing[7]: CSeq: 2
[rtsp @ 0x7fb70280de00] Parsing[25]: User-Agent: Lavf58.45.100
[rtsp @ 0x7fb70280de00] Parsing[19]: Content-Length: 178
[rtsp @ 0x7fb70280de00] SDP: v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 58.45.100
m=audio 0 RTP/AVP 96
b=AS:768
a=rtpmap:96 L16/48000/1
a=control:streamid=0

[rtsp @ 0x7fb70280de00] sdp: v='0'
[rtsp @ 0x7fb70280de00] sdp: o='- 0 0 IN IP4 127.0.0.1'
[rtsp @ 0x7fb70280de00] sdp: s='No Name'
[rtsp @ 0x7fb70280de00] sdp: c='IN IP4 127.0.0.1'
[rtsp @ 0x7fb70280de00] sdp: t='0 0'
[rtsp @ 0x7fb70280de00] sdp: a='tool:libavformat 58.45.100'
[rtsp @ 0x7fb70280de00] sdp: m='audio 0 RTP/AVP 96'
[rtsp @ 0x7fb70280de00] sdp: b='AS:768'
[rtsp @ 0x7fb70280de00] sdp: a='rtpmap:96 L16/48000/1'
[rtsp @ 0x7fb70280de00] audio codec set to: pcm_s16be
[rtsp @ 0x7fb70280de00] audio samplerate set to: 48000
[rtsp @ 0x7fb70280de00] audio channels set to: 1
[rtsp @ 0x7fb70280de00] sdp: a='control:streamid=0'
[rtsp @ 0x7fb70280de00] Sending response:
RTSP/1.0 200 OK
CSeq: 2
Server: Lavf58.45.100

[rtsp @ 0x7fb70280de00] Parsing[66]: Transport: RTP/AVP/UDP;unicast;client_port=13840-13841;mode=record
[rtsp @ 0x7fb70280de00] Parsing[7]: CSeq: 3
[rtsp @ 0x7fb70280de00] Parsing[25]: User-Agent: Lavf58.45.100
[rtsp @ 0x7fb70280de00] Opening: rtp://127.0.0.1:5000No default whitelist set
[udp @ 0x7fb721f375c0] No default whitelist set
[udp @ 0x7fb721f375c0] end receive buffer size reported is 393216
[udp @ 0x7fb721f43dc0] No default whitelist set
[udp @ 0x7fb721f43dc0] end receive buffer size reported is 393216
[rtsp @ 0x7fb70280de00] Listening on: 5000setting jitter buffer size to 500
[rtsp @ 0x7fb70280de00] Sending response:
RTSP/1.0 200 OK
CSeq: 3
Server: Lavf58.45.100
Transport: RTP/AVP/UDP;unicast;mode=receive;source=127.0.0.1;client_port=13840-13841;server_port=5000-5001
Session: 516959051

[rtsp @ 0x7fb70280de00] Parsing[17]: Range: npt=0.000-
Failed to parse interval end specification ''
[rtsp @ 0x7fb70280de00] Parsing[7]: CSeq: 4
[rtsp @ 0x7fb70280de00] Parsing[25]: User-Agent: Lavf58.45.100
[rtsp @ 0x7fb70280de00] Parsing[18]: Session: 516959051
[rtsp @ 0x7fb70280de00] Sending response:
RTSP/1.0 200 OK
CSeq: 4
Server: Lavf58.45.100
Session: 516959051

[rtsp @ 0x7fb70280de00] All info found vq=    0KB sq=    0B f=0/0
[rtsp @ 0x7fb70280de00] stream 0: start_time: -0.023 duration: NOPTS
[rtsp @ 0x7fb70280de00] format: start_time: -0.023 duration: NOPTS (estimate from bit rate) bitrate=768 kb/s
Input #0, rtsp, from 'rtsp://127.0.0.1:30000':
  Metadata:
    title           : No Name
  Duration: N/A, start: -0.023000, bitrate: 768 kb/s
    Stream #0:0, 1, 1/48000: Audio: pcm_s16be, 48000 Hz, 1 channels, s16, 768 kb/s
detected 16 logical cores
[ffplay_abuffer @ 0x7fb721e1a440] Setting 'sample_rate' to value '48000'
[ffplay_abuffer @ 0x7fb721e1a440] Setting 'sample_fmt' to value 's16'
[ffplay_abuffer @ 0x7fb721e1a440] Setting 'channels' to value '1'
[ffplay_abuffer @ 0x7fb721e1a440] Setting 'time_base' to value '1/48000'
[ffplay_abuffer @ 0x7fb721e1a440] tb:1/48000 samplefmt:s16 samplerate:48000 chlayout:(null)
[AVFilterGraph @ 0x7fb721d7d140] query_formats: 2 queried, 3 merged, 0 already done, 0 delayed
Audio frame changed from rate:48000 ch:1 fmt:s16 layout:0 channels serial:-1 to rate:48000 ch:1 fmt:s16 layout:0 channels serial:1
[ffplay_abuffer @ 0x7fb721f460c0] Setting 'sample_rate' to value '48000'
[ffplay_abuffer @ 0x7fb721f460c0] Setting 'sample_fmt' to value 's16'
[ffplay_abuffer @ 0x7fb721f460c0] Setting 'channels' to value '1'
[ffplay_abuffer @ 0x7fb721f460c0] Setting 'time_base' to value '1/48000'
[ffplay_abuffer @ 0x7fb721f460c0] tb:1/48000 samplefmt:s16 samplerate:48000 chlayout:(null)
[AVFilterGraph @ 0x7fb701e31d80] query_formats: 2 queried, 3 merged, 0 already done, 0 delayed

In my case of local loopback, audio was sent from 13840 to 5000 as the log implies in these lines.

[rtsp @ 0x7fb70280de00] Parsing[66]: Transport: RTP/AVP/UDP;unicast;client_port=13840-13841;mode=record

Transport: RTP/AVP/UDP;unicast;mode=receive;source=127.0.0.1;client_port=13840-13841;server_port=5000-5001

The captured packet in WIreshark was like this.

If you can confirm with Wireshark that the packet is correctly sent, the problem will be in the part of playback in ffplay(with an audio driver or something). If the packet was sent but not received, maybe the receiver or network apparatus such as a router or a switching hub is blocking packets or closing the port used for streaming.

Hope it helps to solve your problem.