What’s new in Membrane Framework - Q2/Q3 2019

Our bad! We missed the original date for Q2 update, but we have a legitimate excuse. We have been extremely busy bringing feature-rich v0.4 to you. Lots of other things were happening in the meantime, but our major goal was to deliver quality code to our community. And this time, we worked on a big feature for Membrane Core, wich broadens the usage spectrum of the whole Membrane ecosystem.

Here are some updates about what happened in the last half year at Membrane:

Membrane Core v0.4 – a major leap!

Mid September, we released v0.4 of Membrane Core.

This version includes – but isn’t limited to – the feature some of you have long been waiting for: A/V stream synchronization utilities. This breakthrough functionality took hours of development and testing and all our efforts got focused on delivering it. Therefore we did not have much capacity for other stuff, including this blog. But here it finally is. And we managed to squeeze in an improved testing API too!

Synchronization

Version 0.4 introduces three new utilities: stream synchronization, clocks, and timers. This is particularly interesting from a professional broadcaster perspective since perfect audio/video stream sync is critical for live transmissions, long continuous programming, using different sources and sinks.

Stream Synchronization

Membrane now implements sync mechanism that ensures multiple elements starting their streams at the same time. What’s important, you do not have to modify your existing elements for the mechanism to work. Just make sure you include those elements in Membrane.ParentSpec and the StartOfStream events will be sent to them at the same moment.

Additionally, you can now configure latency of an element. A certain latency value would make the element receive start of stream respectively earlier compared to elements with no latency. This is particularly useful when trying to utilize streams from different sources, e.g. Bluetooth audio (significant delay) with video (low delay). In such cases, you can set higher latency for the Bluetooth source and low or no latency for video player. The rest will happen automatically, Membrane will take care of the timing!

More info about Synchronization can be found here

Clocks

Now, elements (most likely sinks) can be clock providers for pipelines. E.g. a sink sending audio to a sound card can export a clock based on the device’s hardware clock. It would then send an update message to the clock e.g. based on audio samples consumed. The clock would then automatically calculate the ratio between its rate and the rate of the system clock (i.e. VM clock). Other elements of the pipeline that produce audio samples for it can then use this information to adjust their rate, thus preventing buffer underflow/overflow.

Moreover, if there’s just one clock provider in the pipeline, it will be used by the pipeline automatically. Otherwise, it’ configurable via :clock_provider atom within Membrane.Pipeline.Spec struct.

Timers

In order to use a pipeline clock, an element needs a timer to be set up. For each timer, a clock pid can be defined or parent pipeline clock can be used by default. Timer will receive updates from the given clock and adjust its rate of generating ticks that are received by the element and then handled by handle_tick callback.


Note: To learn about the new utilities and their details, see code snippets and examples, please refer to Membrane Guide.


New, Improved Testing API

Another significant change in Membrane Core v0.4 is the testing API. We can now proudly say that testing your Membrane applications has now become more accurate and easier than ever.

Testing Pipeline

Membrane.Testing.Pipeline is now available to reduce boilerplate and ease communication with its elements. It also provides a utility for informing the testing process about playback state changes and received notifications. Once the list of elements of the testing pipeline is provided, it generates the links between them automatically, which further speeds up the process. Furthermore, you can pass a custom testing pipeline if you like.

Testing Source

Membrane.Testing.Source is an element that can output buffers from a data source passed as an option. It is especially useful when testing elements that consume network packets, e.g. RTP parser. It can work with either enumerable data sources or generator functions.

Testing Sink

Membrane.Testing.Sink will notify the pipeline about buffers, caps and events it receives and by default make demands automatically. Alternatively, you can set an appropriate option and the Testing Sink will no longer make demands automatically, but will let you trigger demands by sending it a {:make_demand, size} message instead.

Assertions

To make element integration test possible we introduced two groups of assertions. One working with Testing Pipeline, and the other dedicated for Testing Sinks.


Note: Please refer to Membrane Guide or our Github site for more detailed information about testing utilities in Membrane.


Deprecations

Deprecation of Start and End of stream events in favor of custom callback

%Membrane.Event.StartOfStream{} and %Membrane.Event.EndOfStream{} will no longer be handled by handle_event/4 callback. They will now be handled in their respective dedicated callbacks.


Note: Please refer to our Github site for more detailed information.


Events and Media

In August, our very own Mateusz Front did a talk about Membrane during ElixirConf 2019 in Aurora, CO. If you’re new to Membrane and our community, you might wish to have a peek and maybe learn something you didn’t know so far. Power users can also use the video to introduce Membrane to others.

New Packages

Since our last update, the following packages have been published:

  • FDK-AAC Element – provides elements that can be used for decoding/encoding AAC audio.

  • UDP Element – provides elements that can be used to read from and write to UDP sockets.

  • RTP Format Definition – provides RTP packet format definition (so-called caps) for the Membrane Multimedia Framework.

  • RTP H264 Element – provides elements that can be used for depayloading H.264 video.

  • RTP JitterBuffer Element – provides elements that can be used for buffering incoming packets to reduce network jitter.

  • RTP Element – provides elements that can be used to parse RTP packets.

  • RTP MPEGAudio Elements – provides elements that can be used to depayload MPEG audio RTP packets.

  • FLAC audio format definition – provides FLAC audio format definition (so-called caps) for the Membrane Multimedia Framework.

  • SDL Element – provides an SDL-based video player.

  • FLAC Parser Element – provides an element parsing FLAC encoded audio stream. More info can be found in the docs for element module.

  • IBM SpeechToText Element – provides a Sink wrapping IBM Speech To Text Streaming API client. Currently supports only audio streams in FLAC format.

  • Element Tee – provides elements that can be used to branch stream processing in pipeline, e.g. send data from one source to two or more sinks.

  • Google Cloud SpeechToText Element – provides a Sink wrapping Google Cloud Speech To Text Streaming API client. Currently supports only audio streams in FLAC format.


Note: For full list of packages (sorted by release date), please check https://hex.pm/packages?search=membrane&sort=inserted_at.


That’s it for now. Expect another update soon!