What’s new in Membrane Framework - Q1 2019

This April marks a v0.3.0 release of Membrane Framework that we are very happy to announce! On this occasion, we decided to start a new series of blog posts, published quarterly, with a status update on our framework and its ecosystem.

In these posts we will be presenting you:

  • what and why has changed in the Core of the Framework
  • new elements that were created and how they can be useful
  • important changes in tools and libraries supporting the Framework and developed together

Let’s get started with the first edition!

Membrane Core v0.3.0

This release of Membrane Core is relatively small as most of our focus was on new elements and preparations for A/V synchronisation that is planned for the next release. This doesn’t mean there are no interesting changes, though.

Pad options

The biggest addition in this release is pad options. So far, every element could define options, that element’s user has to provide when the element is created. That is not enough for elements with dynamic pads (that can be linked and unlinked at any time). Let’s consider RTSP server: when connecting a new pad, you may want to provide a mount name that can be used for accessing media by RTSP client. It can also be used in an audio mixer: when connecting new audio source you may want to start it with a specified volume level or even completely muted.

A new way to define pads

To prepare for this feature, we slightly changed the way pads are defined. Instead of defining them in groups with def_{in,out}put_pads/1 you should use def_{in,out}put_pad/2 (notice the singular form) for each pad.

So, this:

becomes this:

The old way is now deprecated. This small change results in nicer formatting of more complex definitions and prevents very deep list nesting, especially when defining pad options.

Defining pad options

This is very similar to defining an element’s options – the same keyword list has to be provided under :options key in pad definition:

The values for pad options are passed while linking pads (as a keyword under :pad key). The former PullBuffer (now renamed to InputBuffer) options are passed under :buffer key:

 

Push to pull connection

When linking push output with pull input pad, the ‘toilet’ mode has to be enabled (providing a special buffer for output generated by the pushing pad that can be ‘flushed’ by the input pad). Until this release that had to be done manually, and forgetting about it resulted in a cryptic error. Now it is now enabled automatically with default size constraints, but still can be configured when linking pads:

 

System time – but which system?

When using Membrane.Time.system_time/0 we realized that name is a bit ambiguous as it may refer both to operating system time (obtained by calling System.os_time/0 or :os.system_time/0) and Erlang’s VM time (obtained via System.system_time/0 or :erlang.system_time/0). If you want to know what’s the difference between two and why is it important, see this chapter from Learn You Some Erlang for great good!

We decided that this distinction has to be made clear and therefore Membrane.Time.system_time/0 is now deprecated in favour of Membrane.Time.os_time/0 and Membrane.Time.vm_time/0 that make it clear from which system the time is taken.

Proxy functions for Membrane.Pipeline

When working with pipelines it’s often the case that pipeline’s module is called Some.Long.Prefix.Pipeline so you may want to alias that module. However, that disallows to alias Membrane.Pipeline as well. This resulted in quite bloated calls to start the pipeline:

We wanted to simplify that and as a result, you no longer have to use functions from Membrane.Pipeline module to start/play/stop the pipeline. Now every pipeline (that has use Membrane.Pipeline line) gets automatically generated proxy functions:start/2, start_link/2, play/1, prepare/1, stop/1, stop_and_terminate/1

 

Documentation

While working on generating docs for pad options we realized that they can be improved. Now the description of available options and pads (along with their options) are placed in moduledoc (like in Membrane.Testing.Sink) improving readability

Additionally, the docs for Membrane Core itself have proper module grouping that dramatically improves navigation!

New elements

FDK AAC

One of the new elements is FDK AAC. It’s based on Fraunhofer FDK AAC library and provides support for encoding and decoding audio using Advanced Audio Coding.

UDP Socket

UDP is widely used in multimedia streaming. Recently, we introduced membrane_element_udp package providing 2 elements: Membrane.Element.UDP.Sink which sends buffers through a UDP Socket and Membrane.Element.UDP.Source which pushes buffers received by a UDP Socket.

Real-time Transfer Stack

One of the big additions to the library of Membrane elements is support for Real-time Transport Protocol. That required a whole set of new elements that usually will be linked like this:


UDP Source
|> RTP Parser
|> RTP Jitter Buffer
|> Payload type appropriate depayloader

 

RTP

Membrane.Element.RTP is the cornerstone of whole RTP Stack. It reads the RTP Header forwarding payload down the pipeline. Properties read from the header are stored in metadata.

You can read more about RTP itself in RFC3550.

RTP Jitter Buffer

When transferring media over the network we have no guarantee that packets will be received in the same order they were sent. The goal of the jitter buffer is to output them in the order they were originally transmitted. Our implementation stores buffers in the store that keeps them ordered by Sequence Number read by RTP Parser from RTP Header. When jitter buffer gets full it sends the oldest buffer through :output pad.

The element’s repo can be found here.

RTP Depayloaders – MPEG1/2 Audio and H264 Video

Media that are transported via RTP are wrapped in the payload that contains additional information specific for each format. An element that extracts media from RTP payload is called depayloader. While depayloading MPEG Audio is rather trivial that can’t be said for H264. RFC6184 specifies multiple ways of packetization of H264 NALUs. We added support for MPEG1/2 Audio and H264. Currently, we support only Single Nal Unit Mode and Non-Interleaved packetization modes for H264.

The H264 depayloader is available here, while MPEG1/2 Audio – here.