DART-MX8M GSTREAMER
From Variscite Wiki
DART-MX8M-MINI Streaming Video
Introduction to GStreamer
GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Major GStreamer commands
There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect
Lists all available elements:
# gst-inspect-1.0
Lists all available sink elements:
# gst-inspect-1.0 | grep sink
For example:
# gst-inspect-1.0 | grep sink alsa: alsasink: Audio sink (ALSA) waylandsink: waylandsink: wayland video sink shm: shmsink: Shared Memory Sink decklink: decklinkaudiosink: Decklink Audio Sink decklink: decklinkvideosink: Decklink Video Sink ximagesink: ximagesink: Video sink soup: souphttpclientsink: HTTP client sink autodetect: autovideosink: Auto video sink autodetect: autoaudiosink: Auto audio sink gdkpixbuf: gdkpixbufsink: GdkPixbuf sink playback: playsink: Player Sink video4linux2: v4l2sink: Video (video4linux2) Sink fbdevsink: fbdevsink: fbdev video sink tcp: tcpclientsink: TCP client sink tcp: tcpserversink: TCP server sink …
Detailed help information on a specific element:
# gst-inspect-1.0 <name of the element>
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank primary + 2 (258)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <sreerenj.balachandran@intel.com>, George Kiagiadakis <george.kiagiadakis@collabora.com>
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.14.0
License LGPL
Source module gst-plugins-bad
Source release date 2018-03-19
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
GstWaylandVideo
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
(string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
(string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
(string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
(string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
(string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
(string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 20000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
window-width : Wayland sink preferred window width in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
window-height : Wayland sink preferred window height in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
display : Wayland display name to connect to, if not supplied via the GstContext
flags: readable, writable
String. Default: null
alpha : Wayland surface alpha value, apply custom alpha value to wayland surface
flags: readable, writable
Float. Range: 0 - 1 Default: 0
enable-tile : When enabled, the sink propose VSI tile modifier to VPU
flags: readable, writable
Boolean. Default: false
The i.MX8M-MINI VPU video decoder element
# gst-inspect-1.0 vpudec
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based video decoder
Klass Codec/Decoder/Video
Description Decode compressed video to raw data
Author Multimedia Team <shmmmw@freescale.com>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.4.4
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoDecoder
+----GstVpuDec
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h265
video/x-vp9
video/x-vp8
video/x-h264
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "vpudec0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
output-format : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
flags: readable, writable
Enum "GstVpuDecOutputFormat" Default: 0, "auto"
(0): auto - enable chroma interleave. (default)
(1): NV12 - NV12 format
(2): I420 - I420 format
(3): YV12 - YV12 format
(4): Y42B - Y42B format
(5): NV16 - NV16 format
(6): Y444 - Y444 format
(7): NV24 - NV24 format
frame-drop : enable adaptive frame drop for smoothly playback
flags: readable, writable
Boolean. Default: true
frame-plus : set number of addtional frames for smoothly playback
flags: readable, writable
Unsigned Integer. Range: 0 - 16 Default: 3
use-vpu-memory : use vpu allocate video frame buffer
flags: readable, writable
Boolean. Default: true
The i.MX8M-MINI VPU H.264 video encoder element
# gst-inspect-1.0 vpuenc_h264
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based AVC/H264 video encoder
Klass Codec/Encoder/Video
Description Encode raw data to compressed video
Author Multimedia Team <shmmmw@freescale.com>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.4.4
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----vpuenc_h264
Implemented Interfaces:
GstPreset
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
stream-format: { (string)avc, (string)byte-stream }
alignment: { (string)au, (string)nal }
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16,
(string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
width: [ 64, 1920, 8 ]
height: [ 64, 1088, 8 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "vpuenc_h264-0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
bitrate : set bit rate in kbps (0 for automatic)
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 0
gop-size : How many frames a group-of-picture shall contain
flags: readable, writable
Unsigned Integer. Range: 0 - 32767 Default: 30
quant : set quant value: H.264(0-51) (-1 for automatic)
flags: readable, writable
Integer. Range: -1 - 51 Default: -1
Test Pattern Stream
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:
- gst-launch-1.0 videotestsrc ! autovideosink
Test pattern with specific parameters on the input element:
# gst-launch-1.0 videotestsrc pattern=circular ! autovideosink
Camera Loopback
Stream video from a camera to a display:
# gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink
Video Recording
The i.MX8M-MINI can encode videos to AVC(H.264) & VP8 at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the src pad capabilities in the vpuenc_264 and vpuenc_vp8 elements details for the complete list of features supported by the i.MX8M-MINI VPU encoder.
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi
Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file (using the SW based jpegenc element - without using the VPU):
# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi
Video Playback
The i.MX8M-MINI can decode and playback HEVC(H.265), AVC(H.264), VP9, VP8, etc. encoded videos at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M-MINI VPU decoder.
Example of a video file playback using the playbin element (making use of the VPU decoder):
# gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink
RTP Video streaming
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223
References
- The GStreamer website
- NXP BSP Linux Users Guide, Multimedia section download link