Revision as of 11:38, 8 September 2019

DART-MX8M Streaming Video

Introduction to GStreamer

GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.

Major GStreamer commands

There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.

gst-inspect

Lists all available elements:

# gst-inspect-1.0

Lists all available sink elements:

# gst-inspect-1.0 | grep sink

For example:

# gst-inspect-1.0 | grep sink
alsa:  alsasink: Audio sink (ALSA)
waylandsink:  waylandsink: wayland video sink
shm:  shmsink: Shared Memory Sink
decklink:  decklinkaudiosink: Decklink Audio Sink
decklink:  decklinkvideosink: Decklink Video Sink
ximagesink:  ximagesink: Video sink
soup:  souphttpclientsink: HTTP client sink
autodetect:  autovideosink: Auto video sink
autodetect:  autoaudiosink: Auto audio sink
gdkpixbuf:  gdkpixbufsink: GdkPixbuf sink
playback:  playsink: Player Sink
video4linux2:  v4l2sink: Video (video4linux2) Sink
fbdevsink:  fbdevsink: fbdev video sink
tcp:  tcpclientsink: TCP client sink
tcp:  tcpserversink: TCP server sink
…

Detailed help information on a specific element:

# gst-inspect-1.0 <name of the element>

For example, the output for waylandsink:

# gst-inspect-1.0 waylandsink
Factory Details:
  Rank                     primary + 2 (258)
  Long-name                wayland video sink
  Klass                    Sink/Video
  Description              Output to wayland surface
  Author                   Sreerenj Balachandran <sreerenj.balachandran@intel.com>, George Kiagiadakis <george.kiagiadakis@collabora.com>

Plugin Details:
  Name                     waylandsink
  Description              Wayland Video Sink
  Filename                 /usr/lib/gstreamer-1.0/libgstwaylandsink.so
  Version                  1.14.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2018-03-19
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSink
                         +----GstVideoSink
                               +----GstWaylandSink

Implemented Interfaces:
  GstVideoOverlay
  GstWaylandVideo

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                           (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:DMABuf)
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                           (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "waylandsink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  sync                : Sync on the clock
                        flags: readable, writable
                        Boolean. Default: true
  max-lateness        : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
                        flags: readable, writable
                        Integer64. Range: -1 - 9223372036854775807 Default: 20000000 
  qos                 : Generate Quality-of-Service events upstream
                        flags: readable, writable
                        Boolean. Default: true
  async               : Go asynchronously to PAUSED
                        flags: readable, writable
                        Boolean. Default: true
  ts-offset           : Timestamp offset in nanoseconds
                        flags: readable, writable
                        Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0 
  enable-last-sample  : Enable the last-sample property
                        flags: readable, writable
                        Boolean. Default: true
  last-sample         : The last sample received in the sink
                        flags: readable
                        Boxed pointer of type "GstSample"
  blocksize           : Size in bytes to pull per buffer (0 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  render-delay        : Additional render delay of the sink in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  max-bitrate         : The maximum bits per second to render (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  show-preroll-frame  : Whether to render video frames during preroll
                        flags: readable, writable
                        Boolean. Default: true
  window-width        : Wayland sink preferred window width in pixel
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  window-height       : Wayland sink preferred window height in pixel
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  display             : Wayland display name to connect to, if not supplied via the GstContext
                        flags: readable, writable
                        String. Default: null
  alpha               : Wayland surface alpha value, apply custom alpha value to wayland surface
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  enable-tile         : When enabled, the sink propose VSI tile modifier to VPU
                        flags: readable, writable
                        Boolean. Default: false

The i.MX8M VPU video decoder element

# gst-inspect-1.0 vpudec
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                VPU-based video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode compressed video to raw data
  Author                   Multimedia Team <shmmmw@freescale.com>

Plugin Details:
  Name                     vpu.imx
  Description              VPU video codec
  Filename                 /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.3.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package           Freescle Gstreamer Multimedia Plugins
  Origin URL               http://www.freescale.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstVpuDec

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
      video/x-vp9
      video/x-vp8
      video/x-vp6-flash
      video/x-h264
      video/mpeg
           systemstream: false
            mpegversion: { (int)1, (int)2 }
      video/mpeg
            mpegversion: 4
      video/x-h263
      video/x-flash-video
             flvversion: 1
      video/x-divx
            divxversion: 3
      video/x-divx
            divxversion: 4
      video/x-divx
            divxversion: { (int)5, (int)6 }
      video/x-xvid
      video/x-cavs
      video/x-wmv
             wmvversion: 3
                 format: WMV3
      video/x-wmv
             wmvversion: 3
                 format: WVC1
      video/x-pn-realvideo
      image/jpeg
      image/webp

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_video_decoder_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpudec0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  output-format       : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
                        flags: readable, writable
                        Enum "GstVpuDecOutputFormat" Default: 0, "auto"
                           (0): auto             - enable chroma interleave. (default)
                           (1): NV12             - NV12 format
                           (2): I420             - I420 format
                           (3): YV12             - YV12 format
                           (4): Y42B             - Y42B format
                           (5): NV16             - NV16 format
                           (6): Y444             - Y444 format
                           (7): NV24             - NV24 format
  frame-drop          : enable adaptive frame drop for smoothly playback
                        flags: readable, writable
                        Boolean. Default: true
  frame-plus          : set number of addtional frames for smoothly playback
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 16 Default: 3 
  use-vpu-memory      : use vpu allocate video frame buffer
                        flags: readable, writable
                        Boolean. Default: true

Test Pattern Stream

Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:

# gst-launch-1.0 videotestsrc ! autovideosink

Test pattern with specific parameters on the input element:

# gst-launch-1.0 videotestsrc pattern=circular ! autovideosink

Camera Loopback

Stream video from a camera to a display:

# gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink

Video Recording

Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file:

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi

Video Playback

The i.MX8M can decode and playback HEVC(H.265), VP9, etc. encoded videos with HDR at resolutions of up to 4Kp60 UHD (3840×2160p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M VPU decoder.

Example of a video file playback using the playbin element (making use of the i.MX8M VPU decoder):

# gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4

Example of a video file playback while explicitly specifying a pipeline (making use of the i.MX8M VPU decoder):

# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink

RTP Video streaming

Encode video stream from camera using a software based JPEG encoder and stream it over RTP:

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223

References

The GStreamer website
NXP BSP Linux Users Guide, Multimedia section download link

@@ Line 2: / Line 2: @@
 --> {{#vardefine:RELEASE_PARAM|{{#urlget:release}}}} <!--
 --> {{#lst:Yocto_Platform_Customization|{{#var:RELEASE_PARAM|RELEASE_SUMO_V1.0_DART-MX8M}}}} <!--
---> {{PageHeader|{{#var:HARDWARE_NAME}} Streaming Video}} {{DocImage|category1=Yocto|category2=DART-MX8M}}[[Category:DART-MX8M-MINI]] __toc__
+--> {{PageHeader|{{#var:HARDWARE_NAME}} Streaming Video}} {{DocImage|category1=Yocto|category2={{#var:HARDWARE_NAME}}}} __toc__
 = Introduction to GStreamer =

DART-MX8M GSTREAMER: Difference between revisions