Personal tools

MX8 GSTREAMER

From Variscite Wiki

Jump to: navigation, search


VAR-SOM-MX8X Streaming Video


1 Introduction to GStreamer

GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.

1.1 Major GStreamer commands

There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.

1.1.1 gst-inspect

Lists all available elements:

# gst-inspect-1.0

Lists all available sink elements:

# gst-inspect-1.0 | grep sink

For example:

# gst-inspect-1.0 | grep sink
pulseaudio:  pulsesink: PulseAudio Audio Sink
hls:  hlssink2: HTTP Live Streaming sink
hls:  hlssink: HTTP Live Streaming sink
bluez:  avdtpsink: Bluetooth AVDTP sink
bluez:  a2dpsink: Bluetooth A2DP sink
coreelements:  filesink: File Sink
coreelements:  fdsink: Filedescriptor Sink
coreelements:  fakesink: Fake Sink
alsa:  alsasink: Audio sink (ALSA)
gdkpixbuf:  gdkpixbufsink: GdkPixbuf sink
…

Detailed help information on a specific element:

# gst-inspect-1.0 <name of the element>

For example, the output for waylandsink:

# gst-inspect-1.0 waylandsink
Factory Details:
  Rank                     primary + 2 (258)
  Long-name                wayland video sink
  Klass                    Sink/Video
  Description              Output to wayland surface
  Author                   Sreerenj Balachandran <sreerenj.balachandran@intel.co

Plugin Details:
  Name                     waylandsink
  Description              Wayland Video Sink
  Filename                 /usr/lib/gstreamer-1.0/libgstwaylandsink.so
  Version                  1.14.4
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2018-10-02
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               Unknown package origin

GObject                                                                         
 +----GInitiallyUnowned                                                         
       +----GstObject                                                           
             +----GstElement                                                    
                   +----GstBaseSink                                   
                         +----GstVideoSink                            
                               +----GstWaylandSink                    
                                                                      
Implemented Interfaces:                                               
  GstVideoOverlay                                               
  GstWaylandVideo                                               

Pad Templates:                                                                  
  SINK template: 'sink'                                                         
    Availability: Always                                                        
    Capabilities:                                                               
      video/x-raw                                                               
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBG
                  width: [ 1, 2147483647 ]                                      
                 height: [ 1, 2147483647 ]                                      
              framerate: [ 0/1, 2147483647/1 ]                                  
      video/x-raw(memory:DMABuf)                                                
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBG
                  width: [ 1, 2147483647 ]                                      
                 height: [ 1, 2147483647 ]                                      
              framerate: [ 0/1, 2147483647/1 ]                                  
                                                                                
Element has no clocking capabilities.                                           
Element has no URI handling capabilities.                                       
                                                                                
Pads:                                                                           
  SINK: 'sink'                                                                  
    Pad Template: 'sink'

Element Properties:                                                             
  name                : The name of the object                                  
                        flags: readable, writable                               
                        String. Default: "waylandsink0"                         
  parent              : The parent of the object                                
                        flags: readable, writable                               
                        Object of type "GstObject"                              
  sync                : Sync on the clock                                       
                        flags: readable, writable                               
                        Boolean. Default: true                                  
  max-lateness        : Maximum number of nanoseconds that a buffer can be late 
                        flags: readable, writable                               
                        Integer64. Range: -1 - 9223372036854775807 Default: 2000
  qos                 : Generate Quality-of-Service events upstream             
                        flags: readable, writable                               
                        Boolean. Default: true                                  
  async               : Go asynchronously to PAUSED                             
                        flags: readable, writable                               
                        Boolean. Default: true                                  
  ts-offset           : Timestamp offset in nanoseconds                         
                        flags: readable, writable                               
                        Integer64. Range: -9223372036854775808 - 922337203685477
  enable-last-sample  : Enable the last-sample property                         
                        flags: readable, writable                               
                        Boolean. Default: true                                  
  last-sample         : The last sample received in the sink                    
                        flags: readable                                         
                        Boxed pointer of type "GstSample"                       
  blocksize           : Size in bytes to pull per buffer (0 = default)          
                        flags: readable, writable                               
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096   
  render-delay        : Additional render delay of the sink in nanoseconds      
                        flags: readable, writable                               
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Defa
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable                               
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Defa
  max-bitrate         : The maximum bits per second to render (0 = disabled)    
                        flags: readable, writable                               
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Defa
  show-preroll-frame  : Whether to render video frames during preroll           
                        flags: readable, writable                               
                        Boolean. Default: true
  window-width        : Wayland sink preferred window width in pixel            
                        flags: readable, writable                               
                        Integer. Range: -1 - 2147483647 Default: -1             
  window-height       : Wayland sink preferred window height in pixel           
                        flags: readable, writable                               
                        Integer. Range: -1 - 2147483647 Default: -1             
  display             : Wayland display name to connect to, if not supplied via 
                        flags: readable, writable                               
                        String. Default: null                                   
  alpha               : Wayland surface alpha value, apply custom alpha value to
                        flags: readable, writable                               
                        Float. Range:               0 -               1 Default:
  enable-tile         : When enabled, the sink propose VSI tile modifier to VPU 
                        flags: readable, writable                               
                        Boolean. Default: false

1.1.2 H.264 video decoder element

# gst-inspect-1.0 v4l2h264dec
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                V4L2 H264 Decoder
  Klass                    Codec/Decoder/Video
  Description              Decodes H264 streams via V4L2 API
  Author                   Nicolas Dufresne <nicolas.dufresne@collabora.com>

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  Filename                 /usr/lib/gstreamer-1.0/libgstvideo4linux2.so
  Version                  1.14.4
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2018-10-02
  Binary package           GStreamer Good Plug-ins source release
  Origin URL               Unknown package origin

GObject                                                                     
 +----GInitiallyUnowned                                                     
       +----GstObject                                                       
             +----GstElement                                                
                   +----GstVideoDecoder                                     
                         +----GstV4l2VideoDec                               
                               +----v4l2h264dec                        
                                                                       
Pad Templates:                                                         
  SINK template: 'sink'                                                
    Availability: Always                                               
    Capabilities:                                                
      video/x-h264                                               
          stream-format: byte-stream                             
              alignment: au                                      
  SRC template: 'src'                                                       
    Availability: Always                                                    
    Capabilities:                                                           
      video/x-raw                                                           
                 format: NV12                                               
                  width: [ 1, 32768 ]                                       
                 height: [ 1, 32768 ]                                  
              framerate: [ 0/1, 2147483647/1 ]                         
                                                                       
Element has no clocking capabilities.                                  
Element has no URI handling capabilities.                              
                                                                 
Pads:                                                            
  SINK: 'sink'                                                   
    Pad Template: 'sink'                                         
  SRC: 'src'                                                     
    Pad Template: 'src'      
                    
Element Properties:                                                         
  name                : The name of the object                              
                        flags: readable, writable                           
                        String. Default: "v4l2h264dec0"                     
  parent              : The parent of the object                            
                        flags: readable, writable                           
                        Object of type "GstObject"                     
  device              : Device location                                
                        flags: readable                                
                        String. Default: "/dev/video12"                
  device-name         : Name of the device                             
                        flags: readable                          
                        String. Default: null                    
  device-fd           : File descriptor of the device            
                        flags: readable                          
                        Integer. Range: -1 - 2147483647 Default: -1
  output-io-mode      : Output side I/O mode (matches sink pad)             
                        flags: readable, writable                           
                        Enum "GstV4l2IOMode" Default: 0, "auto"             
                           (0): auto             - GST_V4L2_IO_AUTO         
                           (1): rw               - GST_V4L2_IO_RW      
                           (2): mmap             - GST_V4L2_IO_MMAP    
                           (3): userptr          - GST_V4L2_IO_USERPTR 
                           (4): dmabuf           - GST_V4L2_IO_DMABUF  
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  capture-io-mode     : Capture I/O mode (matches src pad)                  
                        flags: readable, writable                           
                        Enum "GstV4l2IOMode" Default: 0, "auto"             
                           (0): auto             - GST_V4L2_IO_AUTO         
                           (1): rw               - GST_V4L2_IO_RW           
                           (2): mmap             - GST_V4L2_IO_MMAP         
                           (3): userptr          - GST_V4L2_IO_USERPTR      
                           (4): dmabuf           - GST_V4L2_IO_DMABUF       
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  extra-controls      : Extra v4l2 controls (CIDs) for the device           
                        flags: readable, writable                           
                        Boxed pointer of type "GstStructure"                

1.1.3 H.264 video encoder element

# gst-inspect-1.0 v4l2h264enc
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                V4L2 H.264 Encoder
  Klass                    Codec/Encoder/Video
  Description              Encode H.264 video streams via V4L2 API
  Author                   ayaka <ayaka@soulik.info>

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  Filename                 /usr/lib/gstreamer-1.0/libgstvideo4linux2.so
  Version                  1.14.4
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2018-10-02
  Binary package           GStreamer Good Plug-ins source release
  Origin URL               Unknown package origin

GObject                                                                
 +----GInitiallyUnowned                                                
       +----GstObject                                                  
             +----GstElement                                           
                   +----GstVideoEncoder                                
                         +----GstV4l2VideoEnc                          
                               +----GstV4l2H264Enc                     
                                     +----v4l2h264enc                  
                                                                       
Implemented Interfaces:                                                
  GstPreset                                                            
                                                                 
Pad Templates:                                                   
  SINK template: 'sink'                                          
    Availability: Always                                         
    Capabilities:                                                
      video/x-raw                                    
                 format: NV12                        
                  width: [ 1, 32768 ]                
                 height: [ 1, 32768 ]                
              framerate: [ 0/1, 2147483647/1 ]
  SRC template: 'src'                                                  
    Availability: Always                                               
    Capabilities:                                                      
      video/x-h264                                                     
          stream-format: byte-stream                                   
              alignment: au                                            
                                                                       
Element has no clocking capabilities.                                  
Element has no URI handling capabilities.                              
                                                                       
Pads:                                                                  
  SINK: 'sink'                                                   
    Pad Template: 'sink'                                         
  SRC: 'src'                                                     
    Pad Template: 'src'                                          

Element Properties:                                                    
  name                : The name of the object                         
                        flags: readable, writable                      
                        String. Default: "v4l2h264enc0"                
  parent              : The parent of the object                       
                        flags: readable, writable                      
                        Object of type "GstObject"                     
  qos                 : Handle Quality-of-Service events from downstream
                        flags: readable, writable                       
                        Boolean. Default: false                         
  device              : Device location                                 
                        flags: readable                                 
                        String. Default: "/dev/video13"                 
  device-name         : Name of the device                              
                        flags: readable                                 
                        String. Default: null                           
  device-fd           : File descriptor of the device                   
                        flags: readable                                 
                        Integer. Range: -1 - 2147483647 Default: -1
  output-io-mode      : Output side I/O mode (matches sink pad)         
                        flags: readable, writable                       
                        Enum "GstV4l2IOMode" Default: 0, "auto"         
                           (0): auto             - GST_V4L2_IO_AUTO     
                           (1): rw               - GST_V4L2_IO_RW       
                           (2): mmap             - GST_V4L2_IO_MMAP     
                           (3): userptr          - GST_V4L2_IO_USERPTR  
                           (4): dmabuf           - GST_V4L2_IO_DMABUF   
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  capture-io-mode     : Capture I/O mode (matches src pad)                  
                        flags: readable, writable                           
                        Enum "GstV4l2IOMode" Default: 0, "auto"             
                           (0): auto             - GST_V4L2_IO_AUTO         
                           (1): rw               - GST_V4L2_IO_RW           
                           (2): mmap             - GST_V4L2_IO_MMAP         
                           (3): userptr          - GST_V4L2_IO_USERPTR      
                           (4): dmabuf           - GST_V4L2_IO_DMABUF       
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  extra-controls      : Extra v4l2 controls (CIDs) for the device           
                        flags: readable, writable                           
                        Boxed pointer of type "GstStructure"                

2 Test Pattern Stream

Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:

# gst-launch-1.0 videotestsrc ! autovideosink

Gst1.jpg

Test pattern with specific parameters on the input element:

# gst-launch-1.0 videotestsrc pattern=circular ! autovideosink

Gst2.jpg

3 Camera Loopback

Stream video from a camera to a display:

# gst-launch-1.0 v4l2src device=/dev/videoX ! autovideosink

Replace /dev/videoX with the actual camera device.
By default, the MIPI camera device is /dev/video0 & the parallel camera device is /dev/video1.

4 Video Recording

The i.MX8QXP can encode video to H.264 at 1080p30 using its internal hardware encoder.
For example, you can see the src pad capabilities in the vpuenc element details for the complete list of features supported by the H.264 encoder.

Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! v4l2h264enc ! avimux ! filesink location=./test_h264.avi

5 Video Playback

The i.MX8QXP can decode and playback H.265 encoded video at 4Kp30 and H.264 encoded video at 1080p60 using its internal hardware decoder.
For example, you can see the sink pad capabilities in the vpudec element details for the complete list of features supported by the H.264 decoder.

Example of a video file playback using the playbin element (making use of the i.MX8QXP VPU decoder):

# gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 

Example of H.264 video playback while explicitly specifying a pipeline (making use of the i.MX8QXP VPU decoder):

# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! autovideosink

Example of H.265 video playback while explicitly specifying a pipeline (making use of the i.MX8QXP VPU decoder):

# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux ! queue ! h265parse ! v4l2h265dec ! imxvideoconvert_g2d ! queue ! autovideosink

6 RTP Video streaming

Encode video stream from camera using a hardware H.264 encoder and stream it over RTP:

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! v4l2h264enc ! rtph264pay ! udpsink host=192.168.20.60 port=1223

7 References