DART-MX8M GSTREAMER: Difference between revisions

From Variscite Wiki
No edit summary
 
(50 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{PageHeader|DART-MX8M - Streaming Video}} {{DocImage|category1=DART-MX8M|category2=Yocto}} __toc__
{{INIT_RELEASE_PARAM|RELEASE_SUMO_V1.4_DART-MX8M-MINI}}<!--
--> {{#lst:Yocto_Platform_Customization|{{#var:RELEASE_PARAM}}}} <!--
--> {{#lst:B2QT_Platform_Customization|{{#var:RELEASE_PARAM}}}} <!--
--> {{#lst:Debian_Platform_Customization|{{#var:RELEASE_PARAM}}}} <!--
--> {{PageHeader|{{#var:HARDWARE_NAME}} Streaming Video}} {{DocImage|category1=Yocto|category2=Debian}}[[Category:{{#var:HARDWARE_NAME}}]] __toc__


= Introduction to GStreamer =
{{GSTREAMER_INTRO}}
[https://en.wikipedia.org/wiki/GStreamer GStreamer] is a powerful pipeline-based multimedia framework.<br>
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.<br>
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.<br>
This release uses the [https://source.codeaurora.org/external/imx/gstreamer/ gstreamer-imx] set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
 
== Major GStreamer commands ==
There are two major commands:<br>
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.<br>
gst-launch - allows you to create and execute GStreamer  pipelines.


=== gst-inspect ===
=== gst-inspect ===
Lists all available elements:
Lists all available elements:
  $ gst-inspect-1.0
  # gst-inspect-1.0
Lists all available sink elements:
Lists all available sink elements:
  $ gst-inspect-1.0 | grep sink
  # gst-inspect-1.0 | grep sink
 
For example:
For example:
<pre>
<pre>
root@imx8m-var-dart:~# gst-inspect-1.0 | grep sink
# gst-inspect-1.0 | grep sink
alsa:  alsasink: Audio sink (ALSA)
alsa:  alsasink: Audio sink (ALSA)
waylandsink:  waylandsink: wayland video sink
waylandsink:  waylandsink: wayland video sink
Line 38: Line 34:
</pre>
</pre>
Detailed help information on a specific element:
Detailed help information on a specific element:
  $ gst-inspect-1.0 <name of the element>
  # gst-inspect-1.0 <name of the element>
For example, the output for kmssink:
For example, the output for waylandsink:
<pre>
<pre>
root@imx8m-var-dart:~# gst-inspect-1.0 kmssink
# gst-inspect-1.0 waylandsink
Factory Details:
Factory Details:
   Rank                    secondary (128)
   Rank                    primary + 2 (258)
   Long-name                KMS video sink
   Long-name                wayland video sink
   Klass                    Sink/Video
   Klass                    Sink/Video
   Description              Video sink using the Linux kernel mode setting API
   Description              Output to wayland surface
   Author                  Víctor Jáquez <vjaquez@igalia.com>
   Author                  Sreerenj Balachandran <sreerenj.balachandran@intel.com>, George Kiagiadakis <george.kiagiadakis@collabora.com>


Plugin Details:
Plugin Details:
   Name                    kms
   Name                    waylandsink
   Description              Video sink using the Linux kernel mode setting API
   Description              Wayland Video Sink
   Filename                /usr/lib/gstreamer-1.0/libgstkms.so
   Filename                /usr/lib/gstreamer-1.0/libgstwaylandsink.so
   Version                  1.12.2
   Version                  1.14.0
   License                  LGPL
   License                  LGPL
   Source module            gst-plugins-bad
   Source module            gst-plugins-bad
   Source release date      2017-07-14
   Source release date      2018-03-19
   Binary package          GStreamer Bad Plug-ins source release
   Binary package          GStreamer Bad Plug-ins source release
   Origin URL              Unknown package origin
   Origin URL              Unknown package origin
Line 66: Line 62:
                   +----GstBaseSink
                   +----GstBaseSink
                         +----GstVideoSink
                         +----GstVideoSink
                               +----GstKMSSink
                               +----GstWaylandSink


Implemented Interfaces:
Implemented Interfaces:
   GstVideoOverlay
   GstVideoOverlay
  GstWaylandVideo


Pad Templates:
Pad Templates:
Line 76: Line 73:
     Capabilities:
     Capabilities:
       video/x-raw
       video/x-raw
                 format: { (string)BGRA, (string)BGRx, (string)RGBA, (string)RGBx, (string)UYVY, (string)YUY2, (string)YVYU,
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)I420, (string)YV12, (string)Y42B, (string)NV12, (string)NV12_10LE, (string)NV21, (string)NV16 }
                          (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                          (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:DMABuf)
                format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                          (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                          (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                          (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                   width: [ 1, 2147483647 ]
                   width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
               framerate: [ 0/1, 2147483647/1 ]
               framerate: [ 0/1, 2147483647/1 ]
Element Flags:
  no flags set
Element Implementation:
  Has change_state() function: gst_kms_sink_change_state


Element has no clocking capabilities.
Element has no clocking capabilities.
Line 99: Line 99:
   name                : The name of the object
   name                : The name of the object
                         flags: readable, writable
                         flags: readable, writable
                         String. Default: "kmssink0"
                         String. Default: "waylandsink0"
   parent              : The parent of the object
   parent              : The parent of the object
                         flags: readable, writable
                         flags: readable, writable
Line 139: Line 139:
                         flags: readable, writable
                         flags: readable, writable
                         Boolean. Default: true
                         Boolean. Default: true
   driver-name        : DRM device driver name
   window-width       : Wayland sink preferred window width in pixel
                        flags: readable, writable
                        String. Default: null
  connector-id       : DRM connector id
                         flags: readable, writable
                         flags: readable, writable
                         Integer. Range: -1 - 2147483647 Default: -1  
                         Integer. Range: -1 - 2147483647 Default: -1  
   plane-id            : DRM plane id
   window-height      : Wayland sink preferred window height in pixel
                         flags: readable, writable
                         flags: readable, writable
                         Integer. Range: -1 - 2147483647 Default: -1  
                         Integer. Range: -1 - 2147483647 Default: -1  
   force-modesetting  : When enabled, the sink try to configure the display mode
   display            : Wayland display name to connect to, if not supplied via the GstContext
                         flags: readable, writable
                         flags: readable, writable
                         Boolean. Default: false
                         String. Default: null
   global-alpha       : global alpha
   alpha               : Wayland surface alpha value, apply custom alpha value to wayland surface
                         flags: readable, writable
                         flags: readable, writable
                         Integer. Range: 0 - 255 Default: 0  
                         Float. Range:               0 -               1 Default:               0  
   force-hantrotile    : When enabled, the sink propose hantro tile modifier to VPU
   enable-tile        : When enabled, the sink propose VSI tile modifier to VPU
                         flags: readable, writable
                         flags: readable, writable
                         Boolean. Default: false
                         Boolean. Default: false
</pre>
</pre>


 
{{#ifeq: {{#var:HARDWARE_NAME}}|DART-MX8M|
=== The i.MX VPU video decoder element ===
=== The i.MX8M VPU video decoder element ===
<pre>
<pre>
root@imx8m-var-dart:~# gst-inspect-1.0 vpudec
# gst-inspect-1.0 vpudec
Factory Details:
Factory Details:
   Rank                    primary + 1 (257)
   Rank                    primary + 1 (257)
Line 275: Line 272:
                         Boolean. Default: true
                         Boolean. Default: true
</pre>
</pre>
| {{#ifeq: {{#var:HARDWARE_NAME}}|DART-MX8M-MINI |
=== The i.MX8M-MINI VPU video decoder element ===
<pre>
# gst-inspect-1.0 vpudec
Factory Details:
  Rank                    primary + 1 (257)
  Long-name                IMX VPU-based video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode compressed video to raw data
  Author                  Multimedia Team <shmmmw@freescale.com>
Plugin Details:
  Name                    vpu
  Description              VPU video codec
  Filename                /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.4.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package          Freescle Gstreamer Multimedia Plugins
  Origin URL              http://www.freescale.com
GObject
+----GInitiallyUnowned
      +----GstObject
            +----GstElement
                  +----GstVideoDecoder
                        +----GstVpuDec
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
                  width: [ 1, 2147483647 ]
                height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
 
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
      video/x-vp9
      video/x-vp8
      video/x-h264
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'
Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpudec0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  output-format      : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
                        flags: readable, writable
                        Enum "GstVpuDecOutputFormat" Default: 0, "auto"
                          (0): auto            - enable chroma interleave. (default)
                          (1): NV12            - NV12 format
                          (2): I420            - I420 format
                          (3): YV12            - YV12 format
                          (4): Y42B            - Y42B format
                          (5): NV16            - NV16 format
                          (6): Y444            - Y444 format
                          (7): NV24            - NV24 format
  frame-drop          : enable adaptive frame drop for smoothly playback
                        flags: readable, writable
                        Boolean. Default: true
  frame-plus          : set number of addtional frames for smoothly playback
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 16 Default: 3
  use-vpu-memory      : use vpu allocate video frame buffer
                        flags: readable, writable
                        Boolean. Default: true
</pre>
| {{#ifeq: {{#var:HARDWARE_NAME}}|DART-MX8M-PLUS |
=== The i.MX8M-PLUS VPU video decoder element ===
<pre>
Factory Details:
  Rank                    primary + 1 (257)
  Long-name                IMX VPU-based video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode compressed video to raw data
  Author                  Multimedia Team <shmmmw@freescale.com>


= Test Pattern Stream =
Plugin Details:
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
  Name                    vpu
<br/>Default test pattern:
  Description              VPU video codec
  Filename                /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.5.6
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package          Freescle Gstreamer Multimedia Plugins
  Origin URL              http://www.freescale.com
 
GObject
+----GInitiallyUnowned
      +----GstObject
            +----GstElement
                  +----GstVideoDecoder
                        +----GstVpuDec
 
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
                  width: [ 1, 2147483647 ]
                height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
 
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
      video/x-vp9
      video/x-vp8
      video/x-h264
 
Element has no clocking capabilities.
Element has no URI handling capabilities.
 
Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'
 
Element Properties:
  disable-reorder    : disable vpu reorder when end to end streaming
                        flags: readable, writable
                        Boolean. Default: false
  frame-drop          : enable adaptive frame drop for smoothly playback
                        flags: readable, writable
                        Boolean. Default: true
  frame-plus          : set number of addtional frames for smoothly playback
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 16 Default: 3
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpudec0"
  output-format      : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
                        flags: readable, writable
                        Enum "GstVpuDecOutputFormat" Default: 0, "auto"
                          (0): auto            - enable chroma interleave. (default)
                          (1): NV12            - NV12 format
                          (2): I420            - I420 format
                          (3): YV12            - YV12 format
                          (4): Y42B            - Y42B format
                          (5): NV16            - NV16 format
                          (6): Y444            - Y444 format
                          (7): NV24            - NV24 format
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  use-vpu-memory      : use vpu allocate video frame buffer
                        flags: readable, writable
                        Boolean. Default: true
</pre>
|}}}}}}
 
{{#ifeq: {{#var:HARDWARE_NAME}}|DART-MX8M-MINI|
=== The i.MX8M-MINI VPU H.264 video encoder element ===
<pre>
# gst-inspect-1.0 vpuenc_h264
Factory Details:
  Rank                    primary + 1 (257)
  Long-name                IMX VPU-based AVC/H264 video encoder
  Klass                    Codec/Encoder/Video
  Description              Encode raw data to compressed video
  Author                  Multimedia Team <shmmmw@freescale.com>
 
Plugin Details:
  Name                    vpu
  Description              VPU video codec
  Filename                /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.4.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package          Freescle Gstreamer Multimedia Plugins
  Origin URL              http://www.freescale.com
 
GObject
+----GInitiallyUnowned
      +----GstObject
            +----GstElement
                  +----GstVideoEncoder
                        +----vpuenc_h264
 
Implemented Interfaces:
  GstPreset
 
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: { (string)avc, (string)byte-stream }
              alignment: { (string)au, (string)nal }
 
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16,
                          (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
                  width: [ 64, 1920, 8 ]
                height: [ 64, 1088, 8 ]
              framerate: [ 0/1, 2147483647/1 ]
 
Element has no clocking capabilities.
Element has no URI handling capabilities.
 
Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'
 
Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpuenc_h264-0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false
  bitrate            : set bit rate in kbps (0 for automatic)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  gop-size            : How many frames a group-of-picture shall contain
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 32767 Default: 30
  quant              : set quant value: H.264(0-51) (-1 for automatic)
                        flags: readable, writable
                        Integer. Range: -1 - 51 Default: -1
</pre>
|{{#ifeq: {{#var:HARDWARE_NAME}}|DART-MX8M-PLUS|
=== The i.MX8M-PLUS VPU H.264 video encoder element ===
<pre>
Factory Details:
  Rank                    primary + 1 (257)
  Long-name                IMX VPU-based AVC/H264 video encoder
  Klass                    Codec/Encoder/Video
  Description              Encode raw data to compressed video
  Author                  Multimedia Team <shmmmw@freescale.com>
 
Plugin Details:
  Name                    vpu
  Description              VPU video codec
  Filename                /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.5.6
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package          Freescle Gstreamer Multimedia Plugins
  Origin URL              http://www.freescale.com
 
GObject
+----GInitiallyUnowned
      +----GstObject
            +----GstElement
                  +----GstVideoEncoder
                        +----vpuenc_h264
 
Implemented Interfaces:
  GstPreset
 
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: { (string)avc, (string)byte-stream }
              alignment: { (string)au, (string)nal }
 
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16, (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
                  width: [ 64, 1920 ]
                height: [ 64, 1088 ]
              framerate: [ 0/1, 2147483647/1 ]
 
Element has no clocking capabilities.
Element has no URI handling capabilities.
 
Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'
 
Element Properties:
  bitrate            : set bit rate in kbps (0 for automatic)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  gop-size            : How many frames a group-of-picture shall contain
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 32767 Default: 30
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpuenc_h264-0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false
  quant              : set quant value: H.264(0-51) (-1 for automatic)
                        flags: readable, writable
                        Integer. Range: -1 - 51 Default: -1
  stream-multislice  : the number of slices a picture contains
                        flags: readable, writable
                        Integer. Range: 1 - 2147483647 Default: 1
</pre>
 
=== The i.MX8M-PLUS VPU H.265 video encoder element ===
<pre>
<pre>
$ gst-launch-1.0 videotestsrc ! kmssink
Factory Details:
  Rank                    primary + 1 (257)
  Long-name                IMX VPU-based HEVC video encoder
  Klass                    Codec/Encoder/Video
  Description              Encode raw data to compressed video
  Author                  Multimedia Team <shmmmw@freescale.com>
 
Plugin Details:
  Name                    vpu
  Description              VPU video codec
  Filename                /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.5.6
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package          Freescle Gstreamer Multimedia Plugins
  Origin URL              http://www.freescale.com
 
GObject
+----GInitiallyUnowned
      +----GstObject
            +----GstElement
                  +----GstVideoEncoder
                        +----vpuenc_hevc
 
Implemented Interfaces:
  GstPreset
 
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h265
                variant: itu
                  width: [ 64, 1920 ]
                height: [ 64, 1088 ]
              framerate: [ 0/1, 2147483647/1 ]
 
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16, (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
                  width: [ 64, 1920 ]
                height: [ 64, 1088 ]
              framerate: [ 0/1, 2147483647/1 ]
 
Element has no clocking capabilities.
Element has no URI handling capabilities.
 
Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'
 
Element Properties:
  bitrate            : set bit rate in kbps (0 for automatic)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  gop-size            : How many frames a group-of-picture shall contain
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 32767 Default: 30
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpuenc_hevc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false
  quant              : set quant value: HEVC(0-51) (-1 for automatic)
                        flags: readable, writable
                        Integer. Range: -1 - 51 Default: -1
  stream-multislice  : the number of slices a picture contains
                        flags: readable, writable
                        Integer. Range: 1 - 2147483647 Default: 1
</pre>
</pre>
[[File:gst1.jpg]]
|}}}}
<br/><br/>Test pattern with specific parameters on the input element
 
<pre>$ gst-launch-1.0 videotestsrc pattern=circular ! kmssink </pre>
= Test Pattern Stream =
{{#vardefine:DEFAULT_SINK | {{#varexists:USE_KMSSINK | kmssink can-scale=false | autovideosink }} }}
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.<br>
Default test pattern:
 
# gst-launch-1.0 videotestsrc ! {{#var:DEFAULT_SINK}}
[[File:gst1.jpg]]<br>
<br>
Test pattern with specific parameters on the input element:
# gst-launch-1.0 videotestsrc pattern=circular ! {{#var:DEFAULT_SINK}}
[[File:gst2.jpg]]
[[File:gst2.jpg]]


= Camera Loopback  =
= Camera Loopback  =
Stream video from a camera to a display</br>
Stream video from a camera to a display:
  $ gst-launch-1.0 v4l2src device=/dev/video0 ! kmssink
  # gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink


= Video Recording =  
= Video Recording =  
Save video stream from camera as an MJPEG-encoded AVI file
{{#ifeq: {{#var:HARDWARE_NAME}} | DART-MX8M-MINI |
  $ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test0.avi
The i.MX8M-MINI can encode videos to AVC(H.264) & VP8 at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).<br>
See the src pad capabilities in the [[#The i.MX8M-MINI VPU H.264 video encoder element|vpuenc_264]] and vpuenc_vp8 elements details for the complete list of features supported by the i.MX8M-MINI VPU encoder.<br>
<br>
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
  # gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi
}}
 
{{#ifeq: {{#var:HARDWARE_NAME}} | DART-MX8M-PLUS |
The i.MX8M-PLUS can encode videos to AVC(H.264) & HEVC(H.265) at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).<br>
See the src pad capabilities in the [[#The i.MX8M-PLUS VPU H.264 video encoder element|vpuenc_264]] and [[#The i.MX8M-PLUS VPU H.265 video encoder element|vpuenc_265]] elements details for the complete list of features supported by the i.MX8M-PLUS VPU encoder.<br>
<br>
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi


Example of encoding and saving a short video stream from a camera to an H.265-encoded TS file:
# gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_hevc ! mpegtsmux ! filesink location=./test_h265.ts
}}
Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file{{#ifeq: {{#var:HARDWARE_NAME}} | DART-MX8M-MINI |&nbsp;(using the SW based jpegenc element - without using the VPU)}}:
# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi
{{#ifeq: {{#var:HARDWARE_NAME}} | VAR-SOM-MX8M-NANO | |
= Video Playback =
= Video Playback =
The i.MX8M can decode and playback HEVC(H265), H264, VP9, etc. (see the vpudec element details above for the rest of the codec list) encoded videos with HDR at resolutions of up to 4K UHD (3840×2160).<br>
{{#ifeq: {{#var:HARDWARE_NAME}} | DART-MX8M-MINI |
The i.MX8M-MINI can decode and playback HEVC(H.265), AVC(H.264), VP9, VP8, etc. encoded videos at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).<br>
See the sink pad capabilities in the [[#The i.MX8M-MINI VPU video decoder element|vpudec element details]] for the complete list of codecs supported by the i.MX8M-MINI VPU decoder.<br>
| {{#ifeq: {{#var:HARDWARE_NAME}} | DART-MX8M-PLUS |
The i.MX8M-PLUS can decode and playback AVC(H.264), HEVC(H.265), VP9, VP8, etc. encoded videos with at resolutions of up to 1080p60 (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).<br>
See the sink pad capabilities in the [[#The i.MX8M-PLUS VPU video decoder element|vpudec element details]] for the complete list of codecs supported by the i.MX8M-PLUS VPU decoder.<br>
|
The i.MX8M can decode and playback HEVC(H.265), VP9, etc. encoded videos with HDR at resolutions of up to 4Kp60 UHD (3840×2160p @ 60fps) using its internal hardware Video Processing Unit (VPU).<br>
See the sink pad capabilities in the [[#The i.MX8M VPU video decoder element|vpudec element details]] for the complete list of codecs supported by the i.MX8M VPU decoder.<br>
}} }} }}
 
{{#ifeq: {{#var:HARDWARE_NAME}} | VAR-SOM-MX8M-NANO | |
{{#vardefine:PLAYBACK_VIDEO_CMD1 | {{#varexists:USE_KMSSINK | gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 video-sink="imxvideoconvert_g2d ! queue ! kmssink can-scale=false" audio-sink="alsasink" | gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 }} }}
{{#vardefine:PLAYBACK_VIDEO_CMD2 | {{#varexists:USE_KMSSINK | gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! imxvideoconvert_g2d ! video/x-raw,width=800,height=480 ! kmssink can-scale=false | gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink }} }}
<br>
<br>
Example of a video file playback using the playbin element (making use of the i.MX8M VPU decoder):
Example of a video file playback using the playbin element (making use of the VPU decoder):
  $ gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4
  # {{#var:PLAYBACK_VIDEO_CMD1}}


Example of a video file playback while explicitly specifying a pipeline (making use of the i.MX8M VPU decoder):
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
  $ gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! kmssink
  # {{#var:PLAYBACK_VIDEO_CMD2}}
}}


= RTP Video streaming =
= RTP Video streaming =
Encode video stream from camera using a software jpeg encoder and stream it over RTP
{{GSTREAMER_RTP}}
$ gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223


= References =
= References =
* The GStreamer [https://gstreamer.freedesktop.org/ website]
{{GSTREAMER_REF}}
* NXP BSP Linux Users Guide, Multimedia section [https://www.nxp.com/webapp/Download?colCode=L4.9.51_8MQ_GA_LINUX_DOCS&Parent_nodeId=1513109797983730444282&Parent_pageType=product&Parent_nodeId=1513109797983730444282&Parent_pageType=product download link ]

Latest revision as of 15:07, 21 March 2023

Warning: This page is designed to be used with a 'release' URL parameter.

This page is using the default release RELEASE_SUMO_V1.4_DART-MX8M-MINI.
To view this page for a specific Variscite SoM and software release, please follow these steps:

  1. Visit variwiki.com
  2. Select your SoM
  3. Select the software release
DART-MX8M-MINI Streaming Video

Introduction to GStreamer

GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.

Major GStreamer commands

There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.

gst-inspect

Lists all available elements:

# gst-inspect-1.0

Lists all available sink elements:

# gst-inspect-1.0 | grep sink

For example:

# gst-inspect-1.0 | grep sink
alsa:  alsasink: Audio sink (ALSA)
waylandsink:  waylandsink: wayland video sink
shm:  shmsink: Shared Memory Sink
decklink:  decklinkaudiosink: Decklink Audio Sink
decklink:  decklinkvideosink: Decklink Video Sink
ximagesink:  ximagesink: Video sink
soup:  souphttpclientsink: HTTP client sink
autodetect:  autovideosink: Auto video sink
autodetect:  autoaudiosink: Auto audio sink
gdkpixbuf:  gdkpixbufsink: GdkPixbuf sink
playback:  playsink: Player Sink
video4linux2:  v4l2sink: Video (video4linux2) Sink
fbdevsink:  fbdevsink: fbdev video sink
tcp:  tcpclientsink: TCP client sink
tcp:  tcpserversink: TCP server sink
…

Detailed help information on a specific element:

# gst-inspect-1.0 <name of the element>

For example, the output for waylandsink:

# gst-inspect-1.0 waylandsink
Factory Details:
  Rank                     primary + 2 (258)
  Long-name                wayland video sink
  Klass                    Sink/Video
  Description              Output to wayland surface
  Author                   Sreerenj Balachandran <sreerenj.balachandran@intel.com>, George Kiagiadakis <george.kiagiadakis@collabora.com>

Plugin Details:
  Name                     waylandsink
  Description              Wayland Video Sink
  Filename                 /usr/lib/gstreamer-1.0/libgstwaylandsink.so
  Version                  1.14.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2018-03-19
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSink
                         +----GstVideoSink
                               +----GstWaylandSink

Implemented Interfaces:
  GstVideoOverlay
  GstWaylandVideo

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                           (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:DMABuf)
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                           (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "waylandsink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  sync                : Sync on the clock
                        flags: readable, writable
                        Boolean. Default: true
  max-lateness        : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
                        flags: readable, writable
                        Integer64. Range: -1 - 9223372036854775807 Default: 20000000 
  qos                 : Generate Quality-of-Service events upstream
                        flags: readable, writable
                        Boolean. Default: true
  async               : Go asynchronously to PAUSED
                        flags: readable, writable
                        Boolean. Default: true
  ts-offset           : Timestamp offset in nanoseconds
                        flags: readable, writable
                        Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0 
  enable-last-sample  : Enable the last-sample property
                        flags: readable, writable
                        Boolean. Default: true
  last-sample         : The last sample received in the sink
                        flags: readable
                        Boxed pointer of type "GstSample"
  blocksize           : Size in bytes to pull per buffer (0 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  render-delay        : Additional render delay of the sink in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  max-bitrate         : The maximum bits per second to render (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  show-preroll-frame  : Whether to render video frames during preroll
                        flags: readable, writable
                        Boolean. Default: true
  window-width        : Wayland sink preferred window width in pixel
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  window-height       : Wayland sink preferred window height in pixel
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  display             : Wayland display name to connect to, if not supplied via the GstContext
                        flags: readable, writable
                        String. Default: null
  alpha               : Wayland surface alpha value, apply custom alpha value to wayland surface
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  enable-tile         : When enabled, the sink propose VSI tile modifier to VPU
                        flags: readable, writable
                        Boolean. Default: false

The i.MX8M-MINI VPU video decoder element

# gst-inspect-1.0 vpudec
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                IMX VPU-based video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode compressed video to raw data
  Author                   Multimedia Team <shmmmw@freescale.com>

Plugin Details:
  Name                     vpu
  Description              VPU video codec
  Filename                 /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.4.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package           Freescle Gstreamer Multimedia Plugins
  Origin URL               http://www.freescale.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstVpuDec

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
      video/x-vp9
      video/x-vp8
      video/x-h264

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpudec0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  output-format       : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
                        flags: readable, writable
                        Enum "GstVpuDecOutputFormat" Default: 0, "auto"
                           (0): auto             - enable chroma interleave. (default)
                           (1): NV12             - NV12 format
                           (2): I420             - I420 format
                           (3): YV12             - YV12 format
                           (4): Y42B             - Y42B format
                           (5): NV16             - NV16 format
                           (6): Y444             - Y444 format
                           (7): NV24             - NV24 format
  frame-drop          : enable adaptive frame drop for smoothly playback
                        flags: readable, writable
                        Boolean. Default: true
  frame-plus          : set number of addtional frames for smoothly playback
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 16 Default: 3 
  use-vpu-memory      : use vpu allocate video frame buffer
                        flags: readable, writable
                        Boolean. Default: true

The i.MX8M-MINI VPU H.264 video encoder element

# gst-inspect-1.0 vpuenc_h264
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                IMX VPU-based AVC/H264 video encoder
  Klass                    Codec/Encoder/Video
  Description              Encode raw data to compressed video
  Author                   Multimedia Team <shmmmw@freescale.com>

Plugin Details:
  Name                     vpu
  Description              VPU video codec
  Filename                 /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.4.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package           Freescle Gstreamer Multimedia Plugins
  Origin URL               http://www.freescale.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoEncoder
                         +----vpuenc_h264

Implemented Interfaces:
  GstPreset

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: { (string)avc, (string)byte-stream }
              alignment: { (string)au, (string)nal }
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16,
                           (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
                  width: [ 64, 1920, 8 ]
                 height: [ 64, 1088, 8 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpuenc_h264-0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false
  bitrate             : set bit rate in kbps (0 for automatic)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0 
  gop-size            : How many frames a group-of-picture shall contain
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 32767 Default: 30 
  quant               : set quant value: H.264(0-51) (-1 for automatic)
                        flags: readable, writable
                        Integer. Range: -1 - 51 Default: -1

Test Pattern Stream

Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:

# gst-launch-1.0 videotestsrc ! autovideosink

Gst1.jpg

Test pattern with specific parameters on the input element:

# gst-launch-1.0 videotestsrc pattern=circular ! autovideosink

Gst2.jpg

Camera Loopback

Stream video from a camera to a display:

# gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink

Video Recording

The i.MX8M-MINI can encode videos to AVC(H.264) & VP8 at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the src pad capabilities in the vpuenc_264 and vpuenc_vp8 elements details for the complete list of features supported by the i.MX8M-MINI VPU encoder.

Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi


Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file (using the SW based jpegenc element - without using the VPU):

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi

Video Playback

The i.MX8M-MINI can decode and playback HEVC(H.265), AVC(H.264), VP9, VP8, etc. encoded videos at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M-MINI VPU decoder.


Example of a video file playback using the playbin element (making use of the VPU decoder):

# gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 

Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):

# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink

RTP Video streaming

Encode video stream from camera using a software based JPEG encoder and stream it over RTP:

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! \
     jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223


References