DART-MX8M GSTREAMER: Difference between revisions

From Variscite Wiki
No edit summary
No edit summary
 
(17 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<!-- Set release according to "release" parameter in URL and use RELEASE_SUMO_V1.4_DART-MX8M-MINI as default
{{INIT_RELEASE_PARAM|RELEASE_SUMO_V1.4_DART-MX8M-MINI}}<!--
--> {{#vardefine:RELEASE_PARAM|{{#urlget:release}}}} <!--
--> {{#lst:Yocto_Platform_Customization|{{#var:RELEASE_PARAM}}}} <!--
--> {{#lst:Yocto_Platform_Customization|{{#var:RELEASE_PARAM|RELEASE_SUMO_V1.4_DART-MX8M-MINI}}}} <!--
--> {{#lst:B2QT_Platform_Customization|{{#var:RELEASE_PARAM}}}} <!--
--> {{PageHeader|{{#var:HARDWARE_NAME}} Streaming Video}} {{DocImage|category1=Yocto|category2={{#var:HARDWARE_NAME}}}}[[Category:Debian]] __toc__
--> {{#lst:Debian_Platform_Customization|{{#var:RELEASE_PARAM}}}} <!--
--> {{PageHeader|{{#var:HARDWARE_NAME}} Streaming Video}} {{DocImage|category1=Yocto|category2=Debian}}[[Category:{{#var:HARDWARE_NAME}}]] __toc__


= Introduction to GStreamer =
{{GSTREAMER_INTRO}}
[https://en.wikipedia.org/wiki/GStreamer GStreamer] is a powerful pipeline-based multimedia framework.<br>
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.<br>
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.<br>
This release uses the [https://source.codeaurora.org/external/imx/gstreamer/ gstreamer-imx] set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
 
== Major GStreamer commands ==
There are two major commands:<br>
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.<br>
gst-launch - allows you to create and execute GStreamer  pipelines.


=== gst-inspect ===
=== gst-inspect ===
Line 20: Line 12:
Lists all available sink elements:
Lists all available sink elements:
  # gst-inspect-1.0 | grep sink
  # gst-inspect-1.0 | grep sink
For example:
For example:
<pre>
<pre>
Line 690: Line 683:


= Test Pattern Stream =
= Test Pattern Stream =
{{#vardefine:DEFAULT_SINK | {{#varexists:USE_KMSSINK | kmssink can-scale=false | autovideosink }} }}
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.<br>
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.<br>
Default test pattern:
Default test pattern:
  # gst-launch-1.0 videotestsrc ! autovideosink
 
  # gst-launch-1.0 videotestsrc ! {{#var:DEFAULT_SINK}}
[[File:gst1.jpg]]<br>
[[File:gst1.jpg]]<br>
<br>
<br>
Test pattern with specific parameters on the input element:
Test pattern with specific parameters on the input element:
  # gst-launch-1.0 videotestsrc pattern=circular ! autovideosink
  # gst-launch-1.0 videotestsrc pattern=circular ! {{#var:DEFAULT_SINK}}
[[File:gst2.jpg]]
[[File:gst2.jpg]]


Line 720: Line 715:


Example of encoding and saving a short video stream from a camera to an H.265-encoded TS file:
Example of encoding and saving a short video stream from a camera to an H.265-encoded TS file:
  # gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h265 ! mpegtsmux ! filesink location=./test_h265.ts
  # gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_hevc ! mpegtsmux ! filesink location=./test_h265.ts
}}
}}


Line 740: Line 735:


{{#ifeq: {{#var:HARDWARE_NAME}} | VAR-SOM-MX8M-NANO | |
{{#ifeq: {{#var:HARDWARE_NAME}} | VAR-SOM-MX8M-NANO | |
{{#vardefine:PLAYBACK_VIDEO_CMD1 | {{#varexists:USE_KMSSINK | gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 video-sink="imxvideoconvert_g2d ! queue ! kmssink can-scale=false" audio-sink="alsasink" | gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 }} }}
{{#vardefine:PLAYBACK_VIDEO_CMD2 | {{#varexists:USE_KMSSINK | gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! imxvideoconvert_g2d ! video/x-raw,width=800,height=480 ! kmssink can-scale=false | gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink }} }}
<br>
<br>
Example of a video file playback using the playbin element (making use of the VPU decoder):
Example of a video file playback using the playbin element (making use of the VPU decoder):
  # gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4
  # {{#var:PLAYBACK_VIDEO_CMD1}}


Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
  # gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink
  # {{#var:PLAYBACK_VIDEO_CMD2}}
}}
}}


= RTP Video streaming =
= RTP Video streaming =
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
{{GSTREAMER_RTP}}
# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223


= References =
= References =
* The GStreamer [https://gstreamer.freedesktop.org/ website]
{{GSTREAMER_REF}}
* NXP BSP Linux Users Guide, Multimedia section [https://www.nxp.com/webapp/Download?colCode=L4.14.78_1.0.0_LINUX_DOCS download link ]

Latest revision as of 15:07, 21 March 2023

Warning: This page is designed to be used with a 'release' URL parameter.

This page is using the default release RELEASE_SUMO_V1.4_DART-MX8M-MINI.
To view this page for a specific Variscite SoM and software release, please follow these steps:

  1. Visit variwiki.com
  2. Select your SoM
  3. Select the software release
DART-MX8M-MINI Streaming Video

Introduction to GStreamer

GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.

Major GStreamer commands

There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.

gst-inspect

Lists all available elements: 

# gst-inspect-1.0

Lists all available sink elements: 

# gst-inspect-1.0 | grep sink

For example: 

# gst-inspect-1.0 | grep sink
alsa:  alsasink: Audio sink (ALSA)
waylandsink:  waylandsink: wayland video sink
shm:  shmsink: Shared Memory Sink
decklink:  decklinkaudiosink: Decklink Audio Sink
decklink:  decklinkvideosink: Decklink Video Sink
ximagesink:  ximagesink: Video sink
soup:  souphttpclientsink: HTTP client sink
autodetect:  autovideosink: Auto video sink
autodetect:  autoaudiosink: Auto audio sink
gdkpixbuf:  gdkpixbufsink: GdkPixbuf sink
playback:  playsink: Player Sink
video4linux2:  v4l2sink: Video (video4linux2) Sink
fbdevsink:  fbdevsink: fbdev video sink
tcp:  tcpclientsink: TCP client sink
tcp:  tcpserversink: TCP server sink
…

Detailed help information on a specific element: 

# gst-inspect-1.0 <name of the element>

For example, the output for waylandsink: 

# gst-inspect-1.0 waylandsink
Factory Details:
  Rank                     primary + 2 (258)
  Long-name                wayland video sink
  Klass                    Sink/Video
  Description              Output to wayland surface
  Author                   Sreerenj Balachandran <sreerenj.balachandran@intel.com>, George Kiagiadakis <george.kiagiadakis@collabora.com>

Plugin Details:
  Name                     waylandsink
  Description              Wayland Video Sink
  Filename                 /usr/lib/gstreamer-1.0/libgstwaylandsink.so
  Version                  1.14.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2018-03-19
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSink
                         +----GstVideoSink
                               +----GstWaylandSink

Implemented Interfaces:
  GstVideoOverlay
  GstWaylandVideo

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                           (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:DMABuf)
                 format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
                           (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
                           (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
                           (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "waylandsink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  sync                : Sync on the clock
                        flags: readable, writable
                        Boolean. Default: true
  max-lateness        : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
                        flags: readable, writable
                        Integer64. Range: -1 - 9223372036854775807 Default: 20000000 
  qos                 : Generate Quality-of-Service events upstream
                        flags: readable, writable
                        Boolean. Default: true
  async               : Go asynchronously to PAUSED
                        flags: readable, writable
                        Boolean. Default: true
  ts-offset           : Timestamp offset in nanoseconds
                        flags: readable, writable
                        Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0 
  enable-last-sample  : Enable the last-sample property
                        flags: readable, writable
                        Boolean. Default: true
  last-sample         : The last sample received in the sink
                        flags: readable
                        Boxed pointer of type "GstSample"
  blocksize           : Size in bytes to pull per buffer (0 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  render-delay        : Additional render delay of the sink in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  max-bitrate         : The maximum bits per second to render (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  show-preroll-frame  : Whether to render video frames during preroll
                        flags: readable, writable
                        Boolean. Default: true
  window-width        : Wayland sink preferred window width in pixel
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  window-height       : Wayland sink preferred window height in pixel
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  display             : Wayland display name to connect to, if not supplied via the GstContext
                        flags: readable, writable
                        String. Default: null
  alpha               : Wayland surface alpha value, apply custom alpha value to wayland surface
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  enable-tile         : When enabled, the sink propose VSI tile modifier to VPU
                        flags: readable, writable
                        Boolean. Default: false

The i.MX8M-MINI VPU video decoder element

# gst-inspect-1.0 vpudec
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                IMX VPU-based video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode compressed video to raw data
  Author                   Multimedia Team <shmmmw@freescale.com>

Plugin Details:
  Name                     vpu
  Description              VPU video codec
  Filename                 /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.4.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package           Freescle Gstreamer Multimedia Plugins
  Origin URL               http://www.freescale.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstVpuDec

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h265
      video/x-vp9
      video/x-vp8
      video/x-h264

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpudec0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  output-format       : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
                        flags: readable, writable
                        Enum "GstVpuDecOutputFormat" Default: 0, "auto"
                           (0): auto             - enable chroma interleave. (default)
                           (1): NV12             - NV12 format
                           (2): I420             - I420 format
                           (3): YV12             - YV12 format
                           (4): Y42B             - Y42B format
                           (5): NV16             - NV16 format
                           (6): Y444             - Y444 format
                           (7): NV24             - NV24 format
  frame-drop          : enable adaptive frame drop for smoothly playback
                        flags: readable, writable
                        Boolean. Default: true
  frame-plus          : set number of addtional frames for smoothly playback
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 16 Default: 3 
  use-vpu-memory      : use vpu allocate video frame buffer
                        flags: readable, writable
                        Boolean. Default: true

The i.MX8M-MINI VPU H.264 video encoder element

# gst-inspect-1.0 vpuenc_h264
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                IMX VPU-based AVC/H264 video encoder
  Klass                    Codec/Encoder/Video
  Description              Encode raw data to compressed video
  Author                   Multimedia Team <shmmmw@freescale.com>

Plugin Details:
  Name                     vpu
  Description              VPU video codec
  Filename                 /usr/lib/gstreamer-1.0/libgstvpu.so
  Version                  4.4.4
  License                  LGPL
  Source module            imx-gst1.0-plugin
  Binary package           Freescle Gstreamer Multimedia Plugins
  Origin URL               http://www.freescale.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoEncoder
                         +----vpuenc_h264

Implemented Interfaces:
  GstPreset

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: { (string)avc, (string)byte-stream }
              alignment: { (string)au, (string)nal }
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16,
                           (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
                  width: [ 64, 1920, 8 ]
                 height: [ 64, 1088, 8 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "vpuenc_h264-0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events from downstream
                        flags: readable, writable
                        Boolean. Default: false
  bitrate             : set bit rate in kbps (0 for automatic)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0 
  gop-size            : How many frames a group-of-picture shall contain
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 32767 Default: 30 
  quant               : set quant value: H.264(0-51) (-1 for automatic)
                        flags: readable, writable
                        Integer. Range: -1 - 51 Default: -1

Test Pattern Stream

Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern: 

# gst-launch-1.0 videotestsrc ! autovideosink

Gst1.jpg

Test pattern with specific parameters on the input element: 

# gst-launch-1.0 videotestsrc pattern=circular ! autovideosink

Gst2.jpg

Camera Loopback

Stream video from a camera to a display: 

# gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink

Video Recording

The i.MX8M-MINI can encode videos to AVC(H.264) & VP8 at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the src pad capabilities in the vpuenc_264 and vpuenc_vp8 elements details for the complete list of features supported by the i.MX8M-MINI VPU encoder.

Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file: 

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi


Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file (using the SW based jpegenc element - without using the VPU): 

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi

Video Playback

The i.MX8M-MINI can decode and playback HEVC(H.265), AVC(H.264), VP9, VP8, etc. encoded videos at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M-MINI VPU decoder.


Example of a video file playback using the playbin element (making use of the VPU decoder): 

# gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4

Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder): 

# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 typefind=true ! video/quicktime ! aiurdemux ! queue max-size-time=0 ! vpudec ! autovideosink

RTP Video streaming

Encode video stream from camera using a software based JPEG encoder and stream it over RTP: 

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! \
     jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223


References

Retrieved from "https://variwiki.com/index.php?title=DART-MX8M_GSTREAMER&oldid=22089"
Latest Modules
Support
Variscite Support Center
Contact Support
Product Longevity
Product Compliance Policy
Blog
Variscite Training
Variscite Website
This page was last edited on 21 March 2023, at 15:07.

© 2024 Variscite. All rights reserved