VAR-SOM-MX6 GSTREAM FSLC: Difference between revisions

From Variscite Wiki
No edit summary
 
(41 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{PageHeader|VAR-SOM-MX6 - Streaming Video}} {{DocImage|category1=VAR-SOM-MX6|category2=Yocto}} __toc__
{{PageHeader|VAR-SOM-MX6 - Streaming Video}} {{DocImage|category1=Yocto|category2=Debian}} [[Category:VAR-SOM-MX6]] __toc__


= Introduction to GStreamer =
= Introduction to GStreamer =
[https://en.wikipedia.org/wiki/GStreamer GStreamer]  
[https://en.wikipedia.org/wiki/GStreamer GStreamer] is a powerful pipeline-based multimedia framework.<br>
<br/>GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.<br>
<br/>GStreamer allows a programmer to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing. This is not a GStreamer reference manual but an overview for using it on Variscite i.MX6 products.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX6 products.<br>
<br>This release is based on [https://github.com/Freescale/gstreamer-imx/blob/master/README.md GStreamer-I.MX] community project that allows utilizing internal I.MX VPU and GPU capabilities.
This release uses the [https://github.com/Freescale/gstreamer-imx/blob/master/README.md gstreamer-imx] set of GStreamer 1.0 plugins for Freescale's i.MX platform, which make use of the i.MX multimedia capabilities.


== Major GStreamer commands ==
== Major GStreamer commands ==
There are two major commands:<br/>
There are two major commands:<br>
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
<br/>gst-launch - allows you to create and execute GStreamer  pipelines.
<br/>gst-launch - allows you to create and execute GStreamer  pipelines.
=== gst-inspect ===
=== gst-inspect ===
Lists all available elements:
Lists all available elements:
<pre>$ gst-inspect-1.0 </pre>
$ gst-inspect-1.0
Lists all available i.MX6 specific elements:
Lists all available i.MX6 specific elements:
<pre>$ gst-inspect-1.0 | grep imx</pre>
$ gst-inspect-1.0 | grep imx
For example:
For example:
<pre>root@var-som-mx6:~# gst-inspect-1.0 | grep imx
<pre>
aiur.imx: webm: [Invalid UTF-8] 8\x91\xddq
root@var-som-mx6:~# gst-inspect-1.0 | grep imx
aiur.imxaiurdemux: Aiur universal demuxer
imxipu: imxipuvideotransform: Freescale IPU video transform
vpu.imxvpuenc_h264: VPU-based AVC/H264 video encoder
imxipuimxipuvideosink: Freescale IPU video sink
vpu.imxvpuenc_mpeg4: VPU-based MPEG4 video encoder
imxipuimxipucompositor: Freescale IPU video compositor
vpu.imxvpuenc_h263: VPU-based H263 video encoder
imxg2dimxg2dvideosink: Freescale G2D video sink
vpu.imxvpuenc_jpeg: VPU-based JPEG video encoder
imxg2dimxg2dvideotransform: Freescale G2D video transform
vpu.imxvpudec: VPU-based video decoder
imxg2dimxg2dcompositor: Freescale G2D video compositor
overlaysink.imxoverlaysink: IMX Video (video compositor) Sink
imxpxpimxpxpvideosink: Freescale PxP video sink
imxvideoconvert.imximxvideoconvert_ipu: IMX ipu Video Converter
imxpxpimxpxpvideotransform: Freescale PxP video transform
imxvideoconvert.imximxvideoconvert_g2d: IMX g2d Video Converter
imxaudioimxuniaudiodec: Freescale i.MX uniaudio decoder
imxv4l2.imximxv4l2sink: IMX Video (video4linux2) Sink
imxaudioimxmp3audioenc: Freescale i.MX MP3 encoder
imxv4l2.imximxv4l2src: IMX Video (video4linux2) Source
imxeglvivsinkimxeglvivsink: Freescale EGL video sink
beep.imx: ac3: [Invalid UTF-8] 4J\xa0t
imxvpuimxvpudec: Freescale VPU video decoder
beep.imx: 3ca: [Invalid UTF-8] 4J\xa0t
imxvpu: imxvpuenc_h263: Freescale VPU h.263 video encoder
beep.imxbeepdec: Beep universal decoder
imxvpuimxvpuenc_h264: Freescale VPU h.264 video encoder
imxcompositor.imximxcompositor_ipu: IMX ipu Video Compositor
imxvpuimxvpuenc_mpeg4: Freescale VPU MPEG-4 video encoder
imxcompositor.imximxcompositor_g2d: IMX g2d Video Compositor
imxvpuimxvpuenc_mjpeg: Freescale VPU motion JPEG video encoder
imxmp3enc.imximxmp3enc: imx mp3 audio encoder
imxv4l2videosrcimxv4l2videosrc: V4L2 CSI Video Source
</pre>
</pre>
Detailed help information on a specific element:
Detailed help information on a specific element:
<pre>$ gst-inspect-1.0 imxv4l2sink</pre>
$ gst-inspect-1.0 imxeglvivsink
For example:
For example:
<pre>root@var-som-mx6:~# gst-inspect-1.0 imxv4l2sink
<pre>
root@var-som-mx6:~# gst-inspect-1.0 imxeglvivsink
====== IMXV4L2SINK: 4.0.9 build on Aug  8 2016 12:47:08. ======
====== IMXV4L2SINK: 4.0.9 build on Aug  8 2016 12:47:08. ======
Factory Details:
Factory Details:
   Rank                    primary + 1 (257)
   Rank                    primary + 1 (257)
   Long-name                IMX Video (video4linux2) Sink
   Long-name                Freescale EGL video sink
   Klass                    Sink/Video
   Klass                    Sink/Video
   Description              Displays frames on IMX SoC video4linux2 device
   Description              Video output using the i.MX6 Vivante GPU
   Author                  Multimedia Team <shmmmw@freescale.com>
   Author                  Carlos Rafael Giani <dv@pseudoterminal.org>


Plugin Details:
Plugin Details:
   Name                    imxv4l2.imx
   Name                    imxeglvivsink
   Description              IMX SoC v4l2-based video source/sink
   Description              EGL/GLES sink using Vivante direct textures
   Filename                /usr/lib/gstreamer-1.0/libgstimxv4l2plugins.so
   Filename                /usr/lib/gstreamer-1.0/libgstimxeglvivsink.so
   Version                  4.0.9
   Version                  0.12.2
   License                  LGPL
   License                  LGPL
   Source module            imx-gst1.0-plugin
   Source module            gstreamer-imx
   Binary package          Freescle Gstreamer Multimedia Plugins
   Binary package          Unknown package release
   Origin URL              http://www.freescale.com
   Origin URL              Unknown package origin


GObject
GObject
Line 65: Line 67:
                   +----GstBaseSink
                   +----GstBaseSink
                         +----GstVideoSink
                         +----GstVideoSink
                               +----GstImxV4l2Sink
                               +----GstImxEglVivSink


Implemented Interfaces:
Implemented Interfaces:
  GstNavigation
   GstVideoOverlay
   GstVideoOverlay


Line 75: Line 78:
     Capabilities:
     Capabilities:
       video/x-raw
       video/x-raw
                 format: { RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }
                 format: { I420, YV12, YV21, NV12, NV21, UYVY, RGB16, RGBA, BGRA, RGBx, BGRx, BGR, ARGB, ABGR, xRGB, xBGR }
                  width: [ 1, 2147483647 ]
                height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition)
                format: { RGB16, BGR, RGB, RGBx, BGRx, NV12, UYVY, YUY2, Y42B, I420, YV12, TNVP, TNVF, Y444 }
                   width: [ 1, 2147483647 ]
                   width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
Line 90: Line 88:


Element Implementation:
Element Implementation:
   Has change_state() function: 0x76a0b808
   Has change_state() function: gst_imx_egl_viv_sink_change_state


Element has no clocking capabilities.
Element has no clocking capabilities.
Line 102: Line 100:
   name                : The name of the object
   name                : The name of the object
                         flags: readable, writable
                         flags: readable, writable
                         String. Default: "imxv4l2sink0"
                         String. Default: "imxeglvivsink0"
   parent              : The parent of the object
   parent              : The parent of the object
                         flags: readable, writable
                         flags: readable, writable
Line 142: Line 140:
                         flags: readable, writable
                         flags: readable, writable
                         Boolean. Default: true
                         Boolean. Default: true
   device              : Device location
   fullscreen          : Whether or not to set the created window to fullscreen mode (ignored if application provides a window handle)
                         flags: readable, writable
                         flags: readable, writable
                         String. Default: "/dev/video17"
                         Boolean. Default: false
   overlay-top        : The topmost (y) coordinate of the video overlay; top left corner of screen is 0,0
  force-aspect-ratio  : When enabled, scaling will respect original aspect ratio
                        flags: readable, writable
                        Boolean. Default: true
   native-display      : String identifying the display to use (default value uses the default display)
                        flags: readable, writable
                        String. Default: null
  window-x-coord      : X coordinate of the window's top left corner, in pixels
                         flags: readable, writable
                         flags: readable, writable
                         Integer. Range: -2147483648 - 2147483647 Default: 0
                         Integer. Range: -2147483648 - 2147483647 Default: 0
   overlay-left        : The leftmost (x) coordinate of the video overlay; top left corner of screen is 0,0
   window-y-coord      : Y coordinate of the window's top left corner, in pixels
                         flags: readable, writable
                         flags: readable, writable
                         Integer. Range: -2147483648 - 2147483647 Default: 0
                         Integer. Range: -2147483648 - 2147483647 Default: 0
   overlay-width       : The width of the video overlay; default is equal to screen width
   window-width       : Window width, in pixels (0 = automatically set to the video input width)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0
  overlay-height      : The height of the video overlay; default is equal to screen height
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0
  crop-top            : The topmost (y) coordinate of the video crop; top left corner of image is 0,0
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  crop-left          : The leftmost (x) coordinate of the video crop; top left corner of image is 0,0
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  crop-width          : The width of the video crop; default is equal to negotiated image width
                         flags: readable, writable
                         flags: readable, writable
                         Unsigned Integer. Range: 0 - 2147483647 Default: 0
                         Unsigned Integer. Range: 0 - 2147483647 Default: 0
   crop-height         : The height of the video crop; default is equal to negotiated image height
   window-height       : Window height, in pixels (0 = automatically set to the video input height)
                         flags: readable, writable
                         flags: readable, writable
                         Unsigned Integer. Range: 0 - 2147483647 Default: 0
                         Unsigned Integer. Range: 0 - 2147483647 Default: 0
   rotate              : The orientation degree of the video; default is 0 degree
   borderless-window   : Disable window borders, bypassing any window manager
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0
  force-aspect-ratio  : When enabled, scaling will respect original aspect ratio
                        flags: readable, writable
                        Boolean. Default: false
  deinterlace        : set deinterlace enabled; can't be configed on fly
                        flags: readable, writable
                        Boolean. Default: true
   motion              : The interlace motion setting: 0 - low motion, 1 - medium motion, 2 - high motion.
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2 Default: 2
  reconfig            : Change V4L2 configuration while running; overlay position/size/rotation changed.
                         flags: readable, writable
                         flags: readable, writable
                         Boolean. Default: false
                         Boolean. Default: false
  composition-meta-enable: Enable overlay composition meta processing
                 
                        flags: readable, writable
                        Boolean. Default: false                                 
</pre>
</pre>


=== gst-launch ===
=== gst-launch ===
For Example launch the camera and display on the monitor:
For Example, display a test pattern on the monitor:
<pre>$ gst-launch-1.0 -v videotestsrc ! imxv4l2sink</pre>
$ export DISPLAY=:0
$ gst-launch-1.0 -v videotestsrc ! imxeglvivsink


=== i.MX6 major elements ===
=== i.MX6 major elements ===
Line 200: Line 179:
! scope="col" | description<br/>
! scope="col" | description<br/>
|-
|-
| imxmp3enc.imx
| imxaudio
| imxmp3enc
| imxmp3audioenc
| imx mp3 audio encoder
| imx mp3 audio encoder
|-
|-
| imxv4l2.imx
| imxg2d
| imxv4l2sink
| imxg2dvideosink
| IMX Video (video4linux2) Sink
| IMX G2D video sink ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/blitter-architecture.md blitter-based], [https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
| imxipu
| imxipuvideosink
| IMX IPU video sink ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/blitter-architecture.md blitter-based], [https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
| imxpxp
| imxpxpvideosink
| IMX PxP video sink ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/blitter-architecture.md blitter-based], [https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
| imxeglvivsink
| imxeglvivsink
| IMX EGL video sink ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| overlaysink.imx
| imxv4l2videosrc
| overlaysink
| imxv4l2videosrc
| IMX Video (video compositor) Sink
| IMX V4L2 CSI Video Source ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| imxv4l2.imx
| imxvpu
| imxv4l2src
| imxvpudec
| IMX Video (video4linux2) Source
| IMX VPU video decoder ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| vpu.imx
| imxvpu
| vpudec
| imxvpuenc_h263
| VPU-based video decoder
| VPU-based h.263 video encoder ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| vpu.imx
| imxvpu
| vpuenc_h264
| imxvpuenc_h264
| VPU-based AVC/H264 video encoder
| VPU-based H264 video encoder ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| vpu.imx
| vpuenc_h263
| VPU-based H263 video encoder
|-
|-
| imxvpu
| imxvpuenc_mpeg4
| VPU-based MPEG-4 video encoder ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| vpu.imx
| vpuenc_mpeg4
| VPU-based MPEG4 video encoder
|-
|-
| imxvpu
| imxvpuenc_jpeg
| VPU-based JPEG video encoder ([https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md zerocopy])
|-
|-
| vpu.imx
| vpuenc_jpeg
| VPU-based JPEG video encoder
|-
|-
| imxipu
| imxipuvideotransform
| IMX IPU video transform
|-
|-
| imxvideoconvert.imx
| imxvideoconvert_ipu
| IMX ipu Video Converter
|-
|-
| imxpxp
| imxpxpvideotransform
| IMX PXP video transform
|-
|-
| imxvideoconvert.imx
| imxvideoconvert_g2d
| IMX g2d Video Converter
|-
|-
| imxg2d
| imxg2dvideotransform
| IMX G2D video transform
|-
|-
| imxcompositor.imx
| imxcompositor_ipu
| IMX ipu Video Compositor
|-
|-
| imxipu
| imxipucompositor
| IMX IPU video compositor
|-
|-
| imxcompositor.imx
|-
| imxcompositor_g2d
| imxg2d
| IMX g2d Video Compositor
| imxg2dcompositor
| IMX G2D video compositor
|-
|-
|}
|}
{{Note| Only imxeglvivsink will work with X11 or Wayland. More optimized blitter-base PXP, IPU and G2D sinks will work with fslc_framebuffer only}}
{{Note| See more detailed description [https://github.com/Freescale/gstreamer-imx/blob/master/README.md#available-plugins here]}}


= Test Pattern Stream =
= Test Pattern Stream =
Test pattern allows you to display pre build image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Test pattern allows you to display pre build image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.<br>
<br/>Default test pattern:
Default test pattern:
<pre>$ gst-launch-1.0 -v videotestsrc ! imxv4l2sink</pre>
$ export DISPLAY=:0
 
$ gst-launch-1.0 videotestsrc ! imxeglvivsink borderless-window="true" force-aspect-ratio="false"
 
[[File:gst1.jpg]]
[[File:gst1.jpg]]
<br/><br/>Test pattern with specific paramters on the input element
<br>
<pre>$ gst-launch-1.0 videotestsrc pattern=circular ! imxv4l2sink</pre>
<br>
Test pattern with specific paramters on the input element
$ gst-launch-1.0 videotestsrc pattern=circular ! imxeglvivsink borderless-window="true" force-aspect-ratio="false"
[[File:gst2.jpg]]
[[File:gst2.jpg]]
<br/><br/>Test pattern with specific paramters on the output element
<br>
<pre>$ gst-launch-1.0 videotestsrc ! imxv4l2sink overlay-width=1280 overlay-height=720</pre>
<br>
Test pattern with specific paramters on the output element
$ gst-launch-1.0 videotestsrc ! imxeglvivsink window-width="1280" window-height="720" borderless-window="true" force-aspect-ratio="false"
[[File:gst4.jpg]]
[[File:gst4.jpg]]
<br/><br/>Test pattern with specific paramters on the input and output elements
<br>
<pre>$ gst-launch-1.0 videotestsrc pattern=circular ! imxv4l2sink overlay-width=1280 overlay-height=720</pre>
<br>
Test pattern with specific paramters on the input and output elements
$ gst-launch-1.0 videotestsrc pattern=circular ! imxeglvivsink window-width="1280" window-height="720" borderless-window="true" force-aspect-ratio="false"
[[File:gst3.jpg]]
[[File:gst3.jpg]]


= Camera Loopback  =
= Camera Loopback  =
Stream video from a camera to a display</br>
* Stream video from a camera to a display:<br>
Command:</br>
$ export DISPLAY=:0
gst-launch-1.0 imxv4l2src ! 'video/x-raw, format=(string)$FORMAT, width=$WIDTH, height=$HEIGHT, framerate=(fraction)30/1' ! imxv4l2sink</br>
For example:
<pre>$ gst-launch-1.0 imxv4l2src device=/dev/video0 ! 'video/x-raw,format=(string)I420,width=1920,height=1080,framerate=(fraction)30/1'! imxv4l2sink overlay-width=1920 overlay-height=1080</pre>


This is a 3 elements pipe,the element in the middle is a GStreamer capsfilter element.</br>
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 ! imxeglvivsink  window-width="1920" window-height="1080" borderless-window="true" force-aspect-ratio="false"
Parameter comments:</br>
{{Note| Only imxeglvivsink will work with X11 or Wayland with some rendering artifacts.<br> More optimized blitter-base '''imxg2dvideosink''' and '''imxipuvideosink''' sinks will properly work with fslc_framebuffer DISTRO only}}
:• Get the camera support format and resolution using gst-inspect-1.0 imxv4l2src.</br>
:• Set capsfilter according to the camera's supported capabilities if the user needs other format or resolution.</br>
:• Ensure that the right caps filter has been set, which also needs to be supported by imxv4l2sink.</br>


ov5640 example resolutions:
Get the camera support format and resolution using gst-inspect-1.0 imxv4l2videosrc.<br>
:width=320,height=240
'''imx-capture-mode''' for ov5640 example resolutions:
:width=640,height=480
<pre>
:width=720,height=480
ov5640_mode_VGA_640_480 = 0,
:width=720,height=576
ov5640_mode_QVGA_320_240 = 1,
:width=1280,height=720
ov5640_mode_NTSC_720_480 = 2,
:width=1920,height=1080 </br>
ov5640_mode_PAL_720_576 = 3,
ov5640_mode_720P_1280_720 = 4,
ov5640_mode_1080P_1920_1080 = 5
</pre>
 
* And another pipeline example, the same as above, only with camera continuous auto focus enabled:<br>
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 ''focus-mode=6'' ! imxeglvivsink  window-width="1920" window-height="1080" borderless-window="true" force-aspect-ratio="false"


= Video Record =  
= Video Record =  
command:</br>
command:</br>
gst-launch-1.0 imxv4l2src device=$DEVICE num-buffers=300 ! $INPUT_CAPS ! queue ! vpuenc_XXX ! $MUXER ! filesink location=output.$EXTENSION </br>
gst-launch-1.0 imxv4l2videosrc device=$DEVICE num-buffers=300 imx-capture-mode=$MODE ! queue ! imxvpuenc_XXX ! $MUXER ! filesink location=output.$EXTENSION </br>


Parameter comments:
Parameter comments:
:• $DEVICE could be set to /dev/video, /dev/video0 or according to the system video input device.
:• $DEVICE could be set to /dev/video, /dev/video0 or according to the system video input device.
:• num-buffers - Number of buffers to output before sending EOS. deafult =-1 (unlimited)
:• num-buffers - Number of buffers to output before sending EOS. deafult =-1 (unlimited)
:• INPUT_CAPS should be set according to camera format and resolution  
:• $MODE should be set according to camera format and resolution  
:• vpuenc_XXX can be vpuenc_mpeg4,vpuenc_h263, vpuenc_h264, and vpuenc_jpeg
:• imxvpuenc_XXX can be imxvpuenc_mpeg4,imxvpuenc_h263, imxvpuenc_h264, and imxvpuenc_jpeg
:• MUXER can be set as to qtmux, matroskamux, mp4mux, avimux, or flvmux  
:• MUXER can be set as to qtmux, matroskamux, mp4mux, avimux, or flvmux  
:• EXTENSION is filename extension according to the muxer type.
:• EXTENSION is filename extension according to the muxer type.
</br>
</br>
Record video from a camera into a file.</br>
Record video from a camera into a file.
Encode it to h264 at a bitrate of 10mbit/s (CBR) </br>
720p:
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=4 fps-n=30 num-buffers=300 ! queue ! imxvpuenc_h264 bittrate=10000 ! avimux ! filesink location=test.avi
 
1080p:
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 num-buffers=300 ! queue ! imxvpuenc_h264 bitrate=10000 ! avimux ! filesink location=test.avi
 
To use ov5640 continuous auto focus while recording video to a file append '''focus-mode=6''' to imxv4l2videosrc<br>
For example
 
720p:
720p:
<pre>$ gst-launch-1.0 imxv4l2src device=/dev/video0 num-buffers=300 ! 'video/x-raw,format=(string)I420,width=1280,height=720,framerate=(fraction)30/1' ! queue ! vpuenc_h264 ! avimux ! filesink location=test.avi
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=4 fps-n=30 num-buffers=300 focus-mode=6 ! queue ! imxvpuenc_h264 bittrate=10000 ! avimux ! filesink location=test.avi
</pre>
 
1080p:
1080p:
<pre>$ gst-launch-1.0 imxv4l2src device=/dev/video0 num-buffers=300 ! 'video/x-raw,format=(string)I420,width=1920,height=1080,framerate=(fraction)30/1' ! queue ! vpuenc_h264 ! avimux ! filesink location=test.avi</pre>
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 num-buffers=300 focus-mode=6 ! queue ! imxvpuenc_h264 bitrate=10000 ! avimux ! filesink location=test.avi


= Simple Movie Play =  
= Simple Movie Play =  
<pre>$ gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4  
$ export DISPLAY=:0
$ gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4  
 
= Compositing =
 
The compositor is a new feature in gstreamer-imx 0.11.0. Just like with the compositor from gst-plugins-base 1.5.1 and newer, compositor elements support an arbitrary number of request sink pads, and one srcpad.
 
<pre>
$ gst-launch-1.0  \
  imxg2dcompositor name=c background-color=0x223344 \
      sink_0::xpos=0 sink_0::ypos=90 sink_0::width=160 sink_0::height=110 sink_0::zorder=55 sink_0::fill_color=0xff00ff00 sink_0::alpha=0.39 sink_0::rotation=0 \
      sink_1::xpos=0 sink_1::ypos=20 sink_1::width=620 sink_1::height=380 sink_1::fill_color=0x44441133 ! \
    queue2 ! "video/x-raw, width=800, height=600" ! imxg2dvideosink \
  videotestsrc pattern=0 ! "video/x-raw, framerate=30/1" ! c.sink_0 \
  videotestsrc pattern=18 ! "video/x-raw, framerate=30/1" ! c.sink_1
</pre>
</pre>
{{Note|See detailed description [https://github.com/Freescale/gstreamer-imx/blob/master/README.md#compositing here]}}
= References =
* The GStreamer [https://gstreamer.freedesktop.org/ website]
* The gstreamer-imx [https://github.com/Freescale/gstreamer-imx/blob/master/README.md README]
* The gstreamer-imx [https://github.com/Freescale/gstreamer-imx/blob/master/docs/faq.md FAQ]
* A Deep Dive into Image Processing for i.MX 6 Application Processors [http://cache.freescale.com/files/training/doc/ftf/2014/FTF-CON-F0119.pdf presentation]

Latest revision as of 00:31, 17 November 2023

VAR-SOM-MX6 - Streaming Video

Introduction to GStreamer

GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX6 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for Freescale's i.MX platform, which make use of the i.MX multimedia capabilities.

Major GStreamer commands

There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.

gst-inspect

Lists all available elements:

$ gst-inspect-1.0

Lists all available i.MX6 specific elements:

$ gst-inspect-1.0 | grep imx

For example:

root@var-som-mx6:~# gst-inspect-1.0 | grep imx
imxipu:  imxipuvideotransform: Freescale IPU video transform
imxipu:  imxipuvideosink: Freescale IPU video sink
imxipu:  imxipucompositor: Freescale IPU video compositor
imxg2d:  imxg2dvideosink: Freescale G2D video sink
imxg2d:  imxg2dvideotransform: Freescale G2D video transform
imxg2d:  imxg2dcompositor: Freescale G2D video compositor
imxpxp:  imxpxpvideosink: Freescale PxP video sink
imxpxp:  imxpxpvideotransform: Freescale PxP video transform
imxaudio:  imxuniaudiodec: Freescale i.MX uniaudio decoder
imxaudio:  imxmp3audioenc: Freescale i.MX MP3 encoder
imxeglvivsink:  imxeglvivsink: Freescale EGL video sink
imxvpu:  imxvpudec: Freescale VPU video decoder
imxvpu:  imxvpuenc_h263: Freescale VPU h.263 video encoder
imxvpu:  imxvpuenc_h264: Freescale VPU h.264 video encoder
imxvpu:  imxvpuenc_mpeg4: Freescale VPU MPEG-4 video encoder
imxvpu:  imxvpuenc_mjpeg: Freescale VPU motion JPEG video encoder
imxv4l2videosrc:  imxv4l2videosrc: V4L2 CSI Video Source

Detailed help information on a specific element:

$ gst-inspect-1.0 imxeglvivsink

For example:

root@var-som-mx6:~# gst-inspect-1.0 imxeglvivsink
====== IMXV4L2SINK: 4.0.9 build on Aug  8 2016 12:47:08. ======
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                Freescale EGL video sink
  Klass                    Sink/Video
  Description              Video output using the i.MX6 Vivante GPU
  Author                   Carlos Rafael Giani <dv@pseudoterminal.org>

Plugin Details:
  Name                     imxeglvivsink
  Description              EGL/GLES sink using Vivante direct textures
  Filename                 /usr/lib/gstreamer-1.0/libgstimxeglvivsink.so
  Version                  0.12.2
  License                  LGPL
  Source module            gstreamer-imx
  Binary package           Unknown package release
  Origin URL               Unknown package origin

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSink
                         +----GstVideoSink
                               +----GstImxEglVivSink

Implemented Interfaces:
  GstNavigation
  GstVideoOverlay

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { I420, YV12, YV21, NV12, NV21, UYVY, RGB16, RGBA, BGRA, RGBx, BGRx, BGR, ARGB, ABGR, xRGB, xBGR }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_imx_egl_viv_sink_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "imxeglvivsink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  sync                : Sync on the clock
                        flags: readable, writable
                        Boolean. Default: true
  max-lateness        : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
                        flags: readable, writable
                        Integer64. Range: -1 - 9223372036854775807 Default: 20000000
  qos                 : Generate Quality-of-Service events upstream
                        flags: readable, writable
                        Boolean. Default: true
  async               : Go asynchronously to PAUSED
                        flags: readable, writable
                        Boolean. Default: true
  ts-offset           : Timestamp offset in nanoseconds
                        flags: readable, writable
                        Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
  enable-last-sample  : Enable the last-sample property
                        flags: readable, writable
                        Boolean. Default: true
  last-sample         : The last sample received in the sink
                        flags: readable
                        Boxed pointer of type "GstSample"
  blocksize           : Size in bytes to pull per buffer (0 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096
  render-delay        : Additional render delay of the sink in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  max-bitrate         : The maximum bits per second to render (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  show-preroll-frame  : Whether to render video frames during preroll
                        flags: readable, writable
                        Boolean. Default: true
  fullscreen          : Whether or not to set the created window to fullscreen mode (ignored if application provides a window handle)
                        flags: readable, writable
                        Boolean. Default: false
  force-aspect-ratio  : When enabled, scaling will respect original aspect ratio
                        flags: readable, writable
                        Boolean. Default: true
  native-display      : String identifying the display to use (default value uses the default display)
                        flags: readable, writable
                        String. Default: null
  window-x-coord      : X coordinate of the window's top left corner, in pixels
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0
  window-y-coord      : Y coordinate of the window's top left corner, in pixels
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0
  window-width        : Window width, in pixels (0 = automatically set to the video input width)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  window-height       : Window height, in pixels (0 = automatically set to the video input height)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  borderless-window   : Disable window borders, bypassing any window manager
                        flags: readable, writable
                        Boolean. Default: false
                   

gst-launch

For Example, display a test pattern on the monitor:

$ export DISPLAY=:0
$ gst-launch-1.0 -v videotestsrc ! imxeglvivsink

i.MX6 major elements

Prefix
Element Name
description
imxaudio imxmp3audioenc imx mp3 audio encoder
imxg2d imxg2dvideosink IMX G2D video sink (blitter-based, zerocopy)
imxipu imxipuvideosink IMX IPU video sink (blitter-based, zerocopy)
imxpxp imxpxpvideosink IMX PxP video sink (blitter-based, zerocopy)
imxeglvivsink imxeglvivsink IMX EGL video sink (zerocopy)
imxv4l2videosrc imxv4l2videosrc IMX V4L2 CSI Video Source (zerocopy)
imxvpu imxvpudec IMX VPU video decoder (zerocopy)
imxvpu imxvpuenc_h263 VPU-based h.263 video encoder (zerocopy)
imxvpu imxvpuenc_h264 VPU-based H264 video encoder (zerocopy)
imxvpu imxvpuenc_mpeg4 VPU-based MPEG-4 video encoder (zerocopy)
imxvpu imxvpuenc_jpeg VPU-based JPEG video encoder (zerocopy)
imxipu imxipuvideotransform IMX IPU video transform
imxpxp imxpxpvideotransform IMX PXP video transform
imxg2d imxg2dvideotransform IMX G2D video transform
imxipu imxipucompositor IMX IPU video compositor
imxg2d imxg2dcompositor IMX G2D video compositor
Only imxeglvivsink will work with X11 or Wayland. More optimized blitter-base PXP, IPU and G2D sinks will work with fslc_framebuffer only
See more detailed description here

Test Pattern Stream

Test pattern allows you to display pre build image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:

$ export DISPLAY=:0
$ gst-launch-1.0 videotestsrc ! imxeglvivsink borderless-window="true" force-aspect-ratio="false"

Gst1.jpg

Test pattern with specific paramters on the input element

$ gst-launch-1.0 videotestsrc pattern=circular ! imxeglvivsink borderless-window="true" force-aspect-ratio="false"

Gst2.jpg

Test pattern with specific paramters on the output element

$ gst-launch-1.0 videotestsrc ! imxeglvivsink window-width="1280" window-height="720" borderless-window="true" force-aspect-ratio="false"

Gst4.jpg

Test pattern with specific paramters on the input and output elements

$ gst-launch-1.0 videotestsrc pattern=circular ! imxeglvivsink window-width="1280" window-height="720" borderless-window="true" force-aspect-ratio="false"

Gst3.jpg

Camera Loopback

  • Stream video from a camera to a display:
$ export DISPLAY=:0
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 ! imxeglvivsink  window-width="1920" window-height="1080" borderless-window="true" force-aspect-ratio="false"
Only imxeglvivsink will work with X11 or Wayland with some rendering artifacts.
More optimized blitter-base imxg2dvideosink and imxipuvideosink sinks will properly work with fslc_framebuffer DISTRO only

Get the camera support format and resolution using gst-inspect-1.0 imxv4l2videosrc.
imx-capture-mode for ov5640 example resolutions:

ov5640_mode_VGA_640_480 = 0,
ov5640_mode_QVGA_320_240 = 1,
ov5640_mode_NTSC_720_480 = 2,
ov5640_mode_PAL_720_576 = 3,
ov5640_mode_720P_1280_720 = 4,
ov5640_mode_1080P_1920_1080 = 5 
  • And another pipeline example, the same as above, only with camera continuous auto focus enabled:
$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 focus-mode=6 ! imxeglvivsink  window-width="1920" window-height="1080" borderless-window="true" force-aspect-ratio="false"

Video Record

command:
gst-launch-1.0 imxv4l2videosrc device=$DEVICE num-buffers=300 imx-capture-mode=$MODE ! queue ! imxvpuenc_XXX ! $MUXER ! filesink location=output.$EXTENSION

Parameter comments:

• $DEVICE could be set to /dev/video, /dev/video0 or according to the system video input device.
• num-buffers - Number of buffers to output before sending EOS. deafult =-1 (unlimited)
• $MODE should be set according to camera format and resolution
• imxvpuenc_XXX can be imxvpuenc_mpeg4,imxvpuenc_h263, imxvpuenc_h264, and imxvpuenc_jpeg
• MUXER can be set as to qtmux, matroskamux, mp4mux, avimux, or flvmux
• EXTENSION is filename extension according to the muxer type.


Record video from a camera into a file. Encode it to h264 at a bitrate of 10mbit/s (CBR)
720p:

$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=4 fps-n=30 num-buffers=300 ! queue ! imxvpuenc_h264 bittrate=10000 ! avimux ! filesink location=test.avi

1080p:

$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 num-buffers=300 ! queue ! imxvpuenc_h264 bitrate=10000 ! avimux ! filesink location=test.avi

To use ov5640 continuous auto focus while recording video to a file append focus-mode=6 to imxv4l2videosrc
For example

720p:

$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=4 fps-n=30 num-buffers=300 focus-mode=6 ! queue ! imxvpuenc_h264 bittrate=10000 ! avimux ! filesink location=test.avi

1080p:

$ gst-launch-1.0 imxv4l2videosrc device=/dev/video0 imx-capture-mode=5 fps-n=30 num-buffers=300 focus-mode=6 ! queue ! imxvpuenc_h264 bitrate=10000 ! avimux ! filesink location=test.avi

Simple Movie Play

$ export DISPLAY=:0
$ gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 

Compositing

The compositor is a new feature in gstreamer-imx 0.11.0. Just like with the compositor from gst-plugins-base 1.5.1 and newer, compositor elements support an arbitrary number of request sink pads, and one srcpad.

$ gst-launch-1.0   \
  imxg2dcompositor name=c background-color=0x223344 \
      sink_0::xpos=0 sink_0::ypos=90 sink_0::width=160 sink_0::height=110 sink_0::zorder=55 sink_0::fill_color=0xff00ff00 sink_0::alpha=0.39 sink_0::rotation=0 \
      sink_1::xpos=0 sink_1::ypos=20 sink_1::width=620 sink_1::height=380 sink_1::fill_color=0x44441133 ! \
    queue2 ! "video/x-raw, width=800, height=600" ! imxg2dvideosink \
  videotestsrc pattern=0 ! "video/x-raw, framerate=30/1" ! c.sink_0 \
  videotestsrc pattern=18 ! "video/x-raw, framerate=30/1" ! c.sink_1


See detailed description here

References

  • The GStreamer website
  • The gstreamer-imx README
  • The gstreamer-imx FAQ
  • A Deep Dive into Image Processing for i.MX 6 Application Processors presentation