Testing 01

The i.MX processors family from NXP supports GStreamer as the main multimedia framework. For more information about the GStreamer Framework, please refer to the following links:

• GStreamer: open source multimedia framework
• i.MX 8 GStreamer User Guide

To get started, please be sure the camera is connected to the carrier board. See a list of camera products:

• Variscite's accessories as complementary add-ons to System-on-Modules

All the following examples were extracted from the listed documentation above, if you want to learn more, please read those documents.

Camera

Testing Camera Preview on Display

The following GStreamer pipeline example is the simplest way to open and test a camera device:

# gst-launch-1.0 v4l2src device=/dev/video<X> ! video/x-raw,width=<WIDTH>,height=<HEIGHT> ! autovideosink sync=false

There are some parameters within <> that must be replaced in the above GStreamer pipeline. The first parameter is the device ID (/dev/video0, /dev/video1, …), which can be easily found by using the following V4L2 tool that outputs the available devices:

# v4l2-ctl --list-devices

The next parameters can be found with the following GStreamer tool, which is a very useful tool to get information such as camera resolution, framerate, and supported formats:

# gst-device-monitor-1.0

Quick Examples for Different Resolutions

• VGA 480p 640x480@30fps

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480 ! autovideosink sync=false

• NTSC 720x480@30fps

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=720,height=480 ! autovideosink sync=false

• HD 720p 1280x720@30fps

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! autovideosink sync=false

• FULL HD 1080p: 1920x1080@30fps

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080 ! autovideosink sync=false

• QSXGA 2592x1944@15fps

# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=2592,height=1944 ! autovideosink sync=false

Testing Camera Snapshot

The following GStreamer pipeline example is the simplest way to capture JPEG snapshots:

# gst-launch-1.0 v4l2src device=/dev/video<X> num-buffers=1 ! video/x-raw,width=<WIDTH>,height=<HEIGHT> ! <PLUGIN> ! filesink location=/tmp/<file_name>.jpg

To fill the regular parameters within <>, use the same approach from the previous section. Replace <PLUGIN> to choose which snapshot format you want, such as JPG (jpegenc), PNG (pngenc), or JPEG (jpegenc).

Quick Examples for Snapshots (JPEG)

• VGA 480p 640x480

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw,width=640,height=480 ! jpegenc ! filesink location=/tmp/test_640x480.jpg

• NTSC 720x480

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw,width=720,height=480 ! jpegenc ! filesink location=/tmp/test_720x480.jpg

• HD 720p 1280x720

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=/tmp/test_1280x720.jpg

• FULL HD 1080p: 1920x1080

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw,width=1920,height=1080 ! jpegenc ! filesink location=/tmp/test_1920x1080.jpg

• QSXGA 2592x1944

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw,width=2592,height=1944 ! jpegenc ! filesink location=/tmp/test_2592x1944.jpg

Streaming Video Section

Test Pattern Stream

Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer:

• Default Test Pattern

# gst-launch-1.0 videotestsrc ! autovideosink

• Test Pattern with Specific Parameters on the Input Element

# gst-launch-1.0 videotestsrc pattern=circular ! autovideosink

Camera Loopback

Stream Video from a Camera to a Display

# gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink

Video Recording

Encoding and Saving a Video Stream from a Camera to an MJPEG-encoded AVI

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=/tmp/record.avi

Video Playback

Video File Playback

• Download the Big Buck Bunny video

# wget https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_1MB.mp4

• Run the video

# gst-launch-1.0 filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux name=d d.video_0 ! queue ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! autovideosink

RTP Streaming

Video Streaming from Module to VLC
On the client-side, create a file (example.sdp), then paste the following content into the file:

v=0
m=video 5000 RTP/AVP 96
c=IN IP4 10.0.0.100
a=rtpmap:96 H264/90000
a=fmtp:96 sprop-parameter-sets=Z2QAMqxyhEBQBbsBEAAAAwAQAAADA8DxgxhG\,aOhDgzLIsA\=\=

On line 3: Replace the IP with the server IP;
On line 5: Replace the sprop-parameter-sets value with the respective one. To get this value, run the pipeline below on the server-side.
Open the VLC tool:

vlc example.sdp

Then, just run the pipeline on the server-side:

gst-launch-1.0 -v filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux ! rtph264pay ! udpsink host=<PC_CLIENT_IP> port=5000

NOTE: Replace <PC_CLIENT_IP>.

Multiple Cameras

To display more the one camera at the same display output, use the following GStreamer pipeline changing accordingly. See a generic example below:

gst-launch-1.0 imxcompositor_g2d name=comp 
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=640 sink_0::height=480 
sink_1::xpos=0 sink_1::ypos=480 sink_1::width=640 sink_1::height=480 
sink_2::xpos=640 sink_2::ypos=0 sink_2::width=640 sink_2::height=480 
sink_3::xpos=640 sink_3::ypos=480 sink_3::width=640 sink_3::height=480 ! 
video/x-raw,format=RGB16 ! waylandsink 
v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480 ! comp.sink_0 
v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480 ! comp.sink_1 
v4l2src device=/dev/video2 ! video/x-raw,width=640,height=480 ! comp.sink_2 
v4l2src device=/dev/video3 ! video/x-raw,width=640,height=480 ! comp.sink_3

• VGA (640x480)

gst-launch-1.0 imxcompositor_g2d name=comp \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=640 sink_1::height=480 \
sink_2::xpos=640 sink_2::ypos=0 sink_2::width=640 sink_2::height=480 ! \
video/x-raw,format=RGB16 ! waylandsink \
v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480 ! comp.sink_1 \
v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480 ! comp.sink_2

• HD (1280x720)

gst-launch-1.0 imxcompositor_g2d name=comp \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 \
sink_2::xpos=1280 sink_2::ypos=0 sink_2::width=1280 sink_2::height=720 ! \
video/x-raw,format=RGB16 ! waylandsink \
v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! comp.sink_1 \
v4l2src device=/dev/video1 ! video/x-raw,width=1280,height=720 ! comp.sink_2

• FULL HD (1920x1080)

gst-launch-1.0 imxcompositor_g2d name=comp \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 \
sink_2::xpos=1920 sink_2::ypos=0 sink_2::width=1920 sink_2::height=1080 ! \
video/x-raw,format=RGB16 ! waylandsink \
v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080 ! comp.sink_1 \
v4l2src device=/dev/video1 ! video/x-raw,width=1920,height=1080 ! comp.sink_2

Extra Section

Video Scaling

To display different scaling results use the glimagesink - render_rectangle property.

• For VGA Resolution

# gst-launch-1.0 filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! glimagesink render-rectangle='<0, 0, 720, 480>'

• For Full HD Resolution

# gst-launch-1.0 filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! glimagesink render-rectangle='<0, 0, 1920, 1080>'

Video Rotation

To rotate the video results use the glimagesink - rotate-method property:

• To Rotate 90 Degrees

# gst-launch-1.0 filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! glimagesink rotate-method=1

• To Rotate 180 Degrees

# gst-launch-1.0 filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! glimagesink rotate-method=2

• To Rotate 270 Degrees

# gst-launch-1.0 filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! qtdemux name=d d.video_0 ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! glimagesink rotate-method=3

Debugging

This section describes some debug tool functionalities.

• Buffer Movement

# GST_DEBUG=GST_BUFFER:5 gst-launch-1.0 -v filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! decodebin ! waylandsink sync=false

As you may notice, the video performance was affected by the console log return. To avoid it, keep the debug values in a file:

• Buffer Movement

# GST_DEBUG_FILE=gst_debug.log GST_DEBUG=GST_BUFFER:5 gst-launch-1.0 -v filesrc location=/home/root/Big_Buck_Bunny_720_10s_1MB.mp4 ! decodebin ! waylandsink sync=false

Zero Copy

This section approaches the zero-copy operations using GStreamer pipelines. The zero-copy support is an operation in which the CPU uses the data produced by one element but without requiring any type of transformation. One way of using buffers to operate a zero-copy pipeline is to use properties such as io-mode. This process can be very helpful in order to improve the execution speed of a video processing pipeline.

• Dmabuf

The dmabuf uses buffers of a hardware DMA in order to perform a zero-copy pipeline, as shown below:

# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 io-mode=dmabuf ! \
  'video/x-raw,format=(string)NV12,width=1920,height=1080,framerate=(fraction)30/1' ! \
  queue ! v4l2h264enc output-io-mode=dmabuf-import ! avimux ! filesink location=/tmp/test_zerocopy.avi

In this pipeline, the RAW format is stored by the io-mode property and then used further on the H.264 format encode process.