TODO list for Cheese:

Vidoe Noise.

This biggest problem is performance. Mostly it depend on used video and audio codec, framerate and resolution. We cant do best choice for all situations.

• Video codec

• webm  -

• Pros: It provide good quality and small file size.
• support for realtime mode, with adoptive cpu usage.
• support for variable frame rate
• Bugs:

https://bugzilla.gnome.org/show_bug.cgi?id=627971

• theora -

• Cons: ??? Nod good for really  slow hardware?
• Pros: comparable good performance and quality. Normal file size. Can do variable frame rate ( VFR work with matroska, and not jet supported on ogg)
• Bugs:

https://bugzilla.gnome.org/show_bug.cgi?id=627459

• mjpeg -

• Pros: really light codec.
• Cons: really big file size.
• camera what can provide jpeg will at 320x240 @ 30fps give 7MB

• raw format -

• for resolution 320x240 @ 30fps will give 130MB
• for resolution 320x240 @ 15fps will give 66MB

• Resolution

• It affect performance on both sides, webcam and encoder. I think i should not explain encoder part of it :D If we choice biggest resolution webcam will do minimal framerate. For example output of “uvcdynctrl -f” (046d:0991 Logitech, Inc. QuickCam Pro for Notebooks):

Pixel format: YUYV (YUV 4:2:2 (YUYV); MIME type: video/x-raw-yuv)

  Frame size: 160x120

        Frame rates: 30, 25, 20, 15, 10, 5

  Frame size: 176x144

        Frame rates: 30, 25, 20, 15, 10, 5

  Frame size: 320x240

        Frame rates: 30, 25, 20, 15, 10, 5

  Frame size: 352x288

        Frame rates: 30, 25, 20, 15, 10, 5

  Frame size: 640x480

        Frame rates: 30, 25, 20, 15, 10, 5

  Frame size: 800x600

        Frame rates: 25, 20, 15, 10, 5

  Frame size: 960x720

        Frame rates: 10, 5

  Frame size: 1600x1200

        Frame rates: 5

As you can see, biggest resolution is 1600x1200 with worst frame rate 5fps - this is the best resolution for photo. Best resolution for video will be 640x480, because this is last resolution with  30fps.

• The mess about resolution. This problem exist as long as pixel exist.  Cheese just show all resolution possibilities. IMHO it is info spam. Back to “uvcdynctrl -f” output, this time i added aspek ratio:

160x120     1.333   or   4:3

176x144     1.222   or 11:9

320x240     1.333   or   4:3

352x288     1.222   or 11:9

640x480     1.333   or   4:3

800x600     1.333   or   4:3

960x720     1.333   or   4:3

1600x1200 1.333   or   4:3

I’d say 176x144 and 352x288 is duplicate of 160x120 and 320x240 in other aspect ratio. IMHO if we haw this kind of duplicates, we should filter it out.

Comment: fargiolas do not like this idea. If you have max resolution 352x288 you’d like to get maximum of our cam.

• Other irritation.

• User will probably wont to share videos in Internet. Youtube is most probable way and there we have problem too. First of all it use 16:9 aspect ration. In most cases we will provide 4:3, it will work but looks not so nice :D . I think we can do some software trick to emulate 16:9. For example crope 640x480 (4:3) to 640x360 (16:9)

• Solution: use aspectratiocrop

gst-launch -v v4l2src ! video/x-raw-yuv, width=800, height=600, framerate=15/1 ! ffmpegcolorspace ! aspectratiocrop aspect-ratio=16/9 ! autovideosink

• Other names - we say 640x360, youtube say 360p. Probably  we can do switch “16:9 mode” and show it easy way: 240p, 360p, 480p, 720p. Or just do it by default.

• Frame rate. What we need? What we can get? And can we handle?

• [update from 21.09.2011] Important!!! Webcams use different bayer demosaicing algorithm. My webcams use simple demosaicing after 15fps, this make filling like image is upscaled from smaller resolution.
• Most of webcams tell that they provide 30 fps, in some conditions they can truly do this. Cheese use default settings and provide us what the cam say. What actually we need? DVD use 25 interlased fps. Blu-ray 24fps. Do we wont to get better quality than DVD?
• Independet of what tell webcams in normal conditions thay give less than 15fps. My expensive logitech do 15 fps and cheep build in webcam in netbook (asus 1005) do actually 5fps. Mos of you do not know it and happy with what you get. The frame rate depends on the light conditions in your room (more light more frames) and on driver (seems like logitehc driver provide more FPS than standart UVC driver.) So we have normally a really dynamic stream - framerate changes all time we capture it.
• There is small problem with variable frame rate. Not all codec or container can work with it. Only container what store time stamps of each frame can work with it (One i know is matroska). Ogg do not store time stamps but there is other way to go (see Bug 627459). With codec or container what can work only with monotone frame rate we use rate stabiliser (viderate). It keep same rate all the time, if webcam give us less frames it will generate dublicate frames to fill the holes.
• It come out be a serious performance issue. Cheese use all default options and try to get best from webcam. Webcam tell - it will give 30 fps but provide less than 15fps all the time. To stabilize the situation we use vidorate to give 30fps so codec and container are happy. In this situation 50% of CPU usage is just pointless encoding of duplicate frames. Any way, there is work in progress and probably soon will be solved.
• Wat can we handle? Lets do some performance tests (probably it will be gut suggestion to phoronix test suit):

Capture on netbook Asus eeepc 1005 (Atom N280 @1.66GHz):

$(sleep 30; pkill gst-launch)& time gst-launch-0.10 v4l2src ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! theoraenc ! oggmux ! filesink location=test.ogg

• without rate stabilisation (CPU load 17%):

real    0m30.062s

user    0m4.732s

sys    0m0.068s

• with rate stabilisation ( CPU load 67%):

real    0m30.085s

user    0m19.149s

sys    0m0.024s

Other monoton fps test:

• Vidoe Noise.

• We normally forgot about this aspect.  Webcam is really noisy source. This make encoder a bit besy to fallow the noise.  If we use noise filter we get lower CPU usage and smaller file size. Here are results:
• without filter  (size 935K):

real    0m20.281s

user    0m12.201s

sys    0m0.152s

Here is not filtered video with enabled telemetry http://videobin.org/+1o2/1xf.html

• with filter (size  529K) :

real    0m18.272s

user    0m11.061s

sys    0m0.128s

Here is filtered video with enabled telemetry http://videobin.org/+1o2/1xe.html

Take a look here to learn more about telemetry visualisation.  And here is the my bugreport and patch adding telemetry debuging to gstreamers theora:

https://bugzilla.gnome.org/show_bug.cgi?id=628488

• The filter add some CPU overhead but this is compensated by reduced encoder load. And it dramatically (2x) decrease file size.

• What is about GUI?
• TOOLS:

• check frame rate on linux:

gst-launch -v v4l2src ! ffmpegcolorspace ! fpsdisplaysink

• what you cam say it can do:

uvcdynctrl -f

• to check real frame rate on windows use stopwatch, for example this:

http://tools.arantius.com/stopwatch

record it and play frame by frame:

this is real frame:

and this is real frame:

this is frame was probably produced by codec as sum of two frames from webcam:

After you can capture real time between frames we can measure real frame rate.

• extract all frames to images:

mplayer -nosound -vo jpeg some_file.avi

Audio part of performance porblem

• Benchmarks.

• time gst-launch-0.10 pulsesrc ! audioconvert ! vorbisenc ! oggmux ! filesink location=audio_test.ogg

real    0m30.045s

user    0m2.636s

sys    0m0.640s

CPU 10%

this created one channel  vorbis audio with 44100 sample rate.

• There is no difference between one channel capture and two channel

audio/x-raw-float,channles=1 

Correction: there was a mistake in the test, so i got wrong results.

• Samplerate.

• Codec. Vorbis vs Speex

With default settings, speex use 20%.

• Pulseaudio

• Some times pulseaudio use more CPU than we need to encode audio. This is really strange. Need toi be deeply researched!!!
• For testing use:

gst-launch-0.10 pulsesrc ! audioconvert ! audio/x-raw-float,channles=1,rate=16000 ! fakesink

than list all pulse clients:

pacmd ls

at the end of list we should get some thing like this. If not search for “gst-launch” in it:

....

        index: 0

    driver: <protocol-native.c>

    flags: START_CORKED

    state: RUNNING

    source: 1 <alsa_input.pci-0000_00_1b.0.analog-stereo>

    current latency: 0,00 ms

    requested latency: 5,00 ms

    sample spec: float32le 2ch 16000Hz

    channel map: front-left,front-right

                     Stereo

    resample method: speex-float-1

    owner module: 8

    client: 12 <gst-launch-0.10>

    properties:

            media.name = "Record Stream"

            application.name = "gst-launch-0.10"

            native-protocol.peer = "UNIX socket client"

            native-protocol.version = "16"

            application.process.id = "7189"

Most cpuload is coused by resampler. Here is the cat from man page:

           resample-method= The resampling algorithm to use. Use onhje of  src-sinc-

           best-quality,   src-sinc-medium-quality,   src-sinc-fastest,  src-zero-

           order-hold, src-linear, trivial, speex-float-N, speex-fixed-N,  ffmpeg.

           See  the documentation of libsamplerate for an explanation for the dif‐

           ferent src- methods. The method trivial is  the  most  basic  algorithm

           implemented.  If  you're tight on CPU consider using this. On the other

           hand it has the worst quality of them all. The Speex resamplers take an

           integer  quality  setting in the range 0..9 (bad...good). They exist in

           two flavours: fixed and float. The former uses fixed point numbers, the

           latter relies on floating point numbers. On most desktop CPUs the float

           point resmampler is a lot faster, and it also  offers  slightly  better

           quality. See the output of dump-resample-methods for a complete list of

           all available resamplers. Defaults to  speex-float-3.  The  --resample-

           method  command  line  option  takes precedence. Note that some modules

           overwrite or allow overwriting of the resampler to use.

To adjust resample method change /etc/pulse/daemon.conf:

resample-method = speex-float-1

Bugs:

https://bugzilla.gnome.org/show_bug.cgi?id=627263

• Problems on performance measuring. Time and top are confused by multicore, multi processor or hypertreading. Here is same task with different CPU settings.

• Athom no ht (11% CPU):

real    0m30.042s

user    0m1.468s

sys    0m1.904s

Athom with ht (18% CPU)

real    0m30.040s

user    0m2.380s

sys    0m3.100s

Core 2 Duo 2 cores (6%)

real    0m30.010s

user    0m0.680s

sys    0m1.110s

Core 2 Duo 1 core (2%)

real    0m30.015s

user    0m0.260s

sys    0m0.360s