CSCI 3280

Introduction to Multimedia Systems

(2026 Term 2)

Computer Science & Engineering

The Chinese University of Hong Kong

Announcement

​

• Make your team of the final project (by the end of Feb. 28).

​

• The first assignment is released today (due on Feb. 6)

​

• The second tutorial will be provided on Feb. 11.

Visual Application (1)

Visual Application (2)

Visual Application (3)

Visual Application (4)

Visual Application (5)

Image Representation (1)

How to Represent Light?

• RGB? No

​

• Light spectrum: EM wave in [400nm, 700nm]

How to Represent Light (2)?

• How? Taking samples on the spectrum.

​

• [Hall89] proposed to take 9 samples on the curves.

How to Represent Color (1)?

• The diagram below tells us how can we observe a red object.

​

• But “How can we display the final light spectrum on the RGB monitor?”

How to Represent Color (2)?

• Different spectrums may produce same response in our eyes.

​

• Hence no need to reproduce the exact light spectrum. But reproduce another spectrum that gives us the “same” perceptual color.

XYZ Color Space

• CIE 1931 color spaces which define the relationship between the visible spectrum and the visual sensation of specific colors by human color vision

​

• Convert the light spectrum to XYZ

From XYZ to RGB (1)

• From XYZ to RGB: spectral curve of RGB primitives can also be expressed as,

• In other words,

From XYZ to RGB (2)

• Particularly, we have

• R_linear = 3.2404542X - 1.5371385Y - 0.4985314*Z
• G_linear = -0.9692660X + 1.8760108Y + 0.0415560*Z
• B_linear = 0.0556434X - 0.2040259Y + 1.0572252*Z

Color Model

• RGB model is one of the color models. There exists other color models, different color models are used in different domain with different purpose.

​

• YIQ and YUV are commonly used in TV broadcasting; CMY model is used in printing industry HSV (Hue, Saturation & Value) and HLS (Hue, Lightness & Saturation) are used by artists to retouch the images (e.g. Photoshop)

YIQ Model (1)

• YIQ is used in TV broadcasting.

​

• Y is the luminance (grayness or lightness) component; I and Q are the chrominance components (color components).

​

• It can be easily transformed from RGB by:

YIQ Model (2)

1

• Why YIQ for TV broadcasting?

- By ignoring the I and Q channels and only handling Y (luminance component).

- Our eye is more sensitive to luminance than chrominance components. By separating luminance from chrominance and reducing the bandwidth of chrominance channels, we can reduce the bandwidth without much noticeable artifact.

​

• NTSC adopts YIQ. The ratio of bandwidth allocated for Y:I:Q is 4:2:2

YIQ Model (3)

• Visual comparison: 5 bits per pixel (both images)

​

• Note the red and green “clouds” in the left image

YUV Color Model

• Similar to YIQ.

• This color model is adopted in CD-I (CD-Interactive) and DVI video.
• Just like YIQ, the bandwidth of chrominance components are reduced.

CYM and CYMK Models (1)

• Cyan, Magenta and Yellow (CMY) are the complementary colors of RGB. They are subtractive primaries, i.e.� C = 1.0 - R� M = 1.0 - G� Y = 1.0 - B
• Used in printing industry because color pigments absorbs color light (subtractive) instead of emitting light
• Sometimes an extra channel is added, the black K. So the CYMK model is modified to� K = min(C, M, Y)� C = C - K� M = M - K� Y = Y - K

CYM and CYMK Models (2)

1

• Apply (print) order: from light to dark Y, C, M, B

​

Digital Image (1)

1

• An image is a 2D continuous function of light intensity values.
• In computer, we can only store the discrete version of this function.
• The 2D function is sampled at discrete intervals yielding a 2D matrix of discrete values.
• A sample in the digital image is called pixel.

Digital Image (2)

• Just like digital audio, there are 3 steps:�sampling, quantization and coding.

​

• Image resolution actually specifies the sampling rate. e.g. 320 x 200, 640 x 480, 1024 x 768,

​

• Again, we cannot represent any color intensity values, only discrete color values can be represented.

​

• A clever coding scheme (color index and color lookup table) is used due to the lack of memory.

Gray Image

• For B/W image, a 256-level quantization is usually enough. Pixel values range from 0 to 255.

​

• Therefore 1 byte (8 bits) is needed to code the value of a pixel. For example, (try to find �quantization artifact on �the right)

​

• In fact, the Y component is�actually a grayscale image

Color Image

• In previous slide, the values in 1 channel (B/W image) can be finely represented by 256-level quantization.

​

• Therefore, a color image requires three channels, each with 256-level quantization, altogether (256)3 quantization levels. Or in other words, 3 bytes are needed for each pixel.

​

• Obviously, this is a huge storage requirement, especially for computer before 90’s.

​

• So, people come up with a clever solution, color indexing.

Color Lookup Table

• Instead of directly storing the pixel color in the frame buffer, the actual colors are stored in a color lookup table.
• The frame buffer only stores the color index (indexing to the color table).

Is 8-bit Really Enough? (1)

• When we say 8-bit per channel is usually sufficient. We actually mean “sufficient for representing perceived light intensity”.
• Is 8-bit really sufficient for representing the physical light intensity?
• No. Our vision is only a small moving window in the long light intensity range in real world.

Is 8-bit Really Enough? (2)

• Have you ever experienced the problem of over-exposure and under-exposure in your photographs?

• Our eyes (automatically) and cameras (manually) adjust exposure during capture
• We acquire a relative light intensity and the perceived light intensity is not linear to physical light intensity

High Dynamic Range Image

• For serious applications, we need to acquire quantity closer to physical light intensity

​

• This motivates the development of high dynamic range image. That is, 16-bit per color channel

​

• 16-bit per channel digital cameras are now available in the market but usually expensive

​

• But our monitors are still 8-bit per channel. Therefore high dynamic range images are still not popular

Image Processing - Filtering

• Image filtering is a technique that is utilized to enhance or revise the visual appearance of the image

Motivation - Noise Reduction

• We have a basic assumption for before image filtering.

Motivation - Noise Reduction

• Salt and pepper noise: random occurrences of black and white pixels
• Impulse noise: random occurrences of white pixels
• Gaussian noise: variations in intensity drawn from a Gaussian normal distribution

Image Filtering

• Idea: Use the information coming from the neighboring pixels for processing

​

• Design a transformation function of the local neighborhood at each pixel in the image

– Function specified by a “filter” or mask saying how to combine values from neighbors.

​

• Various uses of filtering:

– Enhance an image (denoise, resize, etc)

– Extract information (texture, edges, etc)

– Detect patterns (template matching)

Linear Filtering

• Filtered value is the linear combination of neighboring pixel values.

​

• Key properties:

• Can be modeled mathematically by convolution

First attempt at a solution

• Let’s replace each pixel with an average of all the values in its neighborhood

​

• Moving average in 1D:

Discrete convolution

• Simple averaging:

- every sample gets the same weight

• Convolution: same idea but with weighted average

- each sample gets its own weight (normally zero far away)

• This is all convolution is: it is a moving weighted average

Discrete filtering in 2D

• Same equation, one more index:

– now the filter is a rectangle you slide around over a grid of numbers

• Usefulness of associativity

Discrete filtering in 2D

• What values belong in the kernel H for the moving average example?

Smoothing by averaging

• What if the filter size was 5 x 5 instead of 3 x 3?

Boundary issues

• What about near the edge?

– the filter window falls off the edge of the image

– need to extrapolate

– methods:

• clip filter (black)

• wrap around

• copy edge

• reflect across edge

Gaussian filter

• What if we want nearest neighboring pixels to have the most influence on the output?
• Removes high-frequency components from the image (“low-pass filter”).

Smoothing with a Gaussian

Smoothing with a Gaussian

• Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing.

More examples

• 2D Convolution.

Signals and Images

• A signal is composed of low and high frequency components

Edge Detection

• Goal: Identify sudden changes (discontinuities) in an image

– Intuitively, most semantic and shape information from the image can be encoded in the edges

– More compact than pixels

​

• Ideal: artistʼs line drawing (but artist is also using object-level knowledge)

​

Edge Detection: Motivation

• Extract information, recognize objects

​

​

​

​

• Recover geometry and viewpoint

​

What Causes an Edge?

Characterizing Edges

• An edge is a place of rapid change in the image intensity function

​

Derivatives with Convolution

• For 2D function f(x,y), the partial derivative is:

• For discrete data, we can approximate using finite differences:

Partial Derivatives of an Image

Original Image

Gradient Magnitude Image

Thresholding Gradient with a Threshold

Designing an Edge Detector

• Criteria for a good edge detector:

– Good detection: the optimal detector should find all real edges, ignoring noise or other artifacts

– Good localization

• the edges detected must be as close as possible to the true edges

• the detector must return one point only for each true edge point

• Cues of edge detection

– Differences in color, intensity, or texture across the boundary

– Continuity and closure

– High-level knowledge

The Canny Edge Detector

The Canny Edge Detector: Recap

Various Kinds of Blurs

Image Deblurring: Motivation

• Strong demand for high quality deblurring

Image Deblurring

• Remove blur and restore a latent sharp image.

Commonly Used Blur Model

• Define as a deconvolution operation.

Blind Deconvolution

• The kernel or PSF is unknown.

Non-blind Deconvolution

• The kernel or PSF is known.

MAP based Approaches (1)

MAP based Approaches (2)

Edge Prediction based Approaches (1)

Edge Prediction based Approaches (2)

Summary

• Digital image: sampling, quantization and coding;

​

• Image filtering, convolution, edge detection;

​

• Image deblurring.