An Exploration in Visual Filters
Authors: Beatriz Manrique & Trina Sriram
EE 2170 - Spring 2017
Introduction:
A Fast Fourier Transform is the mathematical method of translating a function from the time domain to the frequency domain. In doing so, it is traditionally used in observations of time-dependent phenomena. Through this algorithm, the Matlab function is traditionally used to observe the computed discrete Fourier transform (DFT). The Fourier transform decomposes the signal into the individual frequencies that make it up. When the eigenfunctions for the group of translations is expressed, it is a symmetrical function and thus, only half of the transform is expressed. In this examination of frequencies, the Fast Fourier Transform can be applied to a variety of situations in Matlab, including basic representations, sampling down and up and filtering.
Procedure:
The goals of these simulations was to acquire necessary skills for the goal of this project: to build an Instagram filter and eliminate noise/blurriness to remove the noise and improve quality of the image, create blurring filters to soften edges, enhance/diminish a particular color and create a frame around the image using superposition. In part one, we studied the characterization of a waveform through various MATLAB methods including delays, time shifts, scaling and transformations of the independent variable. This included various signals and a construction of a checkerboard. In part two, investigations into sampling up and down were conducted. There was explorations with the value of the samples, interpolation, image quality and experiments in the frequencies of FFT. In part three, The last part of this lab consisted of individual exploration into filters.
Part 1: Characterization of a Waveform
The transformation of a signal is an important concept in the design and analysis of signals and systems. Realistically, signals are transformed by mechanical and electrical systems. However, we apply mathematical operations on a function to easily demonstrate the transformation of a signal in class.Time shifts, time reversals, and time scaling are the basic and recurring examples of transforming the time variable of a signal.
Time shifts shift a signal along the time axis. Shifting to the left is an advance in time, meaning that a signal was moved to an earlier point in time. Shifting to the right is a delay in time, meaning that a signal was moved to a later point in time. If we shift an audio signal, advancing the signal would play it at a time earlier than the original version; likewise, delaying the signal would play it at a later time than the original version. Time reversals typically reverse the signal about the y axis. With audio, reversing a signals plays the signal in reverse. For example, the audio heard when reversing a video with sound is the same signal but played in the opposite direction. Time scalings speed up or slow down a signal by reducing or increasing the wavelength of the signal. An audio signal would speed up if the wavelength is reduced and slow down if the wavelength increases.
Figure 1 illustrates these transformations on a simple triangle pulse. The top left corner shows us the original pulse x(t). Next to that is triangle pulse shifted to the left, placed earlier in time, x(t+2). To the side of that, the triangle pulse is shifted to the right, placed later in time, x(t-2). The rightmost graph of the second row displays the scaling of the triangle wave, x(2t), and decreasing the wavelength. Alongside this is another scaled triangle pulse, this time by ½, x(½ t), and lengthening the wavelength. The leftmost graph of the same row shows the reversal of the triangle pulse about the y axis, x(-t). Because the triangle is symmetrical, the same result is acquired by shifting once to the left, x(t+1). The last row features scaling the signal itself instead of the independent variable of time. The rightmost graph scales the signal by 2, 2x(t), and this increases the amplitude of the pulse. Such a transformation increases the sound for an audio signal or brightness for an image. The next graph scales the signal by 1/2 , ½ x(t), and reduces the amplitude of the pulse. An audio signal would decrease in volume and an image would become darker by lowering the brightness level. Notice that the wavelength remains the same when scaling the signal by a constant. The last transformation is a combination of scaling the signal by ½ and shifting the signal to the left, ½ x(t+1).
Figure 2 continues the concept of transforming a signal, however we now see an image. An image is a 2D signals or a matrix of signals. The same transformations can apply. The top left corner shows us the original image of a cat. Next to that is the image reflected about the y axis. Placing an image in front of an image would produce the same result. To the side of that, the image is reflected about the x axis. The same effect occurs when rotating an image 180°. The rightmost image of the second row displays the scaling of the x axis. Here, the x axis is scaled by ½ because the width of the image has decreased. Alongside this, the y axis of the image is scaled ½ which decreases the height of the image. The width, x axis, of the image appears to be altered only because the image must use the same amount of space as the other images to be displayed alongside them. The leftmost graph of the same row shows the image shifted along the x axis. As a result, we lose the right side of the image. The first image of the last row shows the image shifted along the y axis and the bottom of the image is lost. The last two images illustrate scaling up and scaling down the amplitude of a signal. The first image scales up the amplitude which results in a very bright image. The last image scales down the amplitude and results in a darker image.
Figure 3a displays sin(t) in the time domain. The resulting graph is a widely used and recognizable figure. We can easily recognize that this is an odd function and a shifted cosine function because at t = 0, sin(t)=0. Figure 3b displays sin(t) in the frequency domain. However, we have two figures to analyze. The first is displays the magnitude and the second displays the phase. Both are symmetrical figures, so we can choose to only interpret the first half. Both graphs have a peak at 2 which reflects the number of pulses in a single period of sin(t), or the frequency of a a single pulse for each period. We can also determine the sampling rate from the x axis of both graphs. In this case, the sampling rate was 200.
Figure 4a also displays sin(t) in the time domain. It is not as smooth as the sin(t) in figure 3a. Figure 4b shows sin(t) in the frequency domain. The two figures are much easier to read because the sampling rate has decreased from 200 to 20, according to the x axis of both graphs in figure 4b. The decrease in the sampling rate also explains the rougher appearance of the sin(t) in figure 4a. Less samples are taken from the sin(t) function to produce our signal, so the differences between each sample data point is greater than the differences between the sample data points if the sampling rate were 200. Recognize that the sampling rate does not affect the peak in any of the frequency graphs. The peak remains at 2 because there are still two sin(t) pulses in each period. This illustrates one advantage of plotting a signal in the frequency domain: the frequency will remain the same regardless of the sampling rate and no data is lost as a result.
Taking the FFT of an image functions similarly. Taking the FFT of an image decomposes the image into cosine and sine components. For example, figure 5 displays a checkerboard image. The checkerboard image is a 2D representation of various signals. All the white squares represent the peaks of the signals that the checkerboard is composed of. Taking the FFT of the original checkerboard image results in the figure to the right, an image with one white block in the upper lefthand corner and the rest is black. If we could rotate the image in 3 dimensions, a peak would occur at the white corner. If we increase the matrix size of the checkerboard by doubling the number of squares, figure 5b, the resulting FFT is the same image zoomed out. This is due to maintaining the ratio of white squares to black squares. No change will occur in the FFT.
Taking the FFT of figure 6a, the Hadamard Matrix, results in an image resembling a QR code. If we change the original image from figure 6 to figure 6b, a zoomed in version, the FFT is also zoomed in. The hadamard matrix of figure 6b is 1/16 of the image in figure 6. The FFT of figure 6b is also 1/16 of the FFT of figure 6.
Taking the FFT of more complicated images such as our previously used cat image results in a more complicated FFT. Complex images have more pixels that the FFT must interpret and represent as sine and cosine components of the original image. The FFT of figure 7 resembles a static image. It does not resemble the original image in any way. Taking the FFT of a color image has no notable difference other than the static-like image is now in color, figure 8.
Part 2: Sampling Up and Down
Sampling is the process of taking samples from an original set of data. In regards to signal and system analysis, sampling translates to extracting the necessary amount of samples to reproduce the original signal. The Nyquist sampling rate is twice the bandwidth of the function. Sampling under the Nyquist sampling rate is known as undersampling. Undersampling a set of data means you have less data to store, but reproducing the original data set is impossible to ensure. Less storage with less accuracy. Sampling over the Nyquist sampling rate is known as oversampling. Oversampling a set of data means you have more data than necessary to reproduce the original data set. More storage than necessary to reproduce the original image.
In this course, we have undersampled and oversampled 1D signals and 2D signals, images. Oversampling 1D signals results in the original signal. Undersampling 1D signals results in a completely different signal. In a special case, you can undersample a signal and alias the signal. Aliasing a signal outputs a similar signal to the original signal with a smaller frequency. Figure 9 depicts the aliasing of periodic trigonometric signal. The original sample is in pink and the undersampled, result of aliasing signal is in blue. The blue signal intersects with the pink signal in a periodic pattern, thus the resulting signal is also a periodic signal resembling the original signal. Undersampling a 1D signal, such as audio, results in a distortion of the original signal, or noise within an audio signal.
Oversampling 2D signals, or images, results in a sharper image to a degree. Figure 10 shows sampling at the Nyquist sampling rate (top) and oversampling (bottom). Sampling at the Nyquist rate results in a periodic, very clean signal. Oversampling results in a signal very similar to the original sine signal, and thus, too much information. In 2D signals, oversampling can result in a blur of the image because of too much information that the program displaying the resulting image may or may not be able to handle.
Undersampling an image results in a different image as in the original image has been distorted. Such distortions can come in the forms of noise or if aliasing, the distortions of patterns. Figure 11 depicts aliasing an image. The original image on the left depicts vertical lines on the woman’s clothing and horizontal line on the table cloth. Undersampling the original image, on the right, distorts the lines to the point that the vertical lines have now been rotated, blurred, and distorted. The same can be said for the horizontal lines from the original image.
(Figure 11a) (Figure 11b)
To see the results of oversampling and undersampling the original image, figure 12 shows them side to side. The original image is the leftmost image. The undersampled image, to the immediate right, is blurred because of the loss of information. The oversampled image, to the far right, is actually sharper than the original image, but appears blurred because of Matlab’s limitations.
Figure 12
Taking the concept of oversampling and undersampling can be taken to a checkerboard example. Undersampling the checkerboard image will decrease the quality of the image because not enough samples were taken from the original image. Although oversampling an image results in better quality, it results in more storage than necessary to reproduce the original image.
Figure 13
Part 3: Filtering/Noise
Filtering is a device used in signal processing to remove unwanted components from a signal. There are multiple methods for filtering with partial suppression of only higher frequency signals such as in the case of low-pass filters or partial suppression of only lower frequency signals such as in the case of high-pass filters. There exists the unique situation in which a high pass filter and low pass filter are cascaded in respect to one another in the case of a bandpass or notch filter.
Mathematically, all filters are implemented through a series of relations between impedance and frequency. The transfer function that results as a result of usage of the Fast Fourier Transform indicates how large the magnitude is in terms of output. This indicates the transfer of the signal from its original form to its output form. This is also known as the filter kernel or impulse response to the output signal. As this is still a simple demonstration of a simple linear time invariant system, any operation of this system is still able to explained through convolution.
It is important to note that on MATLAB, there is no frequency limitation because there is no demonstration of a physical signal, such as analysis of a sound signal. Thus, this sets up the ideal environment to conduct
Thus, the analysis of multiple filters aids in the exploration to the end destination using a filter kernel to manipulate images in a variety of ways: blurring, sharpening, embossing, edge detection and more.
Figure 1: Low Pass Filter (high pass frequencies filtered out)
A low pass filter is also colloquially known as a blurring filter, which averages out high pass frequencies. As demonstrated in figure 1, the average low pass filter in digital signal processing just calculates the the average of the pixel and all of its eight immediate neighbors using a convolution kernel. This is done for each pixel and the result replaces the original value.
This is significant because it removes noise from an image, it works a smoothing filter. By removing background noise, the low pass filter does not directly affect the image itself but sharpens previously unseen details. Since the background details of an image often contain the highest amount of noise, this is an effective way to enhance these objects but not decrease sharpness of the background details.
Figure 2: Lowpass Filter through multiplication
When using a lowpass filter through multiplication, it has a clear relation that is related through convolution. As demonstrated in figure two above, the low pass filter varies in its relations depending on whether it’s in the frequency or time domain. Applying the low pass filter in the frequency domain takes away the high frequency signals. That means the low frequency signals are what remain.
Figure 3: Lowpass Butterworth Filter
A butterworth filter is used to cap the magnitude as flat as possible. Mathematically explained, the derivative of the function with respect to keep magnitude as flat as possible. In this case, a low-pass filter would have more resistor values than the capacitor values. This
The image shows the shape of the filter, with the dip as the normalized frequency increases. This indicates that high frequency components will be attenuated from the signal processing.
Figure 4: Output - Lowpass Butterworth Filter
Output is a filtered version of x, where low frequency dominates the graph. The domain and range of the function are scaled to appropriately demonstrate the passage of the signal. When taking the FFT of a signal, the output of the signal is the related to the input of the signal by multiplication. The low pass filter works similarly to take out the high frequency signals out of the signal.
Figure 5: Output - Highpass Butterworth Filter
In the case of a highpass butterworth filter, it uses the same function as a lowpass filter. The dip in the signal happens nearer to the lower frequency. To define that, it’s the cutoff frequency that shapes up the function. The highpass butterworth filter, as a result, filters out all low frequency signals and presents the output.
Figure 6: Output - Highpass Butterworth Filter
As a result, the output is a signal of high frequency component only. You can see that from the graph because any individual wavelength of the signal is indistinguishable. Thus, it is clear that the output signal consists of the high frequency portion alone.
Figure 7: Bandpass Butterworth Filter
Bandpass butterworth filter consists of a High Pass Butterworth filter and Low Pass Butterworth filters. Mathematically, when you convolve the two filters kernels, the Butterworth band pass filter comes into existence. Traditionally, there are two cutoff frequencies: a lowpass and highpass which combine into particular filter kernels.
Figure 8: Desired Signal
Figure 9: Output - Bandpass Butterworth Filter
The desired signal and output are close to one another because of the ability to tailor the filter kernels to one another. This includes the choice of two particular corner frequencies that can cater to the lowpass and highpass Butterworth filters that are present to get the optimized signal out of it.
Figure 10: MATLAB built-in filters
I don’t think we need this image
These are a list of filters that MATLAB gives us as defined on the reference pages. These are good place to start when thinking of filter design in the field of digital signal processing. These MATLAB functions are a place to start when thinking of explorations in Part 4.
Continuing on through the exploration of filtering in images, the actual process of implementing a filter function in an image is simple. Similar to any case, the filter function is the H function. When multiplying the H function by the FFT image, the output will be the FFT of the image. The inverse FFT of the output would become the filtered image itself. This includes manipulation of the FFT matrix to certain values to manually filter the image.
Figure 11: Implementation of the baseline filtering function
As described above, the implementation of the filtering function is demonstrated through the cat image. That includes the original image, taking the FFT of the image, the designated filter function on the top row. The bottom row consists of the product of the FFT and H function, the undoing of the shift and the FFT of the image was taken in return shows the filtered image in the bottom-right corner.
Figure 12: Implementations of various filters
This figure shows the implementation of various given filters from the MATLAB reference sheet. The implementation of each filter is mathematically different. The original image is provided in the top left corner. The motion filter is the approximation of the linear motion of a camera, which typically blurs the image. The Gaussian filter is a lowpass Gaussian filter, which has an impulse function that is a Gaussian function and has no overshoot to a step function input and lowers rise and fall time. The log filter is a Laplacian of Gaussian filter, which is essentially when you first convolve to smooth with a particular Gaussian kernel amount and then, suppress the noise using a Laplace operation. The Prewitt operator is a Prewitt horizontal edge-emphasizing filter, which is a gradient approximation of the image intensity function and is the simplest to implement. It is important to note that Prewitt puts emphasis on horizontal and vertical edge detection. The Sobel horizontal edge-emphasizing filter, which essentially is also a discrete differentiation operator but is more sensitive to diagonal edge detection. The average filter is an averaging filter, which is a mean filtering technique and takes the mean value of every pixel. The disk filter is a circular averaging filter that is also a mean averaging filter. It uses the same process but in the circular radius.
Figure 13: Implementation
This figure shows the implementation of an imfilter through the same process as figure 1. That means going through same process of the original image, taking the FFT of the original image, the FFT of the transfer function, multiplying the transfer function and FFT of the original image and ultimately, displaying the output image. By taking the inverse of the output function, it is a image result of displaying the FFT of the original filter.
All images have noise. Noise is the essentially the accumulation of error in an image through the image process. There are many ways that images can have noise introduced into them. Three prominent ones would be electronic transmission, mechanism for gathering the data and film grain from scanning an image. An exercise to get rid of any noise is by implementing filters. There is the implementation of linear filters, averaging filters, median filters and adaptive filters. This next part will explore how images can have their optimum state by applying a variety of filters to remove noise.
Figure 14: Exploration in different types of noise filtering
There are many different types of noise. The Gaussian noise is also known as white noise and known as normal distribution and is typically with constant mean and variance. The Localvar noise is Zero-mean Gaussian white noise with an intensity-dependent variance so it has more emphasis on the individual speck system. The Poisson noise includes a Poisson noise distribution, which would increase the probability of existing variables. The salt and pepper noise turns on and off certain pixels. The speckle noise is commonly found in radar images, satellite images and medical images and traditionally has to do with multiplicative noise.
Figure 15: FFT of different types of noise filtering
Taking the FFT is the second step in the process of understanding the process of removing noise. The FFT of the original image, Gaussian noise, Localvar noise, Poisson noise, Salt & Pepper noise and Speckle noise is demonstrated in the figure above.
Figure 16: Mathematical Representations of different types of noise filtering
The mathematical expressions on the signals themselves is demonstrated above with their respective FFTs. Observing these signals in the frequency domain demonstrates various signal processing qualities including where the peak frequency is for each filter.
Part 4: Final Exploration into “Instagram” filters
Submitted through the project proposal process, a variety of filters were chosen as part of the exploration portion of this report. The final filters that were implemented were the contrast filter, blur filter, spotlight filter and a 3D stereoscopic filter. The implementation for each filter had trials in RGB and grayscale and had a primary goal of trying to implement it for subjective usage.
Figure 1: Grayscale Contrast
The implementation of the grayscale contrast filter a series of conditional statements, implemented into the contrast command. The imadjust command was the vehicle for the various conditional statements depending on which level of contrast was chosen with R dedicated to the original image, L for low contrast and H for high contrast. The grayscale contrast function was calculated using the contrast value and the imadjust high and low values. The differentiation between high and low contrast depended on the baseline R mode, which was the original image regurgitated, if chosen. Then, if not equal, the H or L condition dictated whether there would be high or low contrast implemented. In the case of Instagram, where user input is ultimately valued, the code would be rewritten to include user input dictated subroutines that would ultimately choose between H or L functions. Mathematically, each individual pixel calculation was dependent on the condition of the loop that executed, which varied from 0 to 1 on the grayscale image array.
Figure 2: Color Contrast
The color contrast filter is similarly implemented to the grayscale contrast filter but instead of using the contrast command, the colorcontrast command was executed. The same variety of conditional statements was included to present a variety of options. The imadjust command was the vehicle for the various conditional statements depending on which level of contrast was chosen with R dedicated to the original image, L for low contrast and H for high contrast. The color contrast function was calculated using the contrast value and the imadjust high and low values. The differentiation between high and low contrast depended on the baseline R mode, which was the original image regurgitated, if chosen. Then, if not equal, the H or L condition dictated whether there would be high or low contrast implemented. In the case of Instagram, where user input is ultimately valued, the code would be rewritten to include user input dictated subroutines that would ultimately choose between H or L functions. Mathematically, each individual pixel calculation was dependent on the condition of the loop that executed, which varied from 0 to 1 on the RGB color arrays.
Figure 3: Grayscale Blur
The grayscale blur filter is a targeted filter that sharpens the focus of the image around a particular focus. This filter is equivalent to putting a particular focus on the selected circle and blurring the outsides. This is implemented in MATLAB through the blur command and changes to the image array to hold specified values. In the case of grayscale, it is the variation of values between 0 and 1 in the grayscale array. In the case of Instagram, an addition would be required to this code outside of the selected circle to specify to the user’s focus.
Figure 4: Color Blur
The color blur filter is a targeted filter that sharpens the focus of the image around a particular focus. This filter is equivalent to putting a particular focus on the selected circle and blurring the outsides. This is implemented in MATLAB through the blur command with changes to the image array to hold specified values. In the case of color, it is the variation of values between 0 and 255 in the RGB array. In the case of Instagram, an addition would be required to this code outside of the selected circle to specify to the user’s focus.
Figure 5: Grayscale Spotlight
The grayscale spotlight function uses the circular focus as the blur filter. However, it uses the circle as a means of sharpening the image inside of it. This is implemented through specification in the image array and using variations of the grayscale values of 0 and 1. The spotlight command in MATLAB along with modified array values creates the specified focus. In the case of Instagram, an addition would be required to this code outside of the selected circle to specify to the user’s focus is required to make sure that the main subject in the photo is focused on appropriately.
Figure 6: Color Spotlight
The color spotlight function uses the circular focus as the blur filter. However, it uses the circle as a means of sharpening the image inside of it. This is implemented through specification in the image array and using variations of the RGB values of 0 and 255. The spotlight command in MATLAB along with modified array values creates the specified focus. In the case of Instagram, an addition would be required to this code outside of the selected circle to specify to the user’s focus is required to make sure that the main subject in the photo is focused on appropriately.
Figure 7: 3D stereoscopic filter
As the cumulative exploration of this project, the 3D stereoscopic filter implements signal processing that is close to that of technology that exists in 3D glasses. Further exploration into this topic demonstrated that the primary RGB values for 3D glasses exist in the RB realm. Thus, first, exploration into separating RGB into distinct arrays was first conducted. However, after realizing that is impossible in the current scope of the project, exploration into how to manipulate RGB arrays lead down the path to simple array changes would produce the distinct RGB filters with red dominant, green dominant and blue dominant. In order to simulate 3D glasses, the next step was taken to include an offset of the red array values against the blue array values to see which option would appropriately solve the problem. The first output image with the offset in opposite directions for red and blue. The second output image has the opposite overlapping for both red and blue.