1 of 1

Stream Optimization for Real-Time Object Detection

Alexander Nuccitelli, Alexander Kelley, Darin Peries, Easton Havemann, James Shepherd

Dr. Ritchey

Problem Definition

Sandia National Labs has been working on a temporal frequency analysis (TFA) algorithm for real-time object detection, specifically for detecting drones. Our project seeks to optimize this algorithm on an FPGA to achieve 30 FPS on a resolution of 1280 x 720.

Why a TFA algorithm?

A TFA algorithm can be used to detect small and fast moving objects that cannot be picked up by an edge detecting algorithm like a convolutional neural network. In addition, a TFA algorithm excels at picking up objects with constant frequencies.

Why an FPGA?�A field programmable gate array (FPGA) accelerator card

had advantages over both CPU’s and GPU’s.

  • Programming directly on hardware: FPGA’s are composed of programmable gates, allowing the hardware to be optimized for specific applications.
  • Power: FPGA’s require much less power than GPU’s.

Methodology

Before improvements were made, a formal code review of the the unoptimized FPGA code was performed. Key areas which impacted performance were found.

Increasing parallelization in FFTs via unrolling

    • Since FFT’s only depend on one specific pixel, multiple pixels’ FFTs can be performed at once.

Improving memory performance

    • Serializing the steps in the TFA algorithm allowed for the removal of intermediate buffers, doubling the frame rate and allowed for building at higher resolutions.
    • Allocating each frame separately allowed for the parallelization of the framebuffer interface, improving the performance by 16 times.

Additional Features

    • Support for processing multiple video streams at once was added.
    • The ability to change the weights of different Fourier coefficients was added, allowing for tuning the algorithm to pick up different frequencies.

Outcomes

A performance analysis was conducted, comparing image size and number of video inputs in order to judge the algorithms capabilities.

Overall our optimizations were extremely successful, increasing overall performance massively. At a resolution of 360 x 640, a speed up of ~40 was found. When processing two streams at once, our algorithm saw a decrease to roughly half its original performance. This is due to the fact that the FPGA is limited to calculating only 8 pixels per clock cycle, so increasing the amount of frames to be processed will cause a decrease in performance. When increasing image size, FPS decreases inversely. This is because image size and frame time are linearly related, and FPS is the inverse of frame time.

Impact

With our improvements, the algorithm can be run in real time on larger video inputs with higher frame rates. This expands the possible use cases for the FPGA implementation of the algorithm.

    • Improvements to Process Speed and Video Size: Processing at a higher speed allows for cameras with greater FPS and higher resolutions to be used. With more frames present, the algorithm is able to analyze more frequencies of motion.
    • New use cases: The ability to change coefficient weights allows for users to tune the algorithm to best read different frequencies. The ability to process multiple inputs allows for the user to monitor multiple different areas at once, all processed on the same FPGA.

References

  1. Stubbs, et al. (2019) . Temporal Frequency Analysis: Target Isolation and Signal Optimization. U.S Department of Energy Office of Scientific and Technical Information. https://www.osti.gov/servlets/purl/1643554

​

Figure 3. Graph of Performance Relative to Image Size

Figure 1. TFA Algorithm Flowchart

Acknowledgements

Thank you very much to Aaron McMullan and Zach Khan from Sandia National Labs, and Dr. Ritchey for their help with this project.

Figure 4. Graph of Performance of Different Implementations

Engineering Analysis

After completion of the algorithm, a full analysis was performed in order to compare our optimizations to the original algorithm, as well as understand and test the full capabilities of the system.

  • A testbench was written in order to ensure correctness of the algorithm and its components.
  • User studies were conducted in order to gather feedback on the design of the video output display. In order to judge the output, blurred squares were moved across an image [Stubbes, et. al] and the algorithm was performed across these squares. Users were then asked how clearly they could see the squares in the input and output images.

​

Figure 2. Comparison of Different Frequencies

25 Hz

5 Hz

Booth Number: 465