Sajjad Salem

Real-time Edge Detection on Ultra96-V2 with MIPI Camera using PYNQ and Vitis HLS

Introduction

In this tutorial, we'll delve into a fascinating project that combines hardware and software components to achieve real-time edge detection using the Ultra96-V2 board and a MIPI camera. The Ultra96-V2 is a powerful development board featuring the ZYNQ UltraScale+ MPSoC, a hybrid platform that integrates both processing system (PS) and programmable logic (PL) components. This project leverages the capabilities of the PYNQ framework, an open-source project that enables Python developers to leverage the programmable logic in Xilinx devices.

Hardware Setup

The Ultra96-V2 board serves as the backbone of our project. Its ZYNQ UltraScale+ MPSoC consists of a processing system (PS) and a programmable logic (PL) section. The MIPI camera is seamlessly integrated, capturing images with the help of the PYNQ framework. The programmable logic (PL) is engaged to grab images from the camera via its registers, providing a crisp 720p resolution for further processing.

Image Processing with Vitis HLS

To enhance the project's capabilities, a 15x15 convolutional filter for edge detection is designed using the Vitis High-Level Synthesis (HLS) tool. Vitis HLS facilitates the conversion of high-level C, C++, or OpenCL code into RTL (Register-Transfer Level) code suitable for FPGA implementation. The resulting IP core is then deployed onto the Vivado design suite, where the programmable logic (PL) applies the filter to the captured images in real time.

Data Transfer with VDMA

The VDMA (Video Direct Memory Access) IP core plays a pivotal role in efficiently moving image data from the programmable logic (PL) to the processing system (PS). Utilizing the DDR memory, VDMA ensures a smooth and rapid transfer of processed images. This seamless communication between PS and PL is orchestrated to optimize performance and maintain a high frame rate throughout the image processing chain.

Python Scripting and Socket Communication

The final piece of the puzzle involves Python scripting within the PYNQ framework. Leveraging the VDMA driver, the processed images are acquired and transmitted to the host PC using Python sockets. This seamless integration showcases the power of combining hardware acceleration with software flexibility. Achieving a remarkable frame rate of over 25 frames per second, the end result is a real-time edge detection system that showcases the full potential of the Ultra96-V2 board and the PYNQ framework.

This post provides a summarized guide for enthusiasts looking to explore the integration of MIPI cameras, FPGA acceleration, and real-time image processing using the Ultra96-V2, PYNQ, and Vitis HLS. The step-by-step breakdown illustrates the synergy between hardware and software components, offering a valuable resource for those keen on developing similar projects.

Github

You can check my python and Vitis HLS codes on this project on its GitHub repository. (click here)

Performance Video:

In this video, you can see the performance of this project. (use CC for the translation)