The objective of this capstone project is to use a Jackal Robot to convert real-world environments into virtual worlds through object detection and rendering techniques. Deep learning algorithms such as Yolo, PlFuHD, and posenet-model are being implemented to accurately recognize and classify objects in the environment. The system is rendering a person from an image in a virtual space based on data gathered by the robot's sensors. The project aims to evaluate the accuracy and efficiency of the system and its potential applications in various fields. Although the project had to be ended prematurely due to a disaster, the system successfully detects and converts persons and chairs to virtual representations. Future work would involve implementing style-gan to handle texture in 3D models. The project seeks to advance the fields of robotics and virtual reality by providing the Jackal Robot with the capability to convert reality into virtuality.
Research Question
Related Work
Dataset
Dataset that is captured from jackal robot of 1640 rgb imags, 391 rgb pointcloud data, 10,206 velodyne point cloud, 1628 depth image dataset of each category shown on right hand side.
Methodology
Conclusions & Future Work
Jackal Robot: Converting Reality to Virtuality through Surrounding Rendering and Object Detection
Nishank Singhal (Capstone project)
Mentor: Dr. Christelle Scharff
Pace University, Seidenberg School of CSIS
References
[1] Couturier, R., Noura, H. N., Salman, O., & Sider, A. (2021). A Deep Learning Object Detection Method for an Efficient Clusters Initialization. arXiv preprint arXiv:2104.13634v3.
[2] Toshev, A., & Szegedy, C. (2014). DeepPose: Human Pose Estimation via Deep Neural Networks. arXiv preprint arXiv:1312.4659v3.
[3] Saito, S., Simon, T., Saragih, J., & Joo, H. (2020). PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. arXiv preprint arXiv:2004.00452v1.
[4] Brazil, G., Kumar, A., Straub, J., Ravi, N., Johnson, J., & Gkioxari, G. (2022). Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild. arXiv preprint arXiv:2207.10660v2.
The accuracy of the YOLOv5 object detection model, PoseNet pose estimation model, and 3D PIFuHD model for generating 3D models of people can vary depending on several factors such as the training data, model architecture, and hyperparameters used. YOLOv5 is a state-of-the-art object detection model that has shown high accuracy on various datasets. PoseNet is also a widely used model for pose estimation that has shown high accuracy on various datasets. The 3D PIFuHD model is a state-of-the-art model for generating high-fidelity 3D models of people from 2D images. The accuracy of these models can be high when trained on suitable datasets and with appropriate hyperparameters but can also vary depending on the quality of the input image, the complexity of the pose, and other factors.
In conclusion, the methodology described involves a sequence of steps to detect and extract a person from an RGB image and generate a high-fidelity 3D model of the person using deep learning models. The YOLOv5 model is used for object detection, PoseNet is used for pose estimation, and PIFuHD is used for 3D surface reconstruction. While the accuracy of each of these models can vary depending on various factors, they have shown to achieve high accuracy in various applications. Overall, this methodology has the potential to be used for a range of applications such as virtual reality, gaming, and animation. However, further research is needed to improve the accuracy of these models and make them more robust to various challenges such as occlusion and complex poses.
Future work in this area could focus on improving the texture recovery process for generating more realistic 3D models. This could involve developing new algorithms and techniques for accurately mapping 2D textures onto 3D models and enhancing the texture resolution to capture finer details. Additionally, further research could be done to position the generated 3D models in virtual environments accurately, accounting for lighting and shadow effects, and improving the overall realism of the scene. One potential application of this technology could be in the creation of virtual fashion try-on platforms, where customers can see how clothes fit and look on them without having to physically try them on.
Abstract
Results
The first step in the methodology involves connecting to the Jackal device, which provides access to an RGB image. This image is then processed using an object annotation algorithm, which helps to identify objects in the image. The YOLO model is then applied to the image, which uses deep learning to detect objects in real-time. In this case, the YOLO model is used to detect the presence of a person in the image.
Next, the image is passed through a PoseNet model, which is a deep learning algorithm that is used to estimate the pose of a person in an image. The PoseNet model uses a neural network to identify key points on the body, such as the elbows, knees, and wrists, and then estimates the position of these points in 3D space. This information is used to generate a 2D skeleton of the person in the image.
Finally, the isolated image of the person is passed through a PIFuHD model, which is a deep learning algorithm that generates a 3D model of the person. The PIFuHD model uses a neural network to reconstruct the 3D surface of the person based on the 2D image input. The resulting 3D model is a high-fidelity representation of the person that can be used for various applications, such as virtual reality, gaming, and animation.
Overall, the methodology involves a combination of object detection, pose estimation, background removal, and 3D modeling techniques to generate a high-fidelity 3D model of a person from a single RGB image.