Getting Started with

Picam360 on JetBot (Jetson Nano)

Revision: 1.1

Last Modified: 2020-06-16

Introduction        1

Precautions        1

Prepare for Setup        1

Camera Holder Printing        3

Firmware Installation        3

Software Setup        4

Camera Calibration        11

Jupyter Notebook        16

Introduction

This document describes how to install JetBot official firmware provided by NVIDIA on Jetson Nano and set up Picam360 on top of it.

Precautions

Functional Limitation

The Picam360 software for the Jetson Nano is available without a license, but the following features are limited when used without a license

• The frame rate is limited to 5FPS

If you would like to purchase a license, please contact us using the Inquiry Form.

Card-type WiFi antenna is required

JetBot's official wiki describes two types of WiFi antennas: the USB dongle type and the M2 card type; the Picam360 sends image data to the Jetson Nano via a USB connection. When using a USB dongle type antenna with two Picam360's connected, the USB communication limit of the Jetson Nano is exceeded and it does not work properly. Therefore, when connecting two Picam360s, please use an M2 card type antenna. Also, there is a report on the Internet that M2 cards will not work with products other than the product number specified in the official Wiki (it is difficult to set up). It is therefore recommended that you use the product with the specified part number.

Prepare for Setup

Picam360 kit

Prepare one of the following models:

• PICAM360-4KHDR (One, or two in case of stereo 3D)

Items not included in Picam360 kit

You’ll also need:

• Jetson Nano Developer Kit + Wi-Fi antenna

*Read about the WiFi antenna to use here.

• microSD Card (64GB UHS-1 minimum)

Jetson Nano uses a microSD card as a boot device and main storage. It is important to have a card of sufficient speed and size for your project. The recommended minimum is a 64GB UHS-1 card.

• Power Supply (5V, over 2.5A)

Supplies power to the Jetson Nano. If you want to use the Jetson Nano as a stereo camera system and not as a JetBot radio control, we recommend using a 5V, 4A or higher DJ jack power supply. Using less than this may cause an error due to insufficient power when the calculation failure increases.

• PC

Used to operate Jetson Nano and Picam360.

• (For initial setup) Keyboard, Mouse, Display (HDMI or DisplayPort)

Camera Holder Printing

3D printer data of a camera holder to fix Picam360 to JetBot. Please use it if you need it.

https://github.com/picam360/picam360-jetbot/blob/master/hardware/camera_holder_stereo.stl

Firmware Installation

Install Jetbot official software. Follow the steps on the official page below to proceed:

https://github.com/NVIDIA-AI-IOT/jetbot/wiki/software-setup

This manual uses jetbot_image_v0p3p2.zip. If you use a different version, you may get errors, so we recommend that you install the same one that this manual uses, unless you have a specific reason to do so.

Software Setup

Once the Jetbot official software installation is complete, the next step is to install the Picam360 softwares.

In this page, we will explain how to use Terminal which comes with Apple's mac, and for Windows users, please use PuTTy etc.

Step1 - First

1. Open Terminal:

1. Connect to your JetBot:

2. Update ubuntu's package list:

3. In this manual, the nono editor is used to rewrite the configuration file. The nano editor can be installed by executing the following command:

4. Create a directory to download the source code for Picam360 softwares:

Step2 - Change JetBot’s settings

Power Mode

1. Change Jetson Nano power mode to 10W (MAXN):

2. Verify the Jetson Nano is in 10W (MAXN) power mode:

UDP Receive Buffer Size

1. Open the system file and add the bottom two lines. If the same item already exists, the value will be updated.

2. Press Ctrl+O, Ctrl+X to save the changes to the file

3. Reboot the device

Step3 - Download resources for JetBot

1. Download picam360-jetbot from github

Step4 - Install picam360-capture

1. Install the items required for the picam360-capture software:

2. Download picam360-capture from github:

3. Matches the versions of picam360-capture and the plugin for jetbbot

1. Check the date of the plugin file and make a note of it:

2. Switch picam360-capture to the date branch of the plugin. Change the date of the following command to the date you wrote it down and run it:

4. Copy the plugin file of picam360-capture for jetbot to picam360-capture directory and authorize to execute it. The date at the end of the source plug-in file must match the actual file. Removes the trailing date in the destination file name:

5. Copy the configuration file of picam360-capture for jetbot to picam360-capture directory:

Tips: config.json.tmp will be copied into the execution environment by make install, which is done in the steps below.

6. Build and install picam360-capture. Execute the following commands in turn. Make sure that there are no errors in each command:

The following items will be generated by the above command:

Step5 - Install picam360-server

1. Install the items required for the picam360-server software:

2. Download picam360-server from github:

3. Copy the plug-in files of picam360-server for jetbot to picam360-server directory:

4. Copy the configuration file of picam360-server for jetbot to picam360-server directory:

5. Build and install picam360-server. Execute the following commands in turn. Make sure that there are no errors in each command:

The following items will be generated by the above command:

Step6 - Set up picam360-capture

1. Navigate the directory:

2. Copy the current configuration file and make a backup so that you can get your changes back:

3. Open the file in a text editor:

4. In the “stream_mixer_defs” element, change the array containing notation of “v4l2_capture cam_num=” to the following. The following changes enable cam_num=0 and cam_num=1, making sure to put a comma between the first element and the second element.

If you want to connect only one camera, comment out cam_num=1 with #. Be sure to delete the previous comma when you do so.

5. Change "num_of_cam" to 2.

For 1 camera case, set 1.

6. At the end of the file, change "license_path" to "/home/jetbot/picam360/license.txt".

7. Press Ctrl+O, Ctrl+X to save the changes to the file

8. Place the purchased license file in the path you have set up. If you don't have a license file, set up a dummy license file. The following command creates a dummy file:

Step7 - Set up picam360-server

1. Copy the current configuration file and make a backup so that you can get your changes back:

2. Open the file in a text editor:

3. Set UUID to "wrtc_uuid".

1. Get UUID from the URL:

https://www.picam360.com/equipment/uuid_generator

2. Set UUID to "wrtc_uuid".

3. If "wrtc_enabled" is true, it will be accessible from the outside via the Internet. Change "wrtc_enabled" to false if you don't want to access it through the internet, the UUID will be the key to connect. Keep your UUID long enough so that no one else will know about it.

4. Uncomment "frame_mode" : "PICAM360MAP3D" and comment out other uncommented "frame_mode".

For 1 camera case, enable "PICAM360MAP".

5. Set 1440 to "frame_width", 720 to "frame_height".

For 1 camera case, 720 to "frame_width", 720 to "frame_height".

6. Press Ctrl+O, Ctrl+X to save the changes to the file

7. Reboot

Step8 - Confirmation of Connection

1. Open http://jetbot.local:9001/ in your browser. Make sure the image is displayed.

2. Drag the left edge of the screen to the center to bring up the menu, enable STEREO, and make sure that the two camera images are displayed.

3. If the image does not appear, please review the settings to see if they are correct.

Camera Calibration

Focus Adjustment

1. Open http://jetbot.local:9001/ in your browser, open the menu and enable STEREO.

2. Rotate the lens while shooting something suitable and fix it at the position where the focus is best.

Calibration Board Printing

1. Download the patterned file from the link below

https://github.com/picam360/picam360-jetbot/blob/master/capture/calib/checkerboard_pattern.png

2. Print the file at 100% printing scale and stick it on a flat, hard board, such as cardboard.

Lens Calibration

Descriptions

Image Circle Calibration : EXE > CENTER

Corrects the center position of the image. This correction is required for your first calibration or when the lens is physically displaced due to contact during transport, etc.

Lens Distortion Calibration: EXE > DISTORTION

Corrects image distortion caused by the lens shape. This correction corrects the distortion of the image due to the shape of the lens design. Therefore, once the correction has been made, there is basically no need to do it again. You may need this correction for your first calibration or when you change your lens to a different model.

Steps

1. Connect to your JetBot:

2. Activate "CALIB" in "frame_mode" in picam360-server.conf:

3. Press Ctrl+O, Ctrl+X to save the changes to the file

4. Restart picam360-server

5. Open http://jetbot.local:9001/ in your browser

6. Drag the left edge of the screen to the center to bring up the menu

Menu Buttons:

7. Got to MENU > JETSONCALIB

8. Select MARKER_MODE > CB

9. Select CLEAR

10. Select CAM > 0 (2nd round is CAM > 1)

11. Recognize the calibration board and register the points

1. Drag the left half of the screen to bring the view to the front and place the calibration board in front of the camera. When the board is recognized, a colored marker will appear. With that, select SET.

Press SET to display the registered points on the right side of the screen.

Tips：If it is difficult to recognize the board, it will be easier to recognize the board if it is done in a bright place or if the board is lit by a smartphone.

Tips : The previous position of the recognized marker is remembered. Therefore, even if the recognition marker is displayed and disappears, the point of the displayed position can be registered by pressing SET.

2. Repeat the point registration while changing the field of view in the left half of the screen so that the lens is roughly covered with points.

Tips：It is easier to recognize the checkerboard when it is caught near the center of the field of view on the left side of the screen.

12. Select EXE > CENTER

13. Select EXE > DISTORTION

14. Select Save

15. Restart the picam360-capture:

16. Return to step 5 and perform the same operation with the other camera (CAM : 1).

17. After performing the calibration with two cameras, enable "PICAM360MAP3D" in "frame_mode" of picam360-server.conf. For 1 camera case, enable “PICAM360MAP”.

18. Press Ctrl+O, Ctrl+X to save the changes to the file

19. Restart picam360-server

Jupyter Notebook

This section explains how to use the Jupyter Notebook to manipulate JetBot using GamePad while viewing Picam360 images.

1. Copy the Jupyter Notebook for Picam360

2. Install the required packages

3. Access http://jetbot.local:8888. Password for JupyterLab is jetbot

4. Open Notebooks > teleoperation > teleoperation-with-picam360.ipynb

5. Run the notebook while following the instructions on the notebook.

• You can operate the JetBot while watching Picam360 footage in your notebook.

• With the GamePad, you can control the JetBot and control the orientation, zoom in and out of the image.

• As always, it can be accessed at http://jetbot.local:9001/, so you can watch it on your PC or smartphone screen, or you can use a head-mounted display to experience what it's like to be on a JetBot!