Lego Cobot
by Dr. Yuhan Jiang
Jetson AGX Orin (Jetson SDK 5.1.2) Install librealsense SDK with Debian packages
The docs suggest a simpler method for the latest JetPack versions.
Register the server’s public key
sudo apt-key adv --keyserver keys.gnupg.net --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE
Add the server to the list of repositories
sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo bionic main" -u
Install the SDK
sudo apt-get install librealsense2-utils
sudo apt-get install librealsense2-dev
Check installation using
realsense-viewer
Install Pycharm
There is a separate tarball for ARM64 processors.
sudo tar xzf pycharm-*.tar.gz -C /opt/
cd /opt/pycharm-2024.2.4/bin
sh pycharm.sh
On Linux, the installation directory contains the launcher shell script pycharm.sh under bin. For example, if you installed PyCharm to /opt/pycharm, you can run the script using the following command:
/opt/pycharm-2024.2.4/bin/pycharm.sh
You can create a symbolic link to the launcher script in a directory from the PATH environment variable. For example, if you want to create a link named pycharm in /usr/local/bin, run the following command:
sudo ln -s /opt/pycharm-2024.2.4/bin/pycharm.sh /usr/local/bin/pycharm
Since /usr/local/bin should be in the PATH environment variable by default, you should be able to run the pycharm command from anywhere in the shell.
pycharm
sudo pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v512 tensorflow==2.12.0+nv23.06
Determine the Lego Brick Sizes, Coordinates, and Rotation Angle
Credit: https://i.sstatic.net/LhZRi.png
https://stackoverflow.com/questions/15956124/minarearect-angles-unsure-about-the-angle-returned
min_area_rectangle = cv2.minAreaRect(coord) #It returns a Box2D structure which contains following details - ( center (x,y), (width, height), angle of rotation ). But to draw this rectangle, we need 4 corners of the rectangle.
length=min_area_rectangle[1][0]; width=min_area_rectangle[1][1]; center_x=min_area_rectangle[0][0]; center_y=min_area_rectangle[0][1]; rotation=min_area_rectangle[2]
brick_length=max(length,width);brick_width=min(length,width);brick_height=height
if length>width:
rotation=rotation+180
else:
rotation=rotation+90
WidowX 250 S
bot.arm.set_ee_cartesian_trajectory(roll=(brick_ang-90)/180*np.pi, moving_time=0.5) # pi rad =180 degree
import os
import cv2
import xml.dom.minidom
image_path="/media/.../B200 LEGO Detection Dataset/images/"
annotation_path="/media/.../B200 LEGO Detection Dataset/annotations/"
files_name = os.listdir(image_path)
font = cv2.FONT_HERSHEY_SIMPLEX
fontScale = 1
fontColor = (255,255,255)
thickness = 2
lineType = 2
for filename_ in files_name:
filename, extension= os.path.splitext(filename_)
img_path =image_path+filename+'.png'
xml_path =annotation_path+filename+'.xml'
print(img_path)
img = cv2.imread(img_path)
if img is None:
pass
dom = xml.dom.minidom.parse(xml_path)
root = dom.documentElement
objects=dom.getElementsByTagName("object")
print(filename)#objects)
i=0
for object in objects:
name = root.getElementsByTagName("name")[i]
name_data=name.childNodes[0].data
bndbox = root.getElementsByTagName('bndbox')[i]
xmin = bndbox.getElementsByTagName('xmin')[0]
ymin = bndbox.getElementsByTagName('ymin')[0]
xmax = bndbox.getElementsByTagName('xmax')[0]
ymax = bndbox.getElementsByTagName('ymax')[0]
xmin_data=xmin.childNodes[0].data
ymin_data=ymin.childNodes[0].data
xmax_data=xmax.childNodes[0].data
ymax_data=ymax.childNodes[0].data
print('Lego Part:',name_data,'@',xmin_data,'\t',ymin_data)
i= i +1
bottomLeftCornerOfText = (int(xmin_data),int(ymin_data))
cv2.putText(img,str(name_data),bottomLeftCornerOfText,font,fontScale,fontColor,thickness,lineType)
cv2.rectangle(img,(int(xmin_data),int(ymin_data)),(int(xmax_data),int(ymax_data)),(55,255,155),2)
#end one image
cv2.imshow('xml',img)
cv2.waitKey(10)
print("all done ====================================")
(Run on JetPack 5.x)
Install Ultralytics Package
Here we will install Ultralytics package on the Jetson with optional dependencies so that we can export the PyTorch models to other different formats. We will mainly focus on NVIDIA TensorRT exports because TensorRT will make sure we can get the maximum performance out of the Jetson devices.
Update packages list, install pip and upgrade to latest
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
Install ultralytics pip package with optional dependencies
pip install ultralytics[export]
Reboot the device
sudo reboot
Install PyTorch and Torchvision
The above ultralytics installation will install Torch and Torchvision. However, these 2 packages installed via pip are not compatible to run on Jetson platform which is based on ARM64 architecture. Therefore, we need to manually install pre-built PyTorch pip wheel and compile/ install Torchvision from source.
Uninstall currently installed PyTorch and Torchvision
pip uninstall torch torchvision
Install PyTorch 2.1.0 according to JP 5.1.2
sudo apt-get install -y libopenblas-base libopenmpi-dev
wget https://developer.download.nvidia.com/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl -O torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
pip install torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl
Install Torchvision v0.16.2 according to PyTorch v2.1.0
sudo apt install -y libjpeg-dev zlib1g-dev
git clone https://github.com/pytorch/vision torchvision
cd torchvision
git checkout v0.16.2
python3 setup.py install --user
Install onnxruntime-gpu
The onnxruntime-gpu package hosted in PyPI does not have aarch64 binaries for the Jetson. So we need to manually install this package. This package is needed for some of the exports.
All different onnxruntime-gpu packages corresponding to different JetPack and Python versions are listed here. However, here we will download and install onnxruntime-gpu 1.17.0 with Python3.8 support.
wget https://nvidia.box.com/shared/static/zostg6agm00fb6t5uisw51qi6kpcuwzd.whl -O onnxruntime_gpu-1.17.0-cp38-cp38-linux_aarch64.whl
pip install onnxruntime_gpu-1.17.0-cp38-cp38-linux_aarch64.whl
onnxruntime-gpu will automatically revert back the numpy version to latest. So we need to reinstall numpy to 1.23.5 to fix an issue by executing:
pip install numpy==1.23.5
install python 3.8.10
https://www.python.org/downloads/release/python-3810/
install CUDA 12.1
https://developer.nvidia.com/cuda-12-1-0-download-archive
install Torch v2.1.0, CUDA 12.1 version
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
install Yolov11
pip install ultralytics
Training Yolov11
from ultralytics import YOLO
# Load a model
model = YOLO("yolo11n.pt") # load a pretrained model (recommended for training)
# Train the model
results = model.train(data="E:\LegoCobot\B200LEGO.yaml", epochs=10, imgsz=640,plots=True,device=[0, 1])
B200LEGO.yaml