Ubtech
Shenzhen, China
My placement is divided into 2 projects, both related to robot manipulation and deep learning Project 1: - Team target: finetune a System controling the robot to pick up and sort the workpiece - The system used YOLO to extract the images of workpiece from the camera of the robot. A refiner-scorer model predicted the position and rotation of the workpiece to instruct robot manipulation and a depth model prevented collisions. - This old project had been lasted for half a year and had enter the last stage, before I took over the data processing, training and testing of the YOLO segmentation model. Project 2: - Team target: use GR00T VLA controlling the robot to transport a large plastic box between shelves - The whole system, GR00T, is an end-to-end Vision-Language-Action model proposed by Nvidia. It collects the images, verbal instructions and robot joints information to control the action of a robot. It is simpler and much more robust than the system in Project 1. - This project involves a modified GR00T module that is developed by several groups with different manipulation tasks. I am currently working on model finetuning and real-world reinforcement learning
Project 1: - Wrote data processing scripts for the segmentation model, to remove damaged and repeated data, and used these scripts to process data returned from the client factory. - Improved the evaluation pipeline of the training module, allowing bad cases to be automatically exported and classified - Introduced specialised data augmentation, such as generating and pasting badcase textures, to increase the performance of model, so that the model accurately identify the workpiece when the platform is covered with dirts and shadows. - Trained a model that performs beyond the requirements (99.9% accuracy) Project 2: - Tuned model hyperparameters and broadcasted the result to the team - Experimented on some frame weighting methods to improve the performance of the model by 5-10% - Introduced an offline reinforcement learning method on the existing module; trained a specialised model to evaluate the task progress so that the system can automatically evaluate the situation. These evaluations are returned to the model during training so that it can distiguish good and bad actions. This is very useful because the robot does not only learn from telemetry operators who cannot control slight actions or make decisions based on force sensors. - Still working on the reinforcement learning and its real-world data collection - Participated in high quality data collection, as the arranger and recorder, which significantly improved the task success rate from 49% to 72% in real testing
- Data science: automated data collecting, processing, clustering, classification and augmentation - Deep learning: stucture and evaluation of segmentation model and VLA model; reinforcement learning - Machine learning utilities: cluster training; deployment of models - Robotics: Ros2, a robotic data transmission system and robot joints modelling - Software engineering: python module programming, including pytorch and transformers - Field engineering: data collections using robot
- File system managing, due to large amount of data and model weights processed and trained - Report and Experiment document writing - Technical communications within the team and report to the leader - Communications with field engineers, data labelers and telemetry operators - Gitlab managing