LLM-Driven Pick and Place: Updating System Prompt + Vision + Motion

Mastering Robotics Software Engineering with ROS: Develop and Deploy Advanced Manufacturing Solution

Module 1: ROS introduction and workspace setup

Introduction to ROS (8:53)
ROS architecture (17:06)
Install Ubuntu 20.4 on Virtual Box (14:39)
Install ROS Noetic (11:27)
Setup Visual Studio Code for ROS (4:08)

Module 2: ROS basics and tools (ROS)

Create ROS workspace (9:01)
The ROS master and the nodes (23:21)
Create a node in C++ (29:19)
Create a node in Python (14:28)
Create a publisher in C++ (14:31)
Create a subscriber in C++ (19:42)
Create a publisher in Python (12:48)
Create a subscriber in Python and play with Turtlesim (23:31)
Introduction on Services (18:01)
Create a service server in C++ (20:59)
Create a service client in C++ (16:28)
Create a service server in Python (11:50)
Create a service client in Python (15:07)
Create a custom msg (18:13)
Create a custom srv (4:43)
Final Project with Turtlesim (95:59)
ROS parameters (6:08)
ROS launch file (10:58)

Module 3: World Modeling and Simulation Environment (ROS & ROS2)

Github repository
Introduction to Gazebo (9:45)
How to structure your project (6:50)
Start to create your world (9:40)
Create your custom model (19:12)
How to load your world via launch file (8:57)

Module 4: Robot Modeling (ROS & ROS2)

Introduction to URDF file (7:41)
Strategy to model your custom Robot (8:20)
Start to create a custom Robot in URDF (12:48)
Visualize the Robot in Rviz (13:44)
Joint State and Robot State Publisher (10:24)
Complete you custom robot (19:59)
How to simplify your Robot model (15:05)
How to model a gripper as end effector (22:55)
How to clone a robot model from Github (33:21)
How to clone a commercial robot: Universal Robot 5 (UR5) (25:08)
How to setup a commercial Gripper: Robotiq (30:59)
Spawn your Robot in Gazebo (11:54)

Module 5: ROS Control (ROS)

Introduction to ROS Control (8:17)
How to calculate intertia values for your custom robot (22:01)
How to configure transmission for the joints of your custom robot (14:57)
How to put a Gazebo plugin (14:12)
How to create and configure controllers for your Custom Robot Joints (17:08)
How to spawn controllers of your robot joint (19:30)
How to tune PID Controllers for your Custom Robot Joints (87:26)
Forward Kinematics with Custom Robot (9:48)
How to use Transform (3:31)
Hot to grasp an object in Gazebo without slipping (26:33)
JointPositionController VS JointTrajectoryController (12:52)

Module 6: Inverse Kinematics with MoveIt and Pick and Place Application (ROS))

Introduction to MoveIt (12:08)
Create a MoveIt configuration Package (19:00)
Play with Motion Planning and update the MoveIt package (36:37)
ROS Control for MoveIt (11:09)
Create an Inverse Kinematics node (28:55)
How to setup the orientation and position of the end effector (19:30)
How to make Collision Objects in Rviz (23:37)
How to code Pick and Place (51:06)
How to adapt the inverse Kinematics node to another robot (37:30)
How to code Pick and Place with Grasp Class

Module 7: Run the Application in Gazebo (ROS)

Course check point and introduction to the module
The Structure of the Launch file (11:23)
Control your robot with MoveIt motion planning plugin (11:34)
Simulate a pick and place in Gazebo (25:03)

Module 8: Perception for Robotics with OpenCV (ROS & ROS2)

Why Computer Vision in Robotics (20:33)
Strategy to integrate perception with robotic arm (14:01)
Type of Perceptions: HSV (11:53)
Type of Perceptions: SURF (6:21)
Type of Perceptions: PCL (4:05)
How to deploy a 2D camera (23:52)
Get an Image with OpenCV (37:04)
Stream the Image with OpenCV (6:15)
Edge Detection with OpenCV (46:49)
Get the position of the object in the Image (46:03)
Convert the object position with the respect of camera_link (80:14)

Module 9: Perception for Robotics with PCL (ROS & ROS2)

Setup the working environment with UR5 (27:46)
How to deploy a 3D camera (26:56)
Write and Read Point Cloud data (41:39)
Use the Voxel filter (25:46)
Create a plane segmentation (27:48)
Apply a pass through filter (12:28)
Extract a cylinder and get it's position (41:22)

Module 10: Pick and Place using camera (ROS)

How to tackle this project (9:13)
Adapt the Computer Vision node (16:43)
Create the service to transform data from camera_link to base_link frame (61:42)
Pick and Place with 2D camera (53:54)
Pick and Place with Grasp class (36:21)
Pick and Place with UR5 and Robotiq gripper in Gazebo / real word (38:53)
Pick and Place with UR5 Robotiq gripper and 3D camera (9:58)

Module 11: Waypoints and Optimize the Trajectory (ROS)

Establish movement repetability (17:12)
What are waypoints and how to use it
How to set Joints Constraints (22:01)
How to draw a circle (19:50)
How to save waypoints and execute trajectory (30:20)
Wrap up and what's next

Module 12: ROS2 Humble (ROS2)

Differences between ROS and ROS2 (10:51)
ROS and ROS2 distributions
Install and setup ROS2 Humble (14:04)
Create ROS2 Workspace (9:19)
Install VS code in Ubuntu 22.04 and ROS2 setup (4:18)
Create packages and basic node in C++ (37:39)
Write a Topic Publisher Subscriber in C++ (37:30)
Create packages and basic node in Python (30:00)
Write a Topic Publisher Subscriber in Python (21:46)
Create Custom Interface (11:47)
Write a service in Python (15:38)
Write a Service in C++ (17:55)
ROS parameters in ROS2 (29:57)
Create a Launch file (30:50)

Module 13: Migrate your project from ROS1 to ROS2 (ROS2)

Migration strategy (9:22)
Migrate custom robot description package (28:23)
ROS2 Control: understanding and deploying Joint Group Position Controller (54:23)
ROS2 Control: setup and deploy Gripper Action Controller (16:20)
ROS2 Control: setup and deploy Joint Trajectory Controller (12:19)
ROS2 Control: Control the robot sending trajectory_msgs from C++ script (11:43)
Replicate the .world from ROS1 (8:31)
Deploy depth camera and the pick object (25:45)
Create MoveIt2 configuration package for a custom robot (55:40)
How to interface Moveit2 with Gazebo (18:24)
Create an inverse kinematics node with MoveIt2 in C++ (23:39)
How to simulate grasping with LinkAttacher in Gazebo (11:38)
Pick and place with a Custom robot: Gazebo interfaced with Rviz (30:27)
UR5, Depth Camera setup in Gazebo with MoveIt (81:24)
Add Robotiq gripper to UR5 and setup in Gazebo with MoveIt (62:50)
Setup the world and LinkAttacher for grasping (15:34)
Test inverse kinematics with UR5 (20:46)
Run Pick and Place with UR5 and Robotiq gripper (8:26)
Point Cloud Library and 3D processing in ROS2 (52:12)
Pick and Place with UR5 Robotiq Gripper using 3D camera in ROS2 (31:58)
Resources and GitHub repository of the master class
How to make your packages portable
Fixing MoveIt Setup Assistant (MSA) Crash after ROS2 Updates

Module 14: Docker for Robotics (ROS & ROS2)

Introduction - Why Docker for Robotics - Installation and Setting up your Environment (22:32)
What is Docker under the Hood (13:19)
How to Create an Image for simulating UR5 Gazebo with Rviz and MoveIt (50:03)
How to run a Container (17:45)
Volumes: Making Data Persistent (19:26)
Using Docker for Complex Hardware — Isaac ROS RealSense Example (21:10)
Multi-Container Robotics with Docker Compose: Connecting Services and Using the Host GUI (33:58)
GUI in Docker (for Rviz, Gazebo, rqt) (12:43)
Using Docker in VS Code and final wrap-up (13:31)
Common Docker Commands Cheatsheet

Module 15: Robotics Applications and LLM [Ollama] Integration in ROS2

Creating the ROS2 Workspace & Cloning xarm_ros2 (12:02)
Building the Custom Package my_xarm6: Robot, World, and Camera Setup (63:25)
Creating the my_xarm6_app Package: First Motion Node (26:46)
CNC Motion Node: Drawing Circles and Squares with Constant Orientation (37:50)
Blind Pick and Place: Fixed Object Positions (29:03)
Vision Integration with OpenCV: Depth Reconstruction via Intrinsics (38:32)
Vision-Based Pick and Place + Custom Service Definition (35:17)
Introduction to Ollama and LLM Integration (12:17)
LLM Task Pipeline: Designing the Action-Oriented Architecture and System Prompt engineering (55:45)
LLM-Driven Pick and Place: Updating System Prompt + Vision + Motion Integration (45:48)
Test Robotics Tasks with LLM Ollama integrated in ROS2 (21:26)
General Task LLM: Unified Prompt and Full Robot Commanding (CNC, pick place with vision, end effector control)) (24:59)

Wrap up and next steps

Wrap up
Next steps

Additional module to be defined based on student's needs and requests

Lessons under constructions

Teach online with

LLM-Driven Pick and Place: Updating System Prompt + Vision + Motion Integration

Lesson content locked

If you're already enrolled, you'll need to login.

Enroll in Course to Unlock