top of page
Homepage

Jiaying [Vina]  wei

banner_v2.png
Course Overview

This graduate-level course explores the integration of computer vision and sensing within robotic fabrication. Students develop 'perception-in-the-loop' systems, bridging computational design and physical assembly.

Logistics
  • Meetings: Mon/Wed 1:30 PM – 4:20 PM
  • Location: DFAB RobotRoom, MMCH
  • Prerequisites: 48-755 or equivalent Python/Robotics proficiency
Instructor

Jiaying Wei
jwei3@andrew.cmu.edu
Office Hours: Fridays 10:00 AM – 12:00 PM (RobotRoom Lobby)

Learning Objectives

Formulate and implement real-time perception-to-action workflows for robotic assembly and fabrication.

Evaluate sensor data (vision, force, depth) to adjust robot trajectories in dynamic physical environments.

Design robotic end-effectors and support systems specifically for perceived architectural constraints.

Execute safe and collaborative multi-robot operations within the CMU DFAB RobotRoom ecosystem.

Analyze the impact of immersive automation on the fidelity of digital-to-analog translation in design.

Document and present complex behavioral robotics logic through technical clarity and student-facing diagrams.

Course Structure & Policies

  • Course Structure: Transition from technical tutorials (Weeks 1-7) to perception-driven project development (Weeks 8-15).
  • Safety: Mandatory robot safety training and adherence to DFAB RobotRoom protocols at all times.
  • AI Policy: AI tools allowed for coding assistance/documentation but must be clearly cited and verified for safety.
  • Collaboration: Peer learning encouraged; however, individual perception logic and final fabrication must be unique.
  • Late Work: 10% deduction per day; milestones are strict due to robot scheduling constraints.
  • Integrity: Hard adherence to CMU standards; safety violations result in immediate lab access revocation.
  • Wellness & Support: Prioritize mental health; contact instructors or CaPS for academic/personal support.

Project & Grading

  • Project Types: Focus on perception-in-the-loop systems (CV/Sensing) influencing robotic toolpaths in real-time.
  • Requirement: All final projects MUST demonstrate a closed-loop system where sensing alters robot behavior.
  • Milestones: P1: Sensing Logic (20%), P2: Adaptive Pathing (30%), Final: Integrated Assembly (50%).
  • Breakdown: Attendance & Safety (10%), Technical Tutorials (20%), Project Milestones (70%).
  • Professionalism: Students are expected to maintain lab organization and professional safety standards.

Phase

Weekly Syllabus & Key Deliverables

Week 1 

Transition from Offline to ROS Planning

  • Course intro, syllabus, rubrics, semester schedule

  • Intro to Linux and ROS Noetic

  • RobotStudio recap and limitations

​​

Lab:

  1. basic Linux, workspace setup,

  2. virtual controller to ROS,

  3. basic package build,

  4. COMPAS environment setup

Week 2

Kinematics

  • Lecture: Mathematical foundations of robot motion.

  • Lab: Solving tool center points (TCP) for custom effectors.

  • Activity: Simulating reachability in Grasshopper.

Week 3 

ROS Fundamentals I

  • Topic/Message/package structure

  • Launch file, XML formatting, referencing, and namespace.

  • MoveIt motion planning setup 

    • What is a URDF

    • Select your planner

    • How to enable/disable collision check between links

    • ​

Lab:

How to build your own package

  • One subscriber, one publisher (in the next week, students will use what they publish to test receiving using COMPAS

  • Project Team Formation: Groups of 3-4 students

  • Brainstorm

Week 4 

ROS Fundamentals II + Proposal

  • Params, services, actions

  • RWS and EGM (ABB specific)

  • Wednesday: Presentation + 1-page project intent submission

Week 5

Intel RealSense & Sensors

  • RGBD camera principles and calibration

  • Generalized sensors application

  • Point cloud processing and object detection

  • Lab with robot: 

    • Turn on camera launch file and adjust params

    • Camera-hand-eye calibration and

    • How to turn on spatial mapping (Octomap)

Week 6

Advanced Planning & Collision Management

  • Dynamic obstacle addition/removal

  • How to attach objects to the robot’s end-effector

  • Lab: object attachment and detaching with MoveIt Python interface

    • ROS Package Building Test:

      • Individual practical test on creating custom ROS packages

      • URDF integration of end-effector

      • basic MoveIt configuration

    • Reachability matrix

Week 7

Sensing Applications

  • Real-time environment monitoring

  • Usage of QR code, color masking, etc

  • Lab: real-time tracking operations/teleoperations

  • Special Topics Lecture: 

    • message synchronization and filtering 

    • Multi-ROS pc communication

    • Some basic CV techniques

Week 8

Spring Break, no classes

Week 9

  • Project Progress Presentations (Monday)

    • First project milestone: Basic robot-sensor communication

    • mature/holistic planning of technical integration, the pipeline, and the expected outcome.

  • Wednesday - Work session

Week 10 

Project Foundation Development

  • Core system setup and basic functionality implementation

  • Sensor integration planning and initial testing

  • Lecture: PCL library and RTAB map

Week 11

Midterm Milestone & Technical Planning

  • COMPAS-Grasshopper-ROS pipeline implementation

  • Collision detection and adaptive planning integration

  • Second project milestone: End-to-end design-to-robot workflow

  • Lecture: learning based segmentation, tracking, and classification (off-the-shelf)

Week 12

Prototype Refinement

  • User testing and workflow optimization

  • Third project milestone: Working prototype demonstration

Week 13 

System Integration & Testing

  • Full system integration and performance optimization

  • Safety validation and robustness testing

  • Documentation development and video preparation

Week 14 

Final Prototype Polish

  • Final debugging and system refinement

  • Presentation preparation and demonstration rehearsal

  • Peer review and feedback incorporation

Week 15

Final Presentations & Submission

  • Public Demonstrations ( for HRI projects in a construction context): Live robot operations with architectural applications

  • Technical Documentation Review: Complete system documentation and code repositories

Selected Student Projects

Note that these projects are owned and created by student, where instructor's primary reponsibility is to supervise and mentor,

See details by link ->

Air Drawing

air_drawing1.gif

See details by link ->

Waste Assembly

Gen-Mod - Joshua Bard.jpg.jpeg
bottom of page