HandCast

A cutting-edge gesture-based 3D lighting control system

Overview Features Demo Tech Stack Architecture

Overview

HandCast is a gesture-based 3D lighting control system that detects hand pointing gestures and allows users to toggle smart bulbs in a realistic 3D environment using pinch gestures. The project translates physical real-world movements into interactions within a virtual 3D space, built using a React Three.js frontend and a Python MediaPipe backend over WebSockets.

Features

Real-time hand tracking — utilizing MediaPipe and webcam input for accurate tracking
3D pointing detection — performing precise ray casting within the 3D space
Pinch gesture recognition — enabling device interactions natively
Smart bulb simulation — featuring realistic 3D models and lighting effects
WebSocket communication — facilitating fast, real-time message passing between frontend and backend
Realistic 3D room environment — featuring walls, a ceiling, and furniture for an immersive experience
Dynamic lighting system — casting shadows and simulating ambient effects
Modern UI / Glassmorphism — providing a modern debug panel with backdrop blur effects
Spatial audio effects — playing sound feedback for bulb interactions

Demo

Tech Stack

Layer	Technology
Frontend	React 19.1, @react-three/fiber, @react-three/drei, Three.js, TailwindCSS
Backend	Python 3.8+, FastAPI, MediaPipe, OpenCV, WebSockets
Database	N/A
Infrastructure	Node.js, Vite

Architecture

┌─────────────────┐    WebSocket    ┌──────────────────────┐
│   React Frontend│ ←──────────────→│   Python Backend     │
│                 │                 │                      │
│ • Three.js 3D   │                 │ • MediaPipe Hand     │
│ • Ray Casting   │                 │   Tracking           │
│ • Pinch Detect  │                 │ • FastAPI Server     │
│ • UI Components │                 │ • Real-time Data     │
└─────────────────┘                 └──────────────────────┘