HandCast
A gesture-based 3D lighting control system using hand tracking and pinch gestures.
Overview
What it does
HandCast is a gesture-based 3D lighting control system that detects hand pointing gestures and allows users to control smart bulbs in a realistic virtual environment using pinch gestures.
The project translates physical real-world hand movements into interactions within a 3D scene. It uses a React and Three.js frontend for rendering and a Python backend with MediaPipe for hand tracking, communicating through WebSockets.
The application includes a realistic 3D room, dynamic lighting, modern user interface elements, and spatial audio to provide an immersive interaction experience.
Problem and solution
Problem
There was no natural way to select a specific electrical device in a smart home system using hand gestures alone.
Solution
HandCast uses raycasting to identify the targeted electrical device in the 3D scene and then controls it through pinch-based interactions.
Highlights
- 01 Real-time hand tracking
- 02 3D pointing detection
- 03 Pinch gesture recognition
- 04 Smart bulb simulation
- 05 WebSocket communication
Features
Real-time Hand Tracking
Uses MediaPipe and webcam input to track hand movements in real time.
3D Pointing Detection
Performs precise ray casting within the 3D environment to determine pointing direction.
Pinch Gesture Recognition
Detects pinch gestures to enable interactions with virtual devices.
Smart Bulb Simulation
Provides realistic 3D smart bulb models with lighting effects.
WebSocket Communication
Enables fast real-time communication between the frontend and backend.
Architecture
A React frontend renders the 3D scene using Three.js and handles ray casting, pinch detection, and UI components.
A Python backend uses MediaPipe for hand tracking, runs a FastAPI server, and streams real-time data.
The frontend and backend communicate through WebSockets for real-time interaction.