Control Lights with Hand Gestures Using MediaPipe

MediaPipe's hand landmark model finds 21 points on each hand. Comparing fingertip and knuckle positions tells you which fingers are up — a simple, touch-free remote.

Video tutorial

▶ Paul McWhorterWatch on YouTube ↗

Video by Paul McWhorter, embedded from YouTube. The written steps below are our own and match the parts list.

Mission steps

1. Prepare Python

   Create a venv with system site packages and pip install mediapipe (64-bit Raspberry Pi OS required).

2. Capture frames

   Use Picamera2 at 640×480; MediaPipe expects RGB frames.

3. Count fingers

   For each finger, compare the tip's y-coordinate to the joint below it.

4. Map gestures

   1 finger = on, 2 = warm white, 5 = rainbow, fist = off. Hold a gesture for half a second before acting.

5. Connect the house

   Instead of LEDs, publish the gesture to MQTT and let Home Assistant run scenes.

Code

import cv2, mediapipe as mp
from picamera2 import Picamera2

cam = Picamera2(); cam.configure(cam.create_video_configuration(main={"format": "RGB888", "size": (640, 480)})); cam.start()
hands = mp.solutions.hands.Hands(max_num_hands=1, min_detection_confidence=0.6)
TIPS, PIPS = [8, 12, 16, 20], [6, 10, 14, 18]

while True:
    rgb = cv2.cvtColor(cam.capture_array(), cv2.COLOR_BGR2RGB)
    res = hands.process(rgb)
    if res.multi_hand_landmarks:
        lm = res.multi_hand_landmarks[0].landmark
        up = sum(lm[t].y < lm[p].y for t, p in zip(TIPS, PIPS))
        print('fingers up:', up)

Tip: test every script on the bench before installing it in a moving machine.

Questions