{"id":1030,"date":"2024-06-26T11:54:19","date_gmt":"2024-06-26T18:54:19","guid":{"rendered":"https:\/\/lucidbeaming.com\/blog\/?p=1030"},"modified":"2025-12-04T13:41:12","modified_gmt":"2025-12-04T21:41:12","slug":"skulls-composing-music-with-computer-vision-and-a-custom-yolo5-ai-model","status":"publish","type":"post","link":"https:\/\/lucidbeaming.com\/blog\/skulls-composing-music-with-computer-vision-and-a-custom-yolo5-ai-model\/","title":{"rendered":"Skulls: composing music with computer vision and a custom YOLO5 AI model"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

A few years ago, I built a primitive computer vision music player (Oracle) using analog video and a basic threshold detector with an Arduino. Since then, outboard AI vision modules have gotten much more specialized and powerful. I decided to try an advanced build of Oracle using the new Grove Vision AI Module V2.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

This post describes the approach and build, as well as a few pitfalls to avoid. Seeed sent me one of their boards for free and that was the motivation to try this out. Ultimately, I want to use the lessons learned here to finish a more comprehensive build of Oracle with more capability. This particular project is called Skulls because of the plastic skulls and teeth used for training and inference targeting.<\/p>\n\n\n\n

The components are a Grove Vision AI Module V2<\/a> (retails for about $26) with an Xiao ESP32 C3 as a controller and interface. The data from the object recognition gets passed to an old Raspberry Pi 3 model A+ using MQTT. The Pi runs Mosquitto<\/a> as an MQTT broker and client, as well as Fluidsynth<\/a> to play the resulting music. A generic 8226 board is used as a WiFi access point to connect the two assemblies wirelessly.<\/p>\n\n\n\n

What worked<\/h3>\n\n\n\n

Assembling the hardware was very simple. The AI Module is very small and mated well with an ESP32. Each board has a separate USB connector. The AI Module needs that for uploading models and checking the camera feed. The ESP32 worked with the standard Arduino IDE. I added the custom board libraries<\/a> from Seeed to ensure compatibility.<\/p>\n\n\n\n

In the beginning I did most of the AI work directly connected to the AI Module and not through the ESP32. It was the only way to see a video feed of what I was getting.<\/p>\n\n\n\n

One of the reasons I put so much effort into this project was to use custom AI models. I wasn’t interested in doing yet another demo of a face recognition or pets or whatever. I’m interested in exploring new human-machine interfaces for creative output. This particular module has the ability to use custom models.<\/p>\n\n\n\n

So, I tried to follow the Seeed instructions for creating a model. It was incredibly time consuming and there were many problems. The most effective tip I can offer is to use the actual camera connected to the board to generate training images AND to clean-up those images in Photoshop or Gimp. I went through a lot of trial and error with paramters and context. Having clean images fixed a lot of the recognition issues. I generated and annotated 176 images for training. That took 5-6 hours and the actual training in the Collab notebook took 2-3 hours with different options.<\/p>\n\n\n\n

Here is my recipe:<\/p>\n\n\n\n