Skulls: composing music with computer vision and a custom YOLO5 AI model

A few years ago, I built a primitive computer vision music player (Oracle) using analog video and a basic threshold detector with an Arduino. Since then, outboard AI vision modules have gotten much more specialized and powerful. I decided to try an advanced build of Oracle using the new Grove Vision AI Module V2.

This post describes the approach and build, as well as a few pitfalls to avoid. Seeed sent me one of their boards for free and that was the motivation to try this out. Ultimately, I want to use the lessons learned here to finish a more comprehensive build of Oracle with more capability. This particular project is called Skulls because of the plastic skulls and teeth used for training and inference targeting.

The components are a Grove Vision AI Module V2 (retails for about $26) with an Xiao ESP32 C3 as a controller and interface. The data from the object recognition gets passed to an old Raspberry Pi 3 model A+ using MQTT. The Pi runs Mosquitto as an MQTT broker and client, as well as Fluidsynth to play the resulting music. A generic 8226 board is used as a WiFi access point to connect the two assemblies wirelessly.

What worked

Assembling the hardware was very simple. The AI Module is very small and mated well with an ESP32. Each board has a separate USB connector. The AI Module needs that for uploading models and checking the camera feed. The ESP32 worked with the standard Arduino IDE. I added the custom board libraries from Seeed to ensure compatibility.

In the beginning I did most of the AI work directly connected to the AI Module and not through the ESP32. It was the only way to see a video feed of what I was getting.

One of the reasons I put so much effort into this project was to use custom AI models. I wasn’t interested in doing yet another demo of a face recognition or pets or whatever. I’m interested in exploring new human-machine interfaces for creative output. This particular module has the ability to use custom models.

So, I tried to follow the Seeed instructions for creating a model. It was incredibly time consuming and there were many problems. The most effective tip I can offer is to use the actual camera connected to the board to generate training images AND to clean-up those images in Photoshop or Gimp. I went through a lot of trial and error with paramters and context. Having clean images fixed a lot of the recognition issues. I generated and annotated 176 images for training. That took 5-6 hours and the actual training in the Collab notebook took 2-3 hours with different options.

Here is my recipe:

  • Use a simple Arduino sketch to record jpegs from the camera onto an SD card.
  • In an image editor, apply Reduce Noise and Levels to the images to normalize them. Don’t use “Auto Levels” or any other automatic toning.
  • The images will be 240px X 240px. Leave them that size. Don’t export larger.
  • In Roboflow choose “Object Detection”, not “Instance Segmentation”, for the project.
  • When annotating, choose consistent spacing between your bounding box and the edges of your object.
  • Yes, you can annotate multiple objects in a single image. It’s recommended.
  • For preprocessing, I chose “Filter Null and “Grayscale”.
  • For augmentation, I chose “Rotate 90”, “Rotation”, and “Cutout”. I did NOT use “Mosaic” as recommended in the Seeed Wiki. That treatment already happens in the Collab training script.
  • I exported the dataset using JSON > COCO. None of the other options were relevant.
  • The example Google Collab notebook I used was the rock/paper/scissors version (Gesture_Detection_Swift-YOLO_192). I only had a few objects and it was the most relevant.
  • I left the image size at 192×192 and trained for 150 epochs. The resulting TFLite INT8 model was 10.9mb.
  • I used the recommend web tool to connect directly to the AI Module and upload the model. It took multiple tries.
  • On the ESP32 I installed MQTT and used that to transmit the data to my Raspberry Pi. I did not use the on-board wifi/MQTT setup of the AI Module.

This was a difficult project because of very confusing and incomplete documentation at mutiple stages. It’s clear to me that the larger companies don’t actually want us to be able to do all this ourselves. There were times it felt intentionally obfuscated to force me to buy a premium tier and some unrelated commercial application. I’m glad I did it though, because I learned some important concepts and limitations of AI training.

Demo

A demo of different sounds and arrangements produced by the assembly.

Conclusion

I’ll use this knowledge to finish a new build of the actual Oracle music composition platform I started. This particular demo is interesting, but a somewhat unpredictable and technically fragile. I found the research on generative music to be the most interesting part. As for the AI, I’m sure all this will be simplified and optimized in the future. I just hope the technology stays open enough for artists to use independently.

MMXX: signals, sounds, sights

I spent most of the year in my art studio while the city around me contracted and calcified due to Covid. I was fortunate that my plans coincided with the timing and degree of changes in the world. It could have very easily gone the other way, as I’ve seen firsthand. Lots of my friends in the art community are struggling.

My work this year reflects more studio and internet based processes. Previous years always included public festivals, performances, and collaborations. Some of that change was to save money, but it was also an effort to make use of what I had around me. It was to stay present and maintain momentum with ongoing projects.

I did actually manage to pull off a few public projects, including a portable projection piece that had animated wolves running on rooftops. I savored that experience and learned a lot from the constraints of lock-down art performances.

Looking back on this year, I see new priorities being formed. While the coding and online projects were effective, the amount of screen time required took a toll. I relished the drawing projects I had and hope to keep working in ways that make a huge mess.

Sightwise

My studio complex has a co-op of artists called FUSE Presents. We hold regular group art shows in normal times and for each show, two artists get featured. I was one of the featured artists for the March 2020 show. That meant I got extra gallery space and special mention in marketing materials.

The work I picked was drawn from a variety of efforts in the previous two years. As a grouping, it represented my current best efforts as a multimedia artist. I worked hard to finalize all the projects and really looked forward to the show.

It combined abstract video, traditional photography, sculptural video projection, installation work, and works on paper.

I designed the show’s poster in open source software called Inkscape.

Unfortunately, the show happened right as the first announcements about the local spread of Covid had begun. People were already quarantined and we heard about the first deaths in our county. That news didn’t exactly motivate people to come out to the art show. Attendance was sparse at best. But, all that work is finished now and ready for future exhibits.

Camel

I found a cigarette tin that had been used as a drug paraphernalia box and decided to build a synthesizer out of it. I had been experimenting with a sound synthesis library called Mozzi and was ready to make a standalone instrument with it. I spent about a month on the fabrication and added a built-in speaker and battery case to make it portable. Sounds pretty rad.

I released my code as open source in a Github repo and a follower from Vienna, Austria replicated my synth using a cake box from Hotel Sacher. (apparently famous for their luxury cakes?)

Wolves

The Wolves project was a major undertaking that took place over 2 years. It began with an interest in the Chernobyl wolves that became a whole genre of art for me.

I began hand digitizing running wolves from video footage and spent a year adding to that collection. I produced hundreds and hundreds of hand drawn SVG frames and wrote some javascript that animated those frames in a variety of ways. I got to the point where I could run a Raspberry Pi and a static video projector with the wolves running on it. I took a break from the project after that.

By the time I returned to the project, the Covid lockdown was in full swing and American city streets looked abandoned. We all started seeing footage of animals wandering into urban areas. It made sense to finish the Wolves project as an urban performance, projecting onto buildings from empty streets.

Building a stable, self-powered and portable rig that could be pulled by bicycle turned out to be harder than I thought. There were so many details and technical issues that I hadn’t imagined. Every time I thought I was a few days from launch, I would have to rebuild something that added weeks.

The first real ride with this through Japantown in northern San Jose was glorious. Absolutely worth the effort. I ended up taking it out on the town many times in the months to come.

Power up test in the backyard
San José City Hall
Japantown, north of downtown San José

The above video is from Halloween, which was amazing because so people were outside walking around. That’s when the most people got to see it in the wild.

But, my favorite moment was taking it out during a power blackout. Whole neighborhoods were dark, except for me and my wolves. I rode by one house where a bunch of kids lived and the family was out in the yard with flashlights. The kids saw my wolves and went crazy, running after them and making wolf howl sounds while the parents laughed. Absolute highlight of the year.

Videogrep

Videogrep is a tool to make video mashups from the time markers in closed captioning files. It’s the kind of thing where you can take a politician’s speech and make him/her say whatever you want by rearranging the parts where they say specific words. It was a novelty in the mid-2000s that was seen on talk shows and such, as a joke. Well, the computer process behind the tool is very useful.

I didn’t create videogrep, Sam Lavigne did and released his code on Github. (BTW, the term “grep” in videogrep comes from a Unix utility (grep) used to search for things) What I did do is use it to find other things besides words, such as breathing noises and partial words. I used videogrep to accentuate mistakes and sound glitches as much as standalone speech and words.

Here is a typical series of commands I would use:

videogrep --input videofile.mp4 -tr

cat videofile.mp4.transcription.txt | tr -s ' ' '\n' | sort | uniq -c | sort -r | awk '{ print $2, $1 }' | sed '/^[0-9]/d' > words.txt

videogrep -i videofile.mp4 -o outputvideo.mp4 -t -p 25 -r -s 'keyword' -st word

ffmpeg -i outputvideo.mp4 -filter_complex "frei0r=nervous,minterpolate='fps=120:scd=none',setpts=N/(29.97*TB),scale=1920:1080:force_original_aspect_ratio=increase,crop=1920:480" -filter:a "atempo=.5,atempo=.5" -r 29.97 -c:a aac -b:a 128k -c:v libx264 -crf 18 -preset veryfast -pix_fmt yuv420p if-stretch-big.mp4

Below is a stretched supercut of the public domain Orson Welles movie The Stranger. I had videogrep search for sounds that were similar to speech but not actual words or language. Below that clip is a search of a bunch of 70s employee training films for the word “blue”. Last is a supercut of one the Trump/Biden debates where the words “football and “racist” are juxtaposed.

Specific repeated words used in a 2020 Presidential Debate: fear, racist, and football

Vid2midi

While working on the videos produced by videogrep, I found a need for soundtracks that were timed to jumps in image sequences. After some experimenting with OpenCV and Python, I found a way to map various image characteristics to musical notation.

I ended up producing a standalone command-line utility called vid2midi that converts videos into MIDI files. The MIDI file can be used in most music software to play instruments and sounds in time with the video. Thus, the problem of mapping music to image changes was solved.

It’s now open source and available on my Github site.

The video above was made with a macro lens on a DSLR and processed with a variety of video tools I use. The soundtrack is controlled by a MIDI file produced by vid2midi.

Bad Liar

This project was originally conceived as a huge smartphone made from a repurposed big screen TV. The idea is that our phones reflect our selves back to use, but as lies.

It evolved into an actual mirror after seeing a “smart mirror” in some movie. The information in the readout scrolling across the bottom simulates a stock market ticker. Except, this is a stock market for emotions. The mirror is measuring your varying emotional states and selling them to network buyers in a simulated commodities exchange.

Screen test showing emotional stock market
Final demo in the studio

Hard Music in Hard Times

TQ zine is an underground experimental music zine from the U.K. I subscribed a few years ago after reading a seminal essay about the “No audience underground”. I look forward to it each month because it’s unpretentious and weird.

They ran an essay contest back in May and I was one of the winners! My prize was a collection of PCBs to use in making modular synthesizers. I plan to turn an old metal lunchbox into a synth with what I received.

Here is a link to the winning essay:

Lunetta Synth PCB prizes from @krustpunkhippy

Books

I spent much of my earlier art career as a documentary photographer. I still make photographs but the intent and subject matter have changed. I’m proud of the photography I made throughout the years and want to find good homes for those projects.

Last year I went to the SF Art Book Fair and was inspired by all the publishers and artists. Lots of really interesting work is still being produced in book form.

Before Covid, I had plans to make mockups of books of my photographs and bring them to this year’s book fair to find a publisher. Of course, the fair was cancelled. I took the opportunity to do the pre-production work anyway. Laying out a book is time consuming and represents a standalone art object in itself.

I chose two existing projects and one new one. American Way is a collection of photos I made during a 3 month American road trip back in 2003. Allez La Ville gathers the best images I made in Haiti while teaching there in 2011-13 and returning in 2016. The most recent, Irrealism, is a folio of computer generated “photographs” I made using a GAN tool.

It was a thrill to hold these books in my hands and look through them, even if they are just mockups. After all these years, I still want my photos to exist in book form in some way.

Allez La Ville, American Way, Irrealism

Art Review Generator

Working on the images for the Irrealism book mentioned above took me down a rabbit hole into the world of machine learning and generative art. I know people who only focus on this now and I can understand why. There is so much power and potential available from modern creative computing tools. That can be good and bad though. I have also seen a lot of mediocre work cloaked in theory and bullshit.

I gained an understanding of generative adversarial networks (GAN) and the basics of setting up Linux boxes for machine learning with Tensorflow and PyTorch. I also learned why the research into ML and artificial intelligence is concentrated at tech companies and universities. It’s insanely expensive!

My work is absolutely on a shoestring budget. I buy old computer screens from thrift stores. I don’t have the resources to set up cloud compute instances with stacked GPU configurations. I have spent a lot of time trying to figure out how to carve a workflow from free tiers and cheap hardware. It ain’t easy.

One helpful resource is Google Collab. It lets “researchers” exchange workbooks with executable code. It also offers free GPU usage (for now, anyway). That’s crucial for any machine learning project.

When I was laying out the Irrealism book, I wanted to use a computer generated text introduction. But, the text generation tools available online weren’t specialized enough to produce “artspeak”. So, I had the idea to build my own art language generator.

The short story is that I accessed 57 years of art reviews from ArtForum magazine and trained a GPT-2 language model with the results. Then I built a web app that generates art reviews using that model, combined with user input. Art Review Generator was born.

This really was a huge project and if you’re interested in the long story, I wrote it up as a blog post a few months ago. See link below.

See examples of generated results and make your own.

Kiosk

Video as art can be tricky to present. I’m not always a fan of the little theaters museums create to isolate viewers. But, watching videos online can be really limited in fidelity of image or sound. Projection is usually limited by ambient light.

I got the idea for this from some advertising signage. It was seeded with a monitor donation (thanks Julie Meridian!) and anchored with a surplus server rack I bought. The killer feature is the audio level rises and falls depending on whether is someone is standing in front of it or not. That way, all my noise and glitch soundtracks aren’t at top volume all the time.

This plays 16 carefully selected videos in a loop and runs autonomously. No remote control or start and stop controls. Currently installed at Kaleid Gallery in downtown San Jose, CA.

Holding the Moment

Hanging out in baggage claim with no baggage or even a flight to catch

In July, the San José Office of Cultural Affairs announced a call for submissions for a public art project called Holding the Moment. The goal was to showcase local artists at Norman Y. Mineta San José International Airport.

COVID-19 changed lives everywhere — locally, nationally, and internationally. The Arts, and individual artists, are among those most severely impacted. In response, the City of San José’s Public Art Program partnered with the Norman Y. Mineta San José International Airport to offer local artists an opportunity to reflect, comment, and on of this global crisis and the current challenging time. More than 327 submissions were received, and juried by a prominent panel of Bay Area artists and arts professionals. Ultimately 96 artworks by 77 San José artists were awarded a $2,500 prize and a place in this six-month exhibition.

SAN JOSE OFFICE OF CULTURAL AFFAIRS

Two of my artworks were chosen for this show and they are on display at the airport until January 9. They picked some challenging pieces, PPE and Mask collage, with interesting back stories of their own.

Here are the stories of the two pieces they chose for exhibition.

PPE

The tale of this image begins in Summer of 1998. I had a newspaper job in Louisiana that went badly. One of the few consolations was a box of photography supplies I was able to take with me. In that box was a 100′ bulk roll of Ilford HP5+ black and white film. My next job happened to involve teaching digital photography so I stored that bulk roll, unopened and unused, for decades. I kept it while I moved often, always thinking there would be some project where I would need a lot of black and white film.

Earlier this year, I was inspired to buy an old Nikon FE2 to make some photos with. I just wanted to do some street photography. After Covid there weren’t many people in the streets to make photos of. But, I did break out that HP5+ that I kept for decades and loaded it onto cassettes for use in the camera I had bought. I also pulled out a Russian Zenitar 16mm f2.8 that I used to shoot skateboarding with.

This past Summer, I went to Alviso Marina County Park often. It’s a large waterfront park near my house that has access to the very bottom of San Francisco bay. People would wear masks out in the park and I even brought one with me. It was absolutely alien to wear protective gear out in a huge expanse like that.

So, my idea was to make a photo that represented that feeling. I brought my FE2 with the old film and Zenitar fisheye to the park, along with a photo buddy to actually press the button. People walking by were weirded out by the outfit, but that’s kind of the desired effect.

This image was enlarged and installed in the right-hand cabinet at the airport show.

An interesting side note to this project was recycling the can that the old film came in. Nowadays that would be made of plastic but they still shipped bulk film in metal cans back then. I took that can and added some knobs and switches to control a glitching noisemaker I had built last year. So, that old film can is now in use as a musical instrument.

The film can that used to hold 100′ of Ilford HP5+ is now a glitch sound machine

Mask Collage

Face masks are a part of life now but a lot of people are really pissed that they have to wear them. I was in the parking lot of a grocery store and a guy in front of me was talking to himself, angry about masks. Turns out he was warming up to argue with the security guard and then the manager. While I was inside shopping (~20 minutes) he spent the whole time arguing loudly with the manager. It was amazing to me how someone could waste that much time with that kind of energy.

When I got back to my studio I decided to draw a picture of that guy in my sketchbook. That kicked off a whole series of drawings over the next month.

I have a box of different kinds of paper I have kept for art projects since the early 90s. In there was a gift from an old roommate: a stack of blank blood test forms. I used those forms as the backgrounds for all the drawings. Yellow and red spray ink from an art colleague who moved away provided the context and emotional twists.

The main image is actually a collage of 23 separate drawings. It was enlarged and installed in the left-hand cabinet at the airport show.

Internet Archive

A few weeks ago, my video Danse des Aliénés won 1st place in the Internet Archive Public Domain Day film contest. It was made entirely from music and films released in 1925.

Danse des Aliénés

Film and music used:

In Youth, Beside the Lonely Sea

Das wiedergefundene Paradies
(The Newly Found Paradise)
Lotte Lendesdorff and Walter Ruttmann

Jeux des reflets et de la vitesse
(Games on Reflection and Speed)
Henri Chomette

Koko Sees Spooks
Dave Fleischer

Filmstudie
Hans Richter

Opus IV
Walther Ruttmann

Joyless Street
Georg Wilhelm Pabst

Danse Macabre Op. 40 Pt 1
(Dance of Death)
Camille Saint-Saëns
Performed by the Philadelphia Symphony Orchestra

Danse Macabre Op. 40 Pt 2
(Dance of Death)
Camille Saint-Saëns
Performed by the Philadelphia Symphony Orchestra

Plans? What plans?

Vaccines are on the way. Hopefully, we’ll see widespread distribution in the next few months. Until then, I’ll still be in my studio working on weird tech art and staying away from angry mask people.

I am focused on future projects that involve a lot of public participation and interactivity. I think we will need new ways of re-socializing and I want to be a part of positive efforts in that direction.

I also have plans for a long road trip from California to the east coast and back again. It will be a chance to rethink the classic American photo project and find new ways to see. But, that depends on how things work out with nature’s plans.

MMXIX: time, noise, light

This year saw the completion of new sound sculptures and large installation work. It offered up new performance contexts and an expansion of exhibition options. The projects have grown in scale and scope, but the internal journey continues.

Wheel of Misfortune

A few years ago I noticed neighborhood kids putting empty water bottle into spokes of the back wheels of their bikes. They got a cool motorcycle sound out of it. One of them had two bottles offset and that produced a rhythmic but offbeat cycle that sounded interesting.

It gave me the idea to use a bicycle wheel for repeating patterns the way drum machines an sequencers do. I also thought it would be an interesting object to build from a visual standpoint.

It took a while, but having the workspace to lay out larger electronics assemblies was helpful. I settled on five sensors in a bladed array reading magnets attached to the spokes.

A first performance at local gallery Anno Domini with Cellista was fun, but the sounds I had associated with the triggers lacked bite. I reworked the Raspberry Pi running Fluidsynth and built 14 new instruments using a glitched noise sound pack I released a few years ago.

To switch between the instruments I came up with a contact mic trigger using a chopstick and an Arduino. It has a satisfying crack when tapped and cycles the noise patches effectively.

The Wheel got a loud and powerful public test at Norcal Noisefest. People responded not only to novelty of the bicycle wheel, but the badass sound it could make.

https://www.instagram.com/p/B0XYC6hjkVq/?utm_source=ig_web_copy_link

Oracle

I get asked to do sound performances more often these days and it can be challenging because I don’t have much outboard musical gear. So, I have a general effort to create more gear to use live. A common need is to have an interesting way of triggering longform loops I created in my studio.

Taking a cue from the grid controllers used by Ableton Live, I had the idea to build a player that keyed off objects placed under a camera. Reading location and size, it could arrange loops in a similar way.

Computer vision test for Oracle

The project kicked off with an analog video stand I found that was used for projecting documents in a business presentation. I connected that to a primitive but very effective computer vision board for Arduino called the Video Experimenter.

After months of testing with different objects I settled on small white rocks that brought inherent contrast. At a library sale I picked up a catalog of pictograms from Chinese oracle bones that had fascinating icons to predict the future with.

Oracle stones

That clinched the theme of an “oracle” divining the future of a musical performance rather than a musician executing a planned performance.

It has turned out to be really flexible for performances and is a crowd favorite, especially when I let people place the stones themselves.

Oracle at First Friday

Delphi

Smashed tv screen for Oracle
Looks cool, huh? I wish I could say it was intentional. I smashed the screen while loading the equipment for SubZERO this year. meh, I just went with it.

People give me things, usually broken things. I don’t collect junk though. I learned the hard way that some things take a lot of work to get going for very little payoff. Also, a lot of modern tech is mostly plastic with melted rivets and tabs instead of screws or bolts. They weren’t meant to be altered or repaired.

Big screen TVs are a good example. One of the ways they got so cheap is the modular way they get made with parts that weren’t meant to last. I got a fairly large one from Brian Eder at Anno Domini and was interested in getting it back up.

Unfortunately, a smashed HDMI board required some eBay parts and it took more time than expected. Once it was lit up again and taking signal I started running all kinds of content through the connector boards.

When hung vertical, it resembled one of those Point-of-Purchase displays you see in cell phone stores. I though about all the imagery they use to sell things and it gave me the idea of showing something more human and real.

In society that fetishizes youth culture and consumption, we tend to fear aging. I decided to find someone at a late stage of of life to celebrate and display four feet high.

That person turned out to be Frank Fiscalini. At 96 years old he has led a full rich life and is still in good health and spirits. It took more than a few conversations to explain why I wanted to film a closeup of his eyes and face, but he came around.

I set the TV up in my studio with his face looping for hours, slowly blinking. I had no real goal or idea of the end. I just lived with Mr. Fiscalini’s face for a while.

I thought a lot about time and how we elevate specific times of our lives over others. In the end, time just keeps coming like waves in the ocean. I happen to have a fair amount of ocean footage I shot with a waterproof camera.

With the waves projected behind his face, my studio was transformed into a quiet meditation on time and humanity.

Other contributions of building scaffolding and P.A. speakers formed the basis of a large-scale installation. Around this time, I had also been reading a strange history of the Oracle of Delphi.

At first the “oracle” was actually a woman whose insane rants were likely the result of hallucinations from living over gas events. A group of men interpreted what she said and ended up manipulating powerful leaders for miles.

Thus Delphi was formed conceptually. The parallels to modern politics seemed plain, but I’ve been thinking a lot about the futility of trying to control or predict the future. This felt like a good time for this particular project.

Balloon synth

The annual SubZERO Festival here in San Jose has been an anchor point for the past few years. One challenge I’ve faced is the strong breeze that blows through in the hour before sunset. For delicate structures and electronics on stands, it’s a problem. Instead of fighting it this year, I decided to make use of it.

I had an idea to put contact mics on balloons so when the wind blew, the balloons would bounce against each other. I thought they might be like bass bumping wind chimes.

Thanks to a generous donation by Balloonatics, I had 15 beautiful green balloons for the night of the festival. Hooked up to mics and an amplifier, they made cool sounds. But, it took a bit more force than the breeze to move them forcefully enough.

https://www.instagram.com/p/BycRPL2jQn-/?utm_source=ig_web_copy_link

Kids figured out they could bump and play with the balloons and they would make cool noises. Sure enough, it drew a huge crowd quickly. People came up to the balloons all night and punched and poked them to get them to make noise.

On the second night, though, the balloons were beat. Some rowdy crowds got too aggro and popped a bunch of them. Anyway, they were a big hit and it was fun to have something like that around.

Belle Foundation grant

An early surprise of the year was getting an envelope from the Belle Foundation with an award for one the year’s grants. I was stoked to be included in this group.

My application was simple and I talked a lot about SubZERO projects and working with older technology. In other words, what I actually do. To get chosen while being real about the art I make was refreshing.

Content Magazine profile

Before I moved back to California in 2012, I worked at an alt-weekly newspaper in Charleston, SC. I photographed all kinds of cultural events and wrote profiles of artists and musicians. But, I was always on the other side of the interview, as the interviewer.

Daniel Garcia from local magazine Content reached out in the beginning of this year and said they were interested in profiling of me and my work. The tables had turned.

Content Magazine spread
Opening portrait and write-up in Content

Writer Johanna Hickle came by my Citadel art studio and spent a generous amount of time listening to me ramble about tech and such. Her write-up was solid and she did a good job distilling a lot of info.

Content Magazine spread
Collage and write-up in Content magazine

It was nerve-wracking for me, though. I knew the power they had to shape the story in different directions. I was relieved when it came out fine and had fun showing it to people.

Norcal Noisefest

In 2017, I went to the Norcal Noisefest in Sacramento. It had a huge impact on my music and approach to anything live. I came back feeling simultaneously assaulted and enlightened.

Over the past two years, I’ve built a variety of live sound sculptures and performed with most of them. This year the focus was on the new Wheel of Misfortune. I reached out to Lob Instagon, who runs the festival, and signed up for a slot as a performer at Norcal Noisefest in October.

Coincidentally, I met Rent Romus at an Outsound show in San Francisco and told him about performing at Noisefest. Rent puts on all kinds of experimental shows in SF and he suggested a preview show at the Luggage Store.

So I ended up with a busy weekend with those shows and an installation at First Friday.

Norcal Noisefest was a blast and I got see a bunch of rad performances. My set sounded like I wanted, but I have a ways to go when it comes to stage presence. Other people were going off. I have to step things up if I going to keep doing noise shows

Flicker glitch

I have been making short-form abstract videos for the past few years. Most have a custom soundtrack or loop I make. This year I collected the best 87 out of over 250 and built a nice gallery for them on this site.

Every once in a while I get requests from other groups and musicians to collaborate or make finished visuals for them. Most people don’t realize how much time goes into these videos and I’m generally reluctant to collaborate in such an unbalanced way.

I was curious about making some longer edited clips though. I responded to two people who reached out and made “music videos” for their pre-existing music. It wasn’t really collaborative, but I was ok with that because email art direction can be tricky.

The first, Sinnen, gave me complete freedom and was releasing an album around the same time. His video was a milestone in my production flow. It was made entirely on my iPhone 7, including original effects, editing and titles. I even exported at 1080p, which is a working resolution unthinkable for a small device just five years ago. They could shoot at that fidelity, but not manipulate or do complex editing like that.

The next video was much more involved. It was for a song by UK metal band Damim. The singer saw my videos on Instagram and reached out for permission to use some of them. I offered to to just make a custom video instead.

All the visuals were done on my iPhone, with multiple generations and layers going through multiple apps. I filled up my storage on a regular basis and was backing it up nightly. Really time consuming. Also, that project required the horsepower and flexibility of Final Cut Pro to edit the final results.

I spent six months in all, probably 50 hours for so. I was ok with that because it was a real world test of doing commissioned video work for someone else’s music. Now I know what it takes to produce a video like that and charge fairly in the future.

New photography

Yes, I am still a photographer. I get asked about it every once in awhile. This year I came out with two different small bodies of work shooting abstracts and digitizing some older work.

Photographs on exhibit at the Citadel
Grounded series at a Citadel show near downtown San Jose, CA

These monochromatic images are sourced from power wires for the local light rail (VTA) sub-station on Tasman Rd. I drove by this cluster everyday on a tech job commute for about a year. I swore that when the contract was over I was going to return and photograph all the patterns I saw overhead.

I did just that and four got framed and exhibited at Citadel. One was donated to Works gallery as part of their annual fundraiser.

Donated photograph at Works
Importance of being grounded at Works

The Polaroids come from a project I had in mind for many years. Back when Polaroid was still manufacturing SX-70 instant prints, I shot hundreds of them. I always envisioned enlarging them huge to totally blow out the fidelity (or lack of it).

Polaroids
Enlarged Polaroid prints

This year I began ordering 4 foot test prints on different mounting substrates. To that I ended up scanning a final edit of 14 from hundreds. To see them lined up on the screen ready for output was a fulfilling moment. Having unfinished work in storage was an issue for me for a long time. This was a convergent conclusion of a range of artistic and personal issues.

Passing it on

Now that I have a working art studio, I have a place to show people when visit from out of town. The younger folks are my favorite because they think the place is so weird and like because of that. I share that sentiment.

My French cousins Toullita and Nylane came by for a day and we made zines. Straight up old school xerox zines with glue and stickers and scissors. It was a rad day filled with weird music and messy work.

More locally, I had two younger cousins from San Francisco, Kieran and Jasmina, spend a day with me. They’ve grown up in a world immersed in virtual experiences and “smart” electronics. My choice for them was tinkering with Adafruit Circuit Playground boards.

Tinkering with Circuit Playgrounds
Cousins collaborating on code for apple triggered capacitance synths

They got to mess with Arduino programs designed to make noise and blink lights. At the end they each built capacitive touch synthesizers that we hooked up to apples. Super fun. Later that night we took them to a family dinner and they got to explain what they had made and put on a little demo.

Next up

The wolves are still howling and running. My longtime project to build standalone wolf projections made a lot of progress this year. I had hoped to finish it before the last First Friday of the year, but that wasn’t in the cards.

https://www.instagram.com/p/B4raBM0DS6Z/?utm_source=ig_web_copy_link

Getting something to work in the studio is one thing. Building it so it is autonomous, self-powered, small, and can handle physical bumps, is a whole different game. But, I do have the bike cargo trailer and power assembly ready. The young cousins even got a chance to help test it.

A new instrument I’ve been working on is a Mozzi driven Arduino synth enclosed in an old metal Camel cigarettes tin. It has been an evergreen project this year, offering low stakes programming challenges to tweak the sounds optimize everything for speed.

https://www.instagram.com/p/B5zjHqYDaMi/?utm_source=ig_web_copy_link

One need I had was a precise drill for specific holes. A hand drill could do it, but I had a cleaner arrangement in mind. As luck would have it, another cousin in San Luis Obispo had an extra drill press to donate. Problem was, it was in rough shape and rusted pretty bad.

I brought it back and doused it in PB B’Laster Penetrating Catalyst. That made quick work of the frozen bolts and a range of grinders and rotary brushes handled the other rust. It looks great and is ready to make holes for the Camel synth.

Finis

It’s been a good year artistically. I had some issues with living situations and money, but it all evened out. I’m grateful to have this kind of life and look forward to another year of building weird shit and making freaky noise.

Embers: a breath powered interactive installation celebrating collaboration

Photo by Jerry Berkstresser

It started with incendiary memories: looking at a fading bonfire with friends at the end of a good day, stoking the fire in a pot belly stove, and watching Haitian women cooking chicken over a bed of coals.

I wanted to build something with modern technology that evoked these visceral feelings and associations. Without using screens or typical presentations, the goal was to create an artwork that a wide variety of people could relate to and connect with. It also had to be driven by their own effort.

The initial work began at the Gray Area Foundation for the Arts in San Francisco, during the 2017 Winter Creative Code Immersive. I was learning the mechanics of building interactive art and was looking for a project to bridge my past experience with modern tools.

In January, I travelled to Washington D.C. to photograph the Women’s March and the Presidential Inauguration. They were very different events, but I was struck by the collective effort that went into both. Ideological opposites, they were still the products of powerful collaboration.

When I got back, I heard a lot of fear and anxiety. I had worked in Haiti with an organization called Zanmi Lakay and it had blossomed into effectiveness through group collaboration. I wanted to harness some of that energy and make art that celebrated it.

Embers was born. The first glimpses came from amber hued blinking LEDs in a workroom at Gray Area. 4 months later, the final piece shimmered radiantly in front of thousands of people at the SubZERO art festival in San Jose, CA. In the end, the project itself was a practical testament to collaboration grounded in its conceptual beginnings.

Building the Prototype

For the Gray Area Immersive Showcase, I completed a working study with 100 individually addressable LEDs, 3 Modern Device Wind Sensors (Rev. C), an Arduino Uno, and 100 hand folded rice paper balloons as diffusers. I worked on it alone at my house and didn’t really know how people would respond to it.

When it debuted at the showcase, it was a hit. People were drawn to the fading and evolving light patterns and were delighted when it lit up as they blew on it. I didn’t have to explain much. People seemed to really get it.

The Dare

In early May, I showed a video clip of the study to local gallery owner Cherri Lakay of Anno Domini. She surprised me by daring me to make it for an upcoming art festival called SubZERO. I hesitated, mostly because I thought building the prototype had already been a lot of work. Her fateful words, “you should get all Tom Sawyer on it.”

So, a plan gestated while working on some music for my next album. It was going to be expensive and time consuming and I wasn’t looking forward to folding 1,500 rice paper balloons. A friend reminded me about the concept of the piece itself, “isn’t it about collaboration anyway? Get some people to help.”

I decided to ask 10 people to get 10 more people together for folding parties, with the goal of coming up with 150 balloons at each party. I would give a brief speech and demo the folding. The scheme was simple enough, but became a complex web of logistics I hadn’t counted on.

In the end, it turned out to be an inspiring and fun experience. 78 people helped out in all, with a wide range of ages and backgrounds.

Building Embers

The prototype worked well enough that I thought scaling it up would just be a matter of quantity. But, issues arose that I hadn’t dealt with in the quick paced immersive workshop. Voltage stabilization and distribution, memory limitations, cost escalation, and platform design were all new challenges.

The core of the piece was an Arduino Mega 2560, followed by 25 strands of 50-count WS2811 LEDs, 16 improved Modern Device wind sensors (Rev. P), and 300 ft. of calligraphy grade rice paper. Plenty of trips to Fry’s Electronics yielded spools of wire in many gauges, CAT6 cabling for the data squids, breadboards, and much more.

My living room was transformed into a mad scientist lab for the next month.

Installation

Just a few days before SubZERO, my house lit up in an amber glow. The LED arrays were dutifully glittering and waning in response to wavering breaths. The power and data squids had been laid out and the Arduino script was close to being finished.

I was confident it would work and was only worried about ambient wind at that point. A friend had built a solid platform table for the project and came over the day of the festival to pick up the project. We took it downtown and found my spot on First St. After unloading and setting up the display tent, I began connecting the electronics.

After a series of resource compromises, I had ended up with 1,250 LEDs and around 1,400 paper balloons. The balloons had to be attached to each LED by hand and that took a while. I tested the power and and data connections and laid out the sensors.

Winding the LED strands in small mounds on the platform took a long time and I was careful not to crush the paper balloons. It was good to have friends and a cousin from San Luis Obispo for help.

Lighting the Fire

I flipped the switches for the Arduino assembly, the LED power brick, and then the sensor array. My friends watched expectantly as precisely nothing happened. After a half hour of panicked debugging, it started to light up but with all the wrong colors and behavior. It wasn’t working.

I spent the first night of the two day festival with the tent flap closed, trying to get the table full of wires and paper to do what I had intended. It was pretty damn stressful. Mostly, I was thinking about all the people who had helped and what I’d tell them. I had to get it lit.

Around 10 minutes before midnight (when the festival closed for the night), it finally began to glow amber and red and respond to wind and breath. Around 10 people got to see it before things shut down. But, it was working. I was so relieved.

It turns out that a $6.45 breakout board had failed. It’s a tiny chip assembly that ramps up the voltage for the data line. I can’t recommend the SparkFun TXB0104 bi-directional voltage level translator as a result. The rest of what I have to say about that chip is pretty NSFW.

I went home and slept like a rock.

The next day was a completely different. I showed up a bit early and turned everything on. It worked perfectly throughout the rest of the festival.

People really responded to it and I spent hours watching people laugh and smile at the effect. They wanted to know how it worked, but also why I had made it. I had some great conversations about where it came from and how people felt interacting with it.

It was an amazing experience and absolutely a community effort.

Photo by Jerry Berkstresser

Photo by Joshua Curry

Thanks to all the people and organizations that helped make this a reality:

Grey Area Art Foundation for the Arts, Anno Domini, SubZERO, Diane Sherman, Tim, Brooklyn Barnard, Anonymous, Chris Carson, Leila Carson-Sealy, Cristen Carson, Jonny Williams, Michael Loo, Elizabeth Loo, Kieran Vahle, Jasmina Vahle, Peter Vahle, Kilty Belt-Vahle, Sara Vargas, Sydney Twyman, Annie Sablosky, Martha Gorman, Nancy Scotton, Melody Traylor, Morgan Wysling, Bianca Smith, Susan Bradley, Jen Pantaleon, Guy Pantaleon, Carloyn Miller, Paolo Vescia, Amelia Hansen, Maddie Vescia, Natalie Vescia, Cathi Brogden, Evelyn Lay Snyder, Alice Adams, Lisa Sadler-Marshall, Gena Doty Sager, Mack Doty, Mary Doty, James W. Murray, Greg Cummings, Vernon Brock, Jerry Berkstresser, Lindsey Cummings, Kyle Knight, Liz Hamm, Rebecca Kohn, Shannon Knepper, John Truong, DIane Soloman, Stephanie Patterson, Robertina Ragazza, Sarah Bernard, Jarid Duran, Deb Barba, Astrogirl, Tara Fukuda, CHristina Smith, Yumi Smith, NN8 Medal Medal, Gary Aquilina, Pamela Aquilina, Dan Blue, Chris Blue, Judi Shade, Dave Shade, Margaret Magill, Jim Magill, Brody Klein, Chip Curry, Jim Camp, Liz Patrick, Diana Roberts, Connie Curry, Tom Lawrence, Maria Vahle Klein, Susan Volmer, Jana Levic

 

Joshua Curry is on Instagram, Twitter, and Facebook as @lucidbeaming