Installing The Show

Today was the first day of setup for the design show. I did most of the work I needed to get done including mounting my prints onto presentation board & trimming them out, painting the pedestal white, hanging the prints and setting up the iMac. I still have to figure out what to do with the Wiimote charging dock, but it's a relief to have most of the work finished.

Ready to print, instructions

It's time to install the show this weekend so I'm frantically wrapping things up. I set all of the pages I want to print out onto a single document to make it easier to print using the big printer in the design lab. I also spent time finishing the instructions poster so people will know what to do with the Wiimote. The program is also ready to go (at least good enough for the show - I could probably work on it for another semester and make it do some really cool things, but it does what I need it to do for the show).

Routing the second Wiimote

I spent some time working on the Max patcher for the loop machine. I got the Wiimote to select the machines using the D-pad like I wanted to, but when I started using the volume and speed features, multiple machines were being affected. After looking around at the schematic, I realized that I had messed up a bit of the wiring. Once I sorted it out it worked perfectly. I made a second control set for the second Wiimote and began to map it to machines 5-8. It will consist of different controls since 5-8 are going to be samples instead of loops. I also spent some time organizing little parts of the patch, moving a few things and deleting unnecessary objects. I also made it so the master volume will automatically set itself as soon as the patch is opened.

Routing Wiimote data, analyzing audio signal

I spent some more time working on the program over the past week. I set up all of the Wiimote buttons that I'm using so far to be routed through a switch system that only sends the incoming signals to specific loop machines. This way, one Wiimote can control 4 different loop machines. The user will be able to switch between the 4 machines by using the D-pad on the Wiimote - up is assigned to one, down to another, etc.

I also started trying to analyze the resulting audio right before it gets sent to the speakers, but I'm not sure that I'll be able to read what the notes are - I'll probably have to work with the overall signal's amplitude/frequency.

I did a little work making the instruction manual as well - so far I've just started to turn the Wiimotes into vector art.

Routing the Wiimote data, streamlining

I mostly worked on programming since tuesday. I cleaned up my schematics by encapsulating large sections of it using "p" objects, and by sending/receiving data without patch cords using the "s" and "r" objects. What you see above the blue segment are the volume and speed controls being split into 4 different outputs that are selected by pressing either up, down, left or right on the D-pad of the Wiimote. The fifth button (attached to the grey switch) tells it not to send any information unless a button is being held - A for the speed control, B for the volume. The "send" objects send the output values to "receive" objects in the 4 different loop machines. This will let the user play with any of the four different loops, picking which one via whatever direction it's assigned to on the D-pad.

Finalized Synesthetic show card

I played around with my card design for a while and decided not to change it too much. I'm happy with the little changes made. I've been super busy lately, and should have some more time to work on the programming this weekend.

Vistaprint is teasing me with this:

Loop machine controls, show card

I did some work routing the controls of one of the Wiimotes to the first four loop control modules in my sampler in Max. It's a little confusing but coming along. I figured out how to add a master volume control that can still allow individual volumes to be manipulated using the Wiimote.

I'm experimenting with using the XYZ inputs to control things like volume and loop speed by holding a button and making a gesture. I also figured out how to encapsulate patchers (shown below) in order to avoid a huge mess of patch cords and obscure programming commands. Any box that starts with "p" is an encapsulated patcher - you have to open it to see its contents.

I worked on my card designs for a while. I think the back needs to be revisited, and have some ideas for adding a few more lines to the front.

Loop Machine/Sampler

After dissecting several existing patchers and watching countless tutorial videos without coming across exactly what I need, I decided to make my own sampler from the ground up. Using the sample~ and groove~ objects, I wound up making a loop machine that works with 8 different sound files simultaneously - 4 are meant to be loops, while the other 4 are meant to be samples (single hit fills), but all 8 slots can be set to loop. Next I need to tie the Wiimote to various control functions in the patch, and then develop a video patch.

Max objects title design

I spent some time thinking about a title for my project, and came across a phenomenon called synesthesia. Synesthesia is when more than one of a person's individual senses are connected. For example, some people who suffer from synesthesia read numbers, letters and words as having specific colors - "3" always appears green, "B" looks red, etc. Others may equate numbers or letters with specific places in 3 dimensional space or time. Since my project is shaping up to be a combination of sound, visuals and physical movement, Synesthetic seems like an appropriate (working) title.

I then went into Max and started placing objects and routing patch cords in order to create a visual design rather than anything functional in the program. I ended up designing the show title typographically by creating letterforms out of the Max elements. Some of them function in some way or another, but many are standalone and solely exist for the sake of creating the overall letterform.

Synth manipulating video

I'm starting to understand Max a bit better, having spent a few hours looking at how the Wiimote patch and this synthesizer patch are constructed. I spent a while trying to get the output values from the synthesizer to control certain inputs in a video patch, and I'm getting the results I want. The most difficult thing moving forward is going to be figuring out how to map individual keys to specific video clips. I took some time to understand how the Wiimote patch works and should be able to connect it to video in the same way as I did with the keyboard. I was having second thoughts about using the Wiimote last week, but the more I look into it the more I want to have it be a part of my project. I almost feel the opposite now - I may not bother with the keyboard and just stick to the Wiimote, adding the third dimension of movement/gesture to the connection I'm trying to make.

Making things in Max

I've been learning how to use Max from watching tutorials and downloading & studying existing patches (programs). Many patches are available for free online and vary from useful widgets & tools for dropping into patches to complete, functional patches that function either in Max or as standalone software. I've been watching lots of examples of synthesizer, sampler and loop machine patches. I've also watched many tutorials involving the Wiimote, but I'm not sure if I'm compelled to use it because it's useful or just because it would be cool. I've been considering changing my input device from the Wiimote to my Micron, a small scale synth that can act as a midi controller.

Whichever I decide on, I'm thinking the program I make will set the input up to play audio clips of each note on the 12-note scale - onto corresponding keys on the Micron, or specific buttons/combinations on the Wiimote (if I use the Wiimote I'd probably set it up as a loop machine instead of a note-for-note synth).

The data produced by the Wiimote/Micron would also have to control a second patch that plays video in sync with the gestures/keystrokes being used to create the music. I've been making video programs and playing around with effects for the past few days, and am starting to figure out a plan on turning the audio into something visual, based off of the sonochromatic scale. I found lots of resources online offering free video loops for VJ shows. I grabbed some and played around with them in Max, and noticed that there's many that have pretty solid color schemes - some stay basically red or blue or purple, etc. I'm pretty sure that I can eventually make something that connects the notes being played to one of 12 different clips to represent that note's color. Using Max's crossfade, several clips can be spliced together at once to represent chords/multiple notes being hit at once.

In the end, with any luck you'll be able to create music using at least one of these devices that simultaneously controls live video in colors that relate to the notes being played.

Max/MSP/Jitter

I've been watching tutorials on the software I got called Max/MSP/Jitter. It's basically a programming language that is visual based instead of code based. You create programs by making schematics in Max, which you'll see at the end of the "color gun" video below. MSP and Jitter and programs that use the schematics you created in Max to alter audio and visuals using pretty much any input method imaginable. I found some really cool videos of things people have done with this language, some more relevant to my project than others.

Using a Wiimote as an input

Wii Synth

Good example of how the schematics work

I'm going to start looking for specific tutorials now that I understand how the program (basically) works. It would be interesting to make something that creates both music and an image corresponding to the music at the same time made from input from a Wiimote since it has so many sensor & button mapping capabilities.

- Ian

Thesis Progress

My thesis project at school is about linking music and design. I plan to deconstruct the connection between the visual and audible spectrums, and use this knowledge to both turn pieces of music into design, and designs into pieces of music. I also plan on doing the same thing with randomly generated or aesthetically pleasing designs to create odd pieces of music, and plan on using ambient noise and sound clips to create more random designs.

Today, I played around with some different programs that are designed to analyze .mp3 and .wav files, turning them into .midi files, which are basically simple instrument sounds programmed into the computer, often on several layers that reproduce different instruments. I used the program on a relatively simple song from my itunes library. The resulting midi file sounded alright, but it's definitely not accurate enough to capture everything that's going on. Overall you get a general sense of the song, but there are parts where the program didn't know what to do and made up something that wasn't there, or left it out all together. It might be interesting to use when turning ambient noise/sound effects into scores, but for now it's not doing me much good.

Instead of turning mp3s into midis, I just went onto freemidi.org and looked around for something that everyone would know, and found the theme to Super Mario. I found a second program that opens midi files and shows you their different layers as scores. You can get rid of layers you don't want, so I got rid of the percussion track and an annoying "coin" track. The program shows you what notes the computer has been programmed to play transposed over actual sheet music, so I took some general shapes from the lead track and let them influence how I arranged the rectangles when there weren't all three.

Interactive Installation, Color to Note Chart

Here's some stuff I've been checking out since we talked about the potential for making an interactive installation on Tuesday.

Blair Neal's Device

Blair Neal made a device that uses a projector wired to a computer to sense drawings (on projector sheets) as they pass over a sensor. The computer turns them into whatever sounds have been assigned to different colors/positions on the paper.

He gave his code to someone else who made their own version of his machine that works with normal paper. He uses mostly prints/illustrations to trigger the sounds rather than dots and lines like in Blair's.

This is the next page of the manual I've been working on to help show the relationship between sound and light wave frequencies. It needs more work - I should have started from E so infrared and ultraviolet would be at the ends of the spectrum rather than overlapping:

Sonochromatic Scale, Chord Structure

I began creating a manual documenting what I've learned about the connections between music and design. I've started off by relating notes to colors in the light spectrum, based off the system that Neil Harbisson uses (see my post below). I then began to analyze how basic chords are composed, and started translating it into the visual world.

Here are the first few pages of my music/color manual, modeled off of Neil Harbisson's Sonochromatic Scale. I'll be adding more as my research continues.

12 Note scale represented in corresponding colors, showing lighter/darker versions meant to represent higher/lower octaves.

Visual representations of notes combining to make all basic major/minor chords. The most basic chords are made up of triads - three notes combined to make one harmonious chord.

Music Project - Where I Started

I'll be posting some links to things that I've been watching / listening to in my research. This is a really interesting Ted talk with an artist named Neil Harbisson who was born completely colorblind. At an early age he developed a sensor that turns whatever he's looking at into sound. It changed the way he looks at the world, and has been working in reverse ever since he became familiarized with the connection. It's about 10 minutes long, definitely worth the watch if you have time.

Neil Harbisson TED Talk

Neil Harbisson on Wikipedia