Grainita is a new musi­cal instru­ment for novices, draw­ing inspi­ra­tions from sand art, ambi­ent music and sto­chas­tic com­po­si­tion. It attempted to study new forms of inter­ac­tion for music, com­bined audio-visual art par­a­digms, com­puter vision and machine impro­vi­sa­tion through algo­rith­mic composition.

(E dit — For a brief period, Grainita is under devel­op­ment for dis­play at the Guth­man Com­pe­ti­tion at GaT­ech. For now, I have an audio demo of the gen­er­a­tive system.

Sand art’ is the name given to the prac­tice of draw­ing or paint­ing with sand, as in the fol­low­ing video.

Notice that this art form has a strong tem­po­ral dimen­sion, the process of cre­at­ing the final image is as much part of the art as the final image itself. It is gen­er­ally abstract in nature, draw­ing on infer­ences and assump­tions by the viewer. In these prop­er­ties, sand art bears large resem­blances to ambi­ent music. More­over, sand art is usu­ally accom­pa­nied by unre­lated music. This inspired me to cre­ate a uni­fied art par­a­digm, a new musi­cal instru­ment for novices with a unique inter­ac­tion scheme.

An impor­tant goal while design­ing grainita was a DIY ethos and cheap mate­r­ial cost. The ini­tial setup con­sisted of a ‘light box’, made with a hol­lowed out IKEA table cost­ing approx­i­mately 7 USD. A strip of LED lights was installed, after spray paint­ing the inte­ri­ors a glossy white for increased reflec­tion. The top of the table was replaced by a translu­cent acrylic sheet. (Thanks to Prof. Tripp Edwards and the power tools of the Design Shop in the Col­lege of Archi­tec­ture, Geor­gia Tech for help­ing out with this!)

Grainita uses com­puter vision to cap­ture the sand on the sur­face and cor­rect­ing for errors, using a USB web­cam to con­nect to Max/MSP. A flaw was found in the afore­men­tioned design; the cam­era was posi­tioned over­head, and cap­tured the hands of the artist while they per­formed with sand. There were some poten­tial solu­tions, but the over­head was too large, and a redesign was necessary.

The mod­i­fi­ca­tion was to place the cam­era under­neath the acrylic sur­face, held up by the four legs of the orig­i­nal IKEA table. This proved to be a sim­ple and func­tional solu­tion, lead­ing to ‘ver­sion 1.1’.

The video feed from the cam­era was processed and tes­sel­lated into a 4X4 grid on the com­puter. The pres­ence of sand in the sub-frame was detected, and if this crosses a cer­tain thresh­old, a MIDI note is gen­er­ated. The exact note gen­er­ated depends on the scale/mode and the root note that the user can set using a graph­i­cal slider interface.

Another goal was to facil­i­tate the mod­i­fi­ca­tion of con­trol para­me­ters with­out hav­ing to stop a per­for­mance and reach out to a com­puter; an Arduino con­nected to a push­but­ton matrix was used to sim­plify this.

Grainita also ‘learns’ from a user’s inputs, and can accom­pany the human per­for­mance. It uses a sec­ond order Markov process to store the note infor­ma­tion; the dura­tion infor­ma­tion is com­puted by an inde­pen­dent first order Markov process. The user can turn on and off the accom­pa­ni­ment, as well as change the tempo at which it plays back using the Arduino and the key­pad matrix. The com­puter impro­vi­sa­tion mod­ule gen­er­ates MIDI as well, which is routed to a DAW using a sep­a­rate chan­nel, allow­ing the user to cre­ate mul­ti­ple lay­ered sounds.

Grainita also has a built-in looper func­tion, which ‘records’ user played mate­r­ial and plays it back in an infi­nite loop. The per­former can turn this on or off, and sim­i­lar to the com­puter impro­vi­sa­tion mod­ule, this is routed to a dif­fer­ent MIDI chan­nel as well, result­ing in a total of 3 chan­nels a user can harness.

Sand is an inter­est­ing medium for visual art, it is tan­gi­ble, but can be shaped quickly and eas­ily. There is some­thing that attracts peo­ple to it, many acquain­tances reached out just to touch the sand I was work­ing with. Work­ing with sand on a rec­tan­gu­lar sur­face pro­vided a 2 dimen­sional spa­tial para­me­ter, which was used to gen­er­ate MIDI con­trol data.

For this, the video feed was divided into two halves, allow­ing for manip­u­la­tion of two out­put MIDI chan­nels. In each half, the posi­tion of the cen­troid of the largest ‘blob’ was cal­cu­lated, and scaled into a 7 bit range for MIDI. These data can be used to con­trol soft­ware syn­the­sizer para­me­ters, inde­pen­dent of the other modules.

There are a few mod­i­fi­ca­tions and improve­ments I had in mind, includ­ing dis­card­ing the Arduino and Key­pad inter­face in favor of tou­chOSC. There are a lot of para­me­ters that can be extracted from the sand pat­terns, and this could con­trol music in unique ways. Watch this page, there is more to come!

Tags: , , , , , , ,