What is all this about “codebending” and patching?
What would happen if you could control a musical instrument with Ms. Pac-Man, or have Katamari Damacy edit novels? Or connect any piece of software to any other piece of software? Codebending is connecting.
However it isn’t just about arbitrary connections – it also aims to challenge the way we look at the software in our lives, and explore the bizarre and exciting things that happen when you push programs and systems to their absolute limits. What happens to Space Invaders if you generate a new enemy 50 times a second and control motion of your ship with brainwaves? Is this not a joyful form of play?
Codebending is the act of connecting computer software in atypical ways, and exploring the potential of systems.
A program can be codebent if it can:
- send messages when things happen in the program
- receive external messages that control the activities of the program
By routing these messages around, you can:
- connect seemingly unrelated computer programs in interesting ways
- patch programs back into themselves to create feedback loops
- employ general utilities and control structures like oscillators, metronomes/clocks, sequencers, etc
- incorporate any hardware as input (think: game controllers, midi, touch screens, microcontrollers, etc. anything that can be digitized)
This means that video games can play other video games, and become generative art. Oscillators can drive web browsers. Poetry can write music. The combinations are basically limitless; being able to make these connections opens all sorts of expressive, aesthetic, and/or generative possibilities. Most importantly, building these connections is a lot of fun.
Codebending treats the creation and exploration of systems as play.
What do you mean by systems? Structures?
Structures are playgrounds are games are instruments are systems.
All software, simple or complex, has a structure. This includes computer games, office software suites, operating system interfaces, music creation programs, web browsers, etc. All of these share something in common: there are rules in place that cause things to change in the system. They’re all structures that can be controlled.
Additionally, many changes and events take place in a program as one interacts with it. If broadcast, these events can be used to control other systems. Parameters in one program (say, the screen coordinates of a character in a game) can be routed to control something else in another piece of software.
If abstracted in this way, software becomes a playground. Programs are systems asking to be controlled, but also controllers themselves. These bendable, malleable play structures each have unique voices, and sing curious things. When patched to communicate with each other, surprising things happen.
Okok. So what is required for codebending?
Two things are required of the software:
- patchability: the software should send information out for other software, and listen for incoming information.
- message/information formatting: programs that are quite different need a standard way to interact with each other.
In other words, codebending requires a notion of input and output “patch points” so that systems can be connected. These patch points need to transmit information formatted in a standard/universal way, such that unrelated/different systems can talk to each other. This also means that whoever writes the software must implement these facilities (although creative tinkerers do have some techniques for strapping makeshift patchpoints onto existing software).
Codebending is like the routing paradigm of modular synthesizers, but applied to software of all types.
Can you say more about these connections?
By making connections between programs, you’re creating larger interactions between pieces of computer software. This treats the activities of any piece of software as a controller. Why should one interact with a video game using a traditional gamepad – why not engage a game with another game? Why shouldn’t the events in a first person shooter be able to trigger copy and paste events in a word processor? Why can’t the size of a window in an operating system be controlled with a piano, or a sine wave?
This kind of external control is often referred to as “modulation.” A more traditional example: when a finger wiggles on a guitar string, it modulates (ie: controls and modifies) the pitch of the fretted note, and thus contributes a vibrato to the sound.
Directly put: by routing connections between software, you’re using information in one computer program to modulate/change another. This routing can be configured in advance, or more interestingly, changed dynamically on the fly. Realtime patching across software systems can produce unexpected and beautiful emergences.
Thus far, most of the codebendable programs I’ve developed utilize video game mechanics. Each game can be thought of as similar to a module in a modular synthesizer. Classic arcade games make for great abstractions, and they each have specific voices when used to modulate other programs. Additionally, when these games are driven and controlled by complex external modulation sources, they become fountains of generative art.
But aren’t these systems too different to talk to each other?
In order to make it possible for divergent systems to communicate, I’ve standardized a very simple notion of “signal” that can be routed from one patch point (aka an OSC address) to another. This “signal” is just a numerical range. By building everything to use values between 0.0 and 1.0 in OSC messages, one can bridge the gaps between different pieces of software.
Thus, if you want to do something in a game, rather than using a gamepad controller, you have to send a message (in the form of a numerical value) on an address corresponding to the thing you want to do.
How do these programs send and receive messages?
I use OSC (OpenSoundControl.org). It is a simple networking protocol that allows for this notion of “patch points,” and thus provides a universal way of communicating with other games/software. There are plenty of other protocols available for allowing software to send and receive messages, but OSC is particularly friendly and suitable for codebending. Most modern programming languages have simple OSC libraries.
By using OSC, patch points can be hooked up to anything that can send or receive OSC messages. For example, if you have a midi controller and know how to route OSC, you have a whole universe of control at your fingers. Also, many existing pieces of software can send and receive OSC messages, and most modern programming languages have libraries that are easy to use.
I’ve written several games and programs with these patch points – visithttp://www.paperkettle.com for more programs.
What values do I use?
I’ve standardized my systems to use float values from 0.0 to 1.0. They are do not go higher or lower than this range. This provides a universal notion of “signal” that allows very different programs to interact.
- For continuous ranges (for example, the x coordinate of some object onscreen) send values between 0.0 and 1.0
- For trigger events, one simply sends a 1.0.
- For gate behavior, 1.0 corresponds to HIGH, 0.0 corresponds to LOW
On what addresses?
It depends on each individual piece of codebendable software and how it implements and names the sending/receiving of OSC messages. These addresses are your patch points, and specified by whoever writes the given program.
This is all very odd. Can you give another concrete example?
Imagine you’re playing a version of Super Mario Brothers that allows for codebending. You wouldn’t use a controller to move Mario or make him jump – you’d send messages to OSC addresses. (This might sound tricky, but it is really easy to quickly route things with software like Pure Data or Max/MSP, and opens a lot of new territory.)
Suppose you want to make Mario jump. You’d find the address that corresponds to making Mario jump, and you’d send a message (say, a numerical value of 1.0) to that address. The system would be listening on that address, receive the message, and make Mario jump. Now imagine patching this to other things, like a metronome/clock, such that Mario jumps every time the metronome ticks.
Some quick points of clarification: to do this, you’d obviously need a version of Super Mario that was written to support codebending. When I talk about sending messages on addresses, I’m referring to OSC addresses. OSC support needs to be coded into the program by the software designer for codebending to be possible.
Now lets assume that when Mario lands, he kills a Goomba. If the game designer included a patch point that corresponds to the death of a Goomba, a message will be sent out on an OSC address specified by the designer. A numerical value of 1.0 would make a good value to send out in such an event. The codebender can then route this information to another program, or even back into the Mario game itself.
Maybe you’d patch things such that the color of the sky in Mario changes every time a Goomba is killed. Or maybe you’d patch this information into a synthesizer to make music. Or perhaps generate a new Mario enemy every time a drum machine plays a kick drum. Or hook the game’s clock up to an oscillator, such that space-time in the Mushroom Kingdom speeds up and slows down following the value of a sine wave. I made a codebendable text editor – imagine remixing a text work by patching your text editor such that it cuts or pastes when a koopa shell bounces off of a warp pipe.
The point is not in any one of these (perhaps contrived) examples – it is the sheer joy of bending these systems into interesting new territories. It is astoundingly fun to have unforseen, hilarious, and/or gorgeous patterns emerge as previously different systems converge.
Codebending is the joy of systems as play.