Skip navigation

Monthly Archives: July 2006

I just read the most excellent review of Manovich’s The Language of New Media; the review is by Bill Warner.  A lot of what Bill says about this book dovetails with some things I’ve been thinking about lately.

First, I have been incredibly amazed, in a dumb sort of way, by the spirit / body connection in computers: I just can’t see how turning on a switch makes a program execute, a program that will in turn operate little on / off switches on a microchip via 0s and 1s.  Ira said in response to this problem that it’s really no more amazing than what happens when you turn on a light switch.  Here is a relevant passage from Manovich that Warner quotes and explicates:

In retrospect, the shift from a material object to a signal accomplished by electronic technologies represents a fundamental conceptual step towards computer media.  In contrast to a permanent imprint in some material, a signal can be modified in real time by passing it through a filter or filters . . . . [A]n electronic filter can modify the signal all at once . . . .  [A]n electronic signal does not have a singular identity — a particular qualitatively different state from all other possible states. . . .  In contrast to a material object, the electronic signal is essentially mutable. . . . This mutability of electronic media is just one step away from the ‘variability’ of new media. . . . Put differently, in the progression from material object to electronic signal to computer media, the first shift is more radical than the second.  (132-133)

Here is the really interesting part, though, about the shift from electronic mutability to computer variability: the change happens through software.  I remember sitting in my office with a tech guy right after getting my second laptop, trying and trying to figure out why we couldn’t get any sound — and then I found a volume button on the side of the machine.  My new laptop has no such button.  Here is Warner explaining and quoting Manovich:

The increase in range of variation in the digital is accounted for by two factgors: “modern digital computers separate hardware and software” (so for example, changing the volume will just be a software change) and second, “because an object is now represented by numbers, that is, it has become computer data that can be modified by software.  In short a media object becomes ‘soft’ — with all the implications contained in this metaphor” (133).  The mutability of TV (with hue, brightness, vertical hold, etc.) becomes the much wider range of variability for display of a page in a browser window.  (Warner 11).

Both Warner and Manovich argue that what’s significant about the new media is not in fact computers but computers-running-software, that it really doesn’t matter what media instantiates the software at all.  Here’s Manovicth:

[T]he fundamental quality of new media that has no historical precedent [is] programmability.  Comparing new media to print, photography, or television will never tell us the whole story.  For although from one point of view new media is indeed another type of media, from another it is simply a particular type of computer data, something stored in files and databases, retrieved and sorted, run through algorithms and written to the output device.  That the data represent pixels and that this device happens to be an output screen is beside the point. . . .  New media may look like media, but this is only the surface.  (47-48; qtd. Warner 14)

That smacks of idealism and transcendence: code is spirit, and matter / media doesn’t matter at all.  But somehow they are both trying to avoid that trend, analyzed so well by N. Katherine Hayles in her posthuman book.  I’m not sure how, but here is a paragraph in Warner dealing with the problem:

Although the phrase “the computer running software” is redundant, it offers a way to emphasize the way a relatively immaterial thing — software — invades and dematerializes its supposedly hard home, what is conventionally called “hardware” but what we sometimes mistakenly identify as “the computer.”  This is a mistake, not just because hardware needs software the way, by analogy, we might say that the human body needs the communications media of neurons, enzymes and electric signals as a condition of life.  From the beginning of computing, even the hardest components of design — the arrangement of circuits and vacuum tubes, the code embedded on read-only memory, and microprocessors made of silicon — were designed to embed “logic blocks” (like “and,” “or,” “invert”) and algorithms first expressed as software.  In other words, there is a very real sense in which the computer is software all the way down.

A chapter of Manovich’s book that Warner spends a lot of time analyzing is called “Selection,” and it is about how software has made creating into a process of selecting and filtering — a fettering of creativity that may be akin to transforming a writer to a mere reader or worse, critic, assembler of other people’s words into sentences and ideas (she wrote nervously, looking away from her own postings).  This reminds me of Maeda’s attack on the new software programs for programming, as well as Ira’s refusal to let his Flash students click on any of the menu options in the “Actions” portion of Flash (he makes them write out the action script by hand).

Brain power: your wish is the computer’s command from
The sci-fi dream of using brain power to move or speak is now within reach, according to two studies on brain-computer interface technology due out Thursday in the science journal Nature. []

Eric asked some provocative questions in his comment on my last post, a discussion of which warranted a new post.

[eric]:"If art exists in the realm of “messing around” (which I agree), than why code? I thought the benefit of coding was to be able to control more precisely what happens on the screen. If you are looking to mess around and randomly create images to find the one that “feels” right, wouldn’t it be easier to do this using a painting program or some other shortcuts that allow easier ways to mess around?"

Yes, it is easier to use a graphics application to “mess around” rather than having to write code. It is also (usually) easier to take a photograph of a sitter, rather than paint a portrait, or to buy pre-rinsed, pre-cut salad greens to prepare a salad, or to listen to an abridged book on tape or to get all your news off the Fox network. However, degree of ease may not be the correct metric in evaluating the creative process. In easel painting, gravity is an important factor; the viscosity of the paint and the amount of pigment load on the brush are directly dependent on it: Not enough viscosity will cause paint to run, but too much will cause clumping. The process of mixing and thinning is a perpetual balancing act between these 2 extremes. In addition, the treatment of the painting surface (the ground) affects the speed brushes can sweep across it. Larger weave patterns and textured gessoes increase brush drag. Per Erics’s question, these craft issues could be seen as inefficiencies in the painting process, and perhaps to be avoided (if possible.) However, these sorts of factors also provoke creative engagement, demanding a multi-modal response by the artist. Even the solitary creative process is therefore collaborative, through the artist’s relationship with her materials.

Writing code has similar inefficiencies and collaborative possibilities. Although, the inefficiencies are less literally material. Rather than brush drag, there is brain drag, as code abstraction remains a perpetual struggle. Code structures, often bundled as modular processing units, have input and output concerns, which can be conceptualized as brush sweeps, having entry and exit points. The issue of thin (running) vs. thick (clumping) in painting also has a parallel concern with regard to economy of encapsulated processing (a code module can do too much or too little). These inefficiencies in coding, similar to the inefficiencies in other creative processes, demand an engaged response by the coder. And focused engagement is at the heart of any creative process.

[eric]:"It also seems like stretching the definition of art to say plugging random numbers into mathematical equations is akin to a creative process. Ira’s process of creation here seems more similar to tossing a stone into a pond and observing the ripples. Yes, each toss would be unique and the artist can observe the ripples and decide which ones are most pleasing. But so what? This is not a human experience that is likely to touch another human being in such a way to be deemed a great piece of art. If it did work as art, it would be more aptly labeled a beautiful natural wonder of the world rather than a man-made masterpiece. Perhaps that is the role of this digital artform - to visualize the aesthetics of math. It reminds me of the artist who photographed the atom and other natural wonders."

The initial part of this argument, “...stretching the definition of art to say plugging random numbers into mathematical equations is akin to a creative process.”, I’ll leave for someone else to address. One friendly suggestion would be to consider when the definition of art is not being stretched and why? With regard to the stone tossing metaphor and a work of art’s relationship to a viewer, I don’t see these problems being distinct to code art. The pejorative tone of the tossing metaphor presupposes a lack of rigor behind the artist’s discernment. Matisse was supposedly able to generate very successful paper cut-outs, late in his life, with surprising alacrity; of course it took him a lifetime of rigorous work to develop this facility. Additionally, while stone tossing is a singular act, coding, as is painting, often is a time-based creative act, with historical layers of interconnected meaning that build up. While a simple random number generator may not produce an image of much interest, hundreds of such structures, twisted and tweaked, organized around an artist’s aesthetic interests, certainly could. And, as I think my protobytes reveal, such “play” doesn’t necessarily always yield geometrically based or mathematically illustrative work. This work has also been well received, which suggests to me that it might be possible to:"...touch another human being..." with code art.

With regard to “...visualize the aesthetics of math", most western art is, to varying degrees, a visualization of mathematical thinking. Whether dealing with perspective, the golden section, color theory, meter or cubism, artists have always incorporated the mathematical thinking of the day within their work. Lest we forget, a painting is most often based on a regular polygon (quadrangle) and most formal decisions within the painting are, at least in part, based on this fundamental mathematical structure. Code is arguably today’s math, but it is also an emergent creative medium. Thus the urge to explore it’s creative potential seems consistent with artistic historical precedent. In addition, contemporary society is hopelessly dependent upon code, storing our memories in digital photos, connecting us through the net, monitoring our health and wealth, etc. Thus it should also be no surprise that code and coding would be considered, by some artists (this one included), very, very relevant content.

I decided to try to reveal my creative coding process with a series of experiments. I wrote the 1st experiment this morning (when I should have been working on my book,) and it’s a pretty big hack job. However, I think it illustrates the basic process of finding/playing with the code (as a creative medium?) The experiment extends through 5 Processing sketches.

Sketch 1:
I wrote a simple triangle plotting algorithm, using basic trig (remember the Unit circle relationships?)
view sketch 1

Sketch 2:
I parameterized the triangle plotting function, giving it more expressive range
view sketch 2

Sketch 3:
I call the parameterized tri function in a loop using the same trig relationships, which now control the x and y position of each triangle as well as the structure of the individual triangles
view sketch 3

Sketch 4:
I just messed with some of the numbers to create something more visually engaging
view sketch 4

Sketch 5:
I structured the spiral plotting into a function with parameters, which internally handles calling the tri function. I also added Processing’s draw function, which starts an animation thread. From the draw function, I call the spiral function passing in random arguments, generating random output results. I also messed around with the numbers to try to make something interesting to watch.
view sketch 5

The last couple of years my brother, father and I have attempted to do an annual male bonding thing. Not the “malest” of groups, we tend not to tote shotguns or camping gear, but prefer a comfortable and clean b&b, near a lake (likely manmade) and of course conveniently (and affordably) located. For 3 Jewish guys from Long Island, we’re easing ourselves into “nature”. The main purpose of these annual trips is to reconnect and maybe get some exercise. Usually an emotionally charged and too frank discussion marks these gatherings, which we all ultimately find stimulating, once the initial anger subsides. This year we met in Ohio and used Oxford as our base.

The 2 full days we spent together included 9 holes of golf, canoeing, swimming and lots of meals out at restaurants; nature was pretty much kept at bay this year, (at least we did canoe on a real river.) As if our involvement with nature paralleled the psychodynamic, we also, for the most part, steered clear of any too provocative discussions, which I now regret. One conversation though, between my brother Eric and I, that did occur (inspiring this post) was about art and programming.

My brother, a tax lawyer turned businessman who retired in his early 30’s, and I have pursued very different paths in our lives. Eric is goal oriented, extremely pragmatic and in control of his emotions. He approaches life strategically and most often makes highly efficient and fiscally prudent decisions. As an aside, although I am more than 2 years older than Eric we both graduated college and grad school in the same years (he also somehow managed to get 4 degrees, while I got 2.) In the past, Eric’s described life as a game, and he is unquestionably the most facile gamer I have ever known. We grew up playing all sorts of competitive games as kids, from pool to ping pong to nearly any game one could play with a pair of dice, deck of cards or rubber ball. We of course always wagered for large stakes, (leveraging our allowances as far as our imaginations would let us.) It is very likely by the time I was 13, I owed my brother tens of millions of dollars, (which I’m thankful he’s yet to collect on.) I wasn’t a bad gamer, but my brother was brilliantly rational and in complete control of his emotions. I on the other hand was ecstatic when I was winning, distraught when I was losing and prone to huge risk taking– with an emotionally based wagering system. It was very common for our games to end in an argument or worse.

Our discussion about art/code grew out of Eric actually reading some of the earlier posts on this blog. He questioned the notion of code as a creative material, because of its lack of direct correspondence. According to him, code could be written many ways and still produce the same output, so the code itself operated, vis-à-vis aesthetics, tangentially to any concept of craft (my words and interpretation here.) Of course code/coding structure encompasses craft with regard to algorithmic application. However, it is easy to show varied correlation(s) between algorithmic efficiency and aesthetics, making the comparison pointless. Thus if Eric is correct, code is not really a primary aesthetic medium, capable of reflecting creative intention or process, but more of a highly developed tool.

In thinking about this problem, I considered a few examples of code-based creativity, outside of computation. When I was in grad school, I studied with Neil Welliver. Welliver was a major figure in the figurative revival in painting in the 50’s who eventually, along with artists such as Alex Katz, Phillip Pearlstein, and Jack Beal, developed a very signature, even formulaic, style of painting. Rather than approaching painting emotionally or romantically (i.e. abstract expressionism), these artists systematically made highly representational paintings. Welliver actually created his large paintings from the top down, working from the top left corner and proceeding row by row to the lower right corner, never going back and reworking the surface of the painting. Thus Welliver worked algorithmically, in the same way a nested for loop processes data in the rows and columns of a table structure. He also worked directly from a smaller study, which he created originally working plein aire.

In considering Eric’s question, would it matter if Welliver’s larger paintings were painted from the bottom up or even side-to-side. The ordering or procedural application of the marks seems almost inconsequential; (although Welliver would have likely disagreed.) In addition, Welliver worked from a very limited palette of (I think) 8 colors, which he mixed beforehand. Thus his color decisions were also somewhat algorithmically predetermined. However, in spite of his system there is obviously a very direct correspondence between Welliver’s hand making each mark and the form experienced when viewing the painting, and to Welliver the nature of these marks was extremely significant in regard to the value of the work.

Another coded art form is musical composition. In composition, notation is purely symbolic and not intended as the public work of art, unlike the notation (marks) in a painting, or the actual musical performance. Similarly code is also not intended (by most people) as the primary expression. However, in composition there is still a correspondence between the notation and the musical form. Of course interpretation, musicianship, the quality of the instrument all influence execution. But changing the order of the notes will always and directly change the essence of the piece. This is not the case with code. Swapping while loops for for loops will have no impact on the (aesthetic) execution of a program, nor will countless other structural changes. Thus the question still remains unanswered. Or perhaps the question itself is problematic.

Is a direct correspondence necessary for a material to be an effective creative medium? If I can code 10 programs that all create the exact same image, does that somehow negate the value of the code as an effective medium? If code is not a medium, but instead a tool, then what is the medium? Is the problem that a specific programming language is too high level and thus includes too many pathways to the same low-level bit processing? Is the bit processing (down at the memory locations) actually the medium?

So I’ll continue hacking away, using code as a creative medium, in spite of my brother’s inspired confusion and doubts. At least I was smart enough not to make a wager with him this time.

Last night at a 4th of July gathering, I met a friendly and thoughtful Botanist. His research involved invasive plants. He was using historical satellite imagery to help identify the effects of invasive plant density on the surrounding flora ecology. Part of his challenge was in actually identifying the plant density within the satellite imagery, which was often below the forest canopy. We spoke a little about the image analysis problem, which of course got me all hot and bothered. In my typical manic and undisciplined fashion, I began asking lots of questions about the imaging algorithms, etc, and I felt him pulling back some. He said he himself wouldn’t be doing the imaging stuff, but a co-investigator (a Geographer) would be dealing with that. I felt we had had our first fight. (Although we had only known each other for about 5 minutes.) I seem to have this effect on certain people, which got me thinking about another recent conversation I had with a Geologist friend, who does do a lot of his own image analysis work. This friend loves speaking about the algorithmic parts of his research, which involves ground water flow. I asked the Geologist if he now prefers the computer flow modeling work more than the actual Geology. He very honestly said he did. (But I could tell it was a guilty pleasure.)

This seems to be a growing problem in many disciplines (at least the perception is); the tools and processes of analysis and production are often as (or even more) fascinating than the original inquiry. I don’t really see this as a “confusing the forest for the trees” sort of problem that should justify a fear or guilt response, but rather see it as indicative of a much deeper and more fundamental problem, namely “technoprocessguiltophobia”. (OK, so this is a silly word that I made up.) At the core of technoprocessguiltophobia is the belief that “serious” work should be the byproduct of focus, intent and most importantly prioritization. Getting lost in a process, such as creating some imaging algorithm, is not the priority when you are focused on the “big picture” (plant ecology,) and there are others who are experts in this sort of thing: Regardless if you’re fascinated by a (tactical) process, it’s essential to remain focused on the goals of your (strategic) research and stay the course. However, what probably got you interested in your original discipline to begin with was this very sort of digressive fascination (if you’re lucky.)

Our evolving digital tools are fascinating. They are also decadent, often silly, expensive and very time-sucking. But of course the technology is really us, so arguably techno-phobia a is a strange sort of self-loathing (another blog post here?) The same urge to understand the structure of a leaf in the 18th century is what’s driving us to model it with bits in the 21st century, which is also the same urge to develop the tools to model it. Only we seem to give value to certain activities that have historical (romantic) precedence–so a naturalist’s traditional process of looking, touching, measuring, etc seems more benign (guilt-free) than a bunch of detached bit shifting happening in a computer. I am in no way suggesting that the naturalist should stop going to nature and doing fieldwork. However, I am strongly recommending that the naturalist let herself get lost and fascinated in the process, even if the process falls outside of their current expertise or seems digressive to the original research mission. (Obviously this might not work for junior faculty in the current system, myself included.) Ultimately, I would argue that the sense of play (re)introduced into their process, through process-focus, will invigorate their research, broaden their perspective and help eradicate the disciplinary silos strangling universities (especially the segments that don’t have access to huge government/industry funding streams.)

Well back to working on my very digressive (and currently pretty buggy) 3D rendering engine, something 8 years of classical painting training prepared me well for?

Laura lent me The Visual Mind: Art and Mathematics (1995), which I’ve been thumbing through, in between struggling to develop a homespun 3D engine for my book. I’ve been working on the same stupid back face culling problem for what seems like 2 weeks now, and well, I’m not feeling too much love for my a-analytical brain lately. (Perhaps the problem could be due to a faulty basal ganglia).
I think the book (The Visual Mind, not mine) is significant as an historical landmark, documenting a relatively early period of development in computer graphics. I don’t think the book actually deals very much with art and mathematics though. The book does deal with interesting mathematical ideas and the problems (especially pre-CGI) of visualizing them, and it presents the math in an inspiring way, which is, I guess, its connection to art. However, I’d challenge the idea that a beautiful bronze cast of some 4D math function is a synthesis of math and art. Rather, I’d place the work solidly within mathematics, its “newness” perhaps extending the rendering/drafting medium (pencil, compass…casting, graphing calculator, etc) I don’t mean to devalue the beautiful works discussed in the book, generated by equally beautiful mathematical ideas. But, I think the rational framework underlying the work/process imposes itself too pragmatically–precluding the process from getting too “out of control” for unexpected a-rational “stuff” to be found.
I think a true fusion between math and art would lead to work that wouldn’t necessarily illustrate the math/art connection, or at least not as didactically as many of the book’s examples. Some good arguments could be made for the contemplative and aesthetic aspects of the processes/works in the book as justification for their signification as “Art”. However, I would argue back (not too forcefully) that the term “craft” encompasses notions of both the contemplative and aesthetic. For mathematical work to approach art, for me, it needs to transcend the rational. I think Escher, albeit very didactically, illustrated this in his work. It is the impossibility of his seemingly mathematically created and precise worlds that is fascinating (beautiful.) Of course many artists throughout history have created transcendent mathematically inspired/based art: Piero Della Francesca’s Resurrection immediately comes to mind. I remember seeing the piece in Sansepolcro and being overwhelmed, and I assure you I wasn’t thinking about perspective or any other mathematical idea at the time. The raw human emotional drama of the piece subsumed any of the (not trivial) math underlying it.
Ironically, it seems through the development of powerful applied mathematical tools (computation) that new math/art integration struggles ensue, which will make another interesting thing to consider in a future post (preferably not by me.)
Back to obsessing about my basal ganglia.