Brain power: your wish is the computer’s command from PhysOrg.com
The sci-fi dream of using brain power to move or speak is now within reach, according to two studies on brain-computer interface technology due out Thursday in the science journal Nature. [...]

Eric asked some provocative questions in his comment on my last post, a discussion of which warranted a new post.

[eric]:"If art exists in the realm of “messing around” (which I agree), than why code? I thought the benefit of coding was to be able to control more precisely what happens on the screen. If you are looking to mess around and randomly create images to find the one that “feels” right, wouldn’t it be easier to do this using a painting program or some other shortcuts that allow easier ways to mess around?"

Yes, it is easier to use a graphics application to “mess around” rather than having to write code. It is also (usually) easier to take a photograph of a sitter, rather than paint a portrait, or to buy pre-rinsed, pre-cut salad greens to prepare a salad, or to listen to an abridged book on tape or to get all your news off the Fox network. However, degree of ease may not be the correct metric in evaluating the creative process. In easel painting, gravity is an important factor; the viscosity of the paint and the amount of pigment load on the brush are directly dependent on it: Not enough viscosity will cause paint to run, but too much will cause clumping. The process of mixing and thinning is a perpetual balancing act between these 2 extremes. In addition, the treatment of the painting surface (the ground) affects the speed brushes can sweep across it. Larger weave patterns and textured gessoes increase brush drag. Per Erics’s question, these craft issues could be seen as inefficiencies in the painting process, and perhaps to be avoided (if possible.) However, these sorts of factors also provoke creative engagement, demanding a multi-modal response by the artist. Even the solitary creative process is therefore collaborative, through the artist’s relationship with her materials.

Writing code has similar inefficiencies and collaborative possibilities. Although, the inefficiencies are less literally material. Rather than brush drag, there is brain drag, as code abstraction remains a perpetual struggle. Code structures, often bundled as modular processing units, have input and output concerns, which can be conceptualized as brush sweeps, having entry and exit points. The issue of thin (running) vs. thick (clumping) in painting also has a parallel concern with regard to economy of encapsulated processing (a code module can do too much or too little). These inefficiencies in coding, similar to the inefficiencies in other creative processes, demand an engaged response by the coder. And focused engagement is at the heart of any creative process.

[eric]:"It also seems like stretching the definition of art to say plugging random numbers into mathematical equations is akin to a creative process. Ira’s process of creation here seems more similar to tossing a stone into a pond and observing the ripples. Yes, each toss would be unique and the artist can observe the ripples and decide which ones are most pleasing. But so what? This is not a human experience that is likely to touch another human being in such a way to be deemed a great piece of art. If it did work as art, it would be more aptly labeled a beautiful natural wonder of the world rather than a man-made masterpiece. Perhaps that is the role of this digital artform - to visualize the aesthetics of math. It reminds me of the artist who photographed the atom and other natural wonders."

The initial part of this argument, “...stretching the definition of art to say plugging random numbers into mathematical equations is akin to a creative process.”, I’ll leave for someone else to address. One friendly suggestion would be to consider when the definition of art is not being stretched and why? With regard to the stone tossing metaphor and a work of art’s relationship to a viewer, I don’t see these problems being distinct to code art. The pejorative tone of the tossing metaphor presupposes a lack of rigor behind the artist’s discernment. Matisse was supposedly able to generate very successful paper cut-outs, late in his life, with surprising alacrity; of course it took him a lifetime of rigorous work to develop this facility. Additionally, while stone tossing is a singular act, coding, as is painting, often is a time-based creative act, with historical layers of interconnected meaning that build up. While a simple random number generator may not produce an image of much interest, hundreds of such structures, twisted and tweaked, organized around an artist’s aesthetic interests, certainly could. And, as I think my protobytes reveal, such “play” doesn’t necessarily always yield geometrically based or mathematically illustrative work. This work has also been well received, which suggests to me that it might be possible to:"...touch another human being..." with code art.

With regard to “...visualize the aesthetics of math", most western art is, to varying degrees, a visualization of mathematical thinking. Whether dealing with perspective, the golden section, color theory, meter or cubism, artists have always incorporated the mathematical thinking of the day within their work. Lest we forget, a painting is most often based on a regular polygon (quadrangle) and most formal decisions within the painting are, at least in part, based on this fundamental mathematical structure. Code is arguably today’s math, but it is also an emergent creative medium. Thus the urge to explore it’s creative potential seems consistent with artistic historical precedent. In addition, contemporary society is hopelessly dependent upon code, storing our memories in digital photos, connecting us through the net, monitoring our health and wealth, etc. Thus it should also be no surprise that code and coding would be considered, by some artists (this one included), very, very relevant content.

I decided to try to reveal my creative coding process with a series of experiments. I wrote the 1st experiment this morning (when I should have been working on my book,) and it’s a pretty big hack job. However, I think it illustrates the basic process of finding/playing with the code (as a creative medium?) The experiment extends through 5 Processing sketches.

Sketch 1:
I wrote a simple triangle plotting algorithm, using basic trig (remember the Unit circle relationships?)
view sketch 1

Sketch 2:
I parameterized the triangle plotting function, giving it more expressive range
view sketch 2

Sketch 3:
I call the parameterized tri function in a loop using the same trig relationships, which now control the x and y position of each triangle as well as the structure of the individual triangles
view sketch 3

Sketch 4:
I just messed with some of the numbers to create something more visually engaging
view sketch 4

Sketch 5:
I structured the spiral plotting into a function with parameters, which internally handles calling the tri function. I also added Processing’s draw function, which starts an animation thread. From the draw function, I call the spiral function passing in random arguments, generating random output results. I also messed around with the numbers to try to make something interesting to watch.
view sketch 5

The last couple of years my brother, father and I have attempted to do an annual male bonding thing. Not the “malest” of groups, we tend not to tote shotguns or camping gear, but prefer a comfortable and clean b&b, near a lake (likely manmade) and of course conveniently (and affordably) located. For 3 Jewish guys from Long Island, we’re easing ourselves into “nature”. The main purpose of these annual trips is to reconnect and maybe get some exercise. Usually an emotionally charged and too frank discussion marks these gatherings, which we all ultimately find stimulating, once the initial anger subsides. This year we met in Ohio and used Oxford as our base.

The 2 full days we spent together included 9 holes of golf, canoeing, swimming and lots of meals out at restaurants; nature was pretty much kept at bay this year, (at least we did canoe on a real river.) As if our involvement with nature paralleled the psychodynamic, we also, for the most part, steered clear of any too provocative discussions, which I now regret. One conversation though, between my brother Eric and I, that did occur (inspiring this post) was about art and programming.

My brother, a tax lawyer turned businessman who retired in his early 30’s, and I have pursued very different paths in our lives. Eric is goal oriented, extremely pragmatic and in control of his emotions. He approaches life strategically and most often makes highly efficient and fiscally prudent decisions. As an aside, although I am more than 2 years older than Eric we both graduated college and grad school in the same years (he also somehow managed to get 4 degrees, while I got 2.) In the past, Eric’s described life as a game, and he is unquestionably the most facile gamer I have ever known. We grew up playing all sorts of competitive games as kids, from pool to ping pong to nearly any game one could play with a pair of dice, deck of cards or rubber ball. We of course always wagered for large stakes, (leveraging our allowances as far as our imaginations would let us.) It is very likely by the time I was 13, I owed my brother tens of millions of dollars, (which I’m thankful he’s yet to collect on.) I wasn’t a bad gamer, but my brother was brilliantly rational and in complete control of his emotions. I on the other hand was ecstatic when I was winning, distraught when I was losing and prone to huge risk taking– with an emotionally based wagering system. It was very common for our games to end in an argument or worse.

Our discussion about art/code grew out of Eric actually reading some of the earlier posts on this blog. He questioned the notion of code as a creative material, because of its lack of direct correspondence. According to him, code could be written many ways and still produce the same output, so the code itself operated, vis-à-vis aesthetics, tangentially to any concept of craft (my words and interpretation here.) Of course code/coding structure encompasses craft with regard to algorithmic application. However, it is easy to show varied correlation(s) between algorithmic efficiency and aesthetics, making the comparison pointless. Thus if Eric is correct, code is not really a primary aesthetic medium, capable of reflecting creative intention or process, but more of a highly developed tool.

In thinking about this problem, I considered a few examples of code-based creativity, outside of computation. When I was in grad school, I studied with Neil Welliver. Welliver was a major figure in the figurative revival in painting in the 50’s who eventually, along with artists such as Alex Katz, Phillip Pearlstein, and Jack Beal, developed a very signature, even formulaic, style of painting. Rather than approaching painting emotionally or romantically (i.e. abstract expressionism), these artists systematically made highly representational paintings. Welliver actually created his large paintings from the top down, working from the top left corner and proceeding row by row to the lower right corner, never going back and reworking the surface of the painting. Thus Welliver worked algorithmically, in the same way a nested for loop processes data in the rows and columns of a table structure. He also worked directly from a smaller study, which he created originally working plein aire.

In considering Eric’s question, would it matter if Welliver’s larger paintings were painted from the bottom up or even side-to-side. The ordering or procedural application of the marks seems almost inconsequential; (although Welliver would have likely disagreed.) In addition, Welliver worked from a very limited palette of (I think) 8 colors, which he mixed beforehand. Thus his color decisions were also somewhat algorithmically predetermined. However, in spite of his system there is obviously a very direct correspondence between Welliver’s hand making each mark and the form experienced when viewing the painting, and to Welliver the nature of these marks was extremely significant in regard to the value of the work.

Another coded art form is musical composition. In composition, notation is purely symbolic and not intended as the public work of art, unlike the notation (marks) in a painting, or the actual musical performance. Similarly code is also not intended (by most people) as the primary expression. However, in composition there is still a correspondence between the notation and the musical form. Of course interpretation, musicianship, the quality of the instrument all influence execution. But changing the order of the notes will always and directly change the essence of the piece. This is not the case with code. Swapping while loops for for loops will have no impact on the (aesthetic) execution of a program, nor will countless other structural changes. Thus the question still remains unanswered. Or perhaps the question itself is problematic.

Is a direct correspondence necessary for a material to be an effective creative medium? If I can code 10 programs that all create the exact same image, does that somehow negate the value of the code as an effective medium? If code is not a medium, but instead a tool, then what is the medium? Is the problem that a specific programming language is too high level and thus includes too many pathways to the same low-level bit processing? Is the bit processing (down at the memory locations) actually the medium?

So I’ll continue hacking away, using code as a creative medium, in spite of my brother’s inspired confusion and doubts. At least I was smart enough not to make a wager with him this time.

Last night at a 4th of July gathering, I met a friendly and thoughtful Botanist. His research involved invasive plants. He was using historical satellite imagery to help identify the effects of invasive plant density on the surrounding flora ecology. Part of his challenge was in actually identifying the plant density within the satellite imagery, which was often below the forest canopy. We spoke a little about the image analysis problem, which of course got me all hot and bothered. In my typical manic and undisciplined fashion, I began asking lots of questions about the imaging algorithms, etc, and I felt him pulling back some. He said he himself wouldn’t be doing the imaging stuff, but a co-investigator (a Geographer) would be dealing with that. I felt we had had our first fight. (Although we had only known each other for about 5 minutes.) I seem to have this effect on certain people, which got me thinking about another recent conversation I had with a Geologist friend, who does do a lot of his own image analysis work. This friend loves speaking about the algorithmic parts of his research, which involves ground water flow. I asked the Geologist if he now prefers the computer flow modeling work more than the actual Geology. He very honestly said he did. (But I could tell it was a guilty pleasure.)

This seems to be a growing problem in many disciplines (at least the perception is); the tools and processes of analysis and production are often as (or even more) fascinating than the original inquiry. I don’t really see this as a “confusing the forest for the trees” sort of problem that should justify a fear or guilt response, but rather see it as indicative of a much deeper and more fundamental problem, namely “technoprocessguiltophobia”. (OK, so this is a silly word that I made up.) At the core of technoprocessguiltophobia is the belief that “serious” work should be the byproduct of focus, intent and most importantly prioritization. Getting lost in a process, such as creating some imaging algorithm, is not the priority when you are focused on the “big picture” (plant ecology,) and there are others who are experts in this sort of thing: Regardless if you’re fascinated by a (tactical) process, it’s essential to remain focused on the goals of your (strategic) research and stay the course. However, what probably got you interested in your original discipline to begin with was this very sort of digressive fascination (if you’re lucky.)

Our evolving digital tools are fascinating. They are also decadent, often silly, expensive and very time-sucking. But of course the technology is really us, so arguably techno-phobia a is a strange sort of self-loathing (another blog post here?) The same urge to understand the structure of a leaf in the 18th century is what’s driving us to model it with bits in the 21st century, which is also the same urge to develop the tools to model it. Only we seem to give value to certain activities that have historical (romantic) precedence–so a naturalist’s traditional process of looking, touching, measuring, etc seems more benign (guilt-free) than a bunch of detached bit shifting happening in a computer. I am in no way suggesting that the naturalist should stop going to nature and doing fieldwork. However, I am strongly recommending that the naturalist let herself get lost and fascinated in the process, even if the process falls outside of their current expertise or seems digressive to the original research mission. (Obviously this might not work for junior faculty in the current system, myself included.) Ultimately, I would argue that the sense of play (re)introduced into their process, through process-focus, will invigorate their research, broaden their perspective and help eradicate the disciplinary silos strangling universities (especially the segments that don’t have access to huge government/industry funding streams.)

Well back to working on my very digressive (and currently pretty buggy) 3D rendering engine, something 8 years of classical painting training prepared me well for?

Laura lent me The Visual Mind: Art and Mathematics (1995), which I’ve been thumbing through, in between struggling to develop a homespun 3D engine for my book. I’ve been working on the same stupid back face culling problem for what seems like 2 weeks now, and well, I’m not feeling too much love for my a-analytical brain lately. (Perhaps the problem could be due to a faulty basal ganglia).
I think the book (The Visual Mind, not mine) is significant as an historical landmark, documenting a relatively early period of development in computer graphics. I don’t think the book actually deals very much with art and mathematics though. The book does deal with interesting mathematical ideas and the problems (especially pre-CGI) of visualizing them, and it presents the math in an inspiring way, which is, I guess, its connection to art. However, I’d challenge the idea that a beautiful bronze cast of some 4D math function is a synthesis of math and art. Rather, I’d place the work solidly within mathematics, its “newness” perhaps extending the rendering/drafting medium (pencil, compass…casting, graphing calculator, etc) I don’t mean to devalue the beautiful works discussed in the book, generated by equally beautiful mathematical ideas. But, I think the rational framework underlying the work/process imposes itself too pragmatically–precluding the process from getting too “out of control” for unexpected a-rational “stuff” to be found.
I think a true fusion between math and art would lead to work that wouldn’t necessarily illustrate the math/art connection, or at least not as didactically as many of the book’s examples. Some good arguments could be made for the contemplative and aesthetic aspects of the processes/works in the book as justification for their signification as “Art”. However, I would argue back (not too forcefully) that the term “craft” encompasses notions of both the contemplative and aesthetic. For mathematical work to approach art, for me, it needs to transcend the rational. I think Escher, albeit very didactically, illustrated this in his work. It is the impossibility of his seemingly mathematically created and precise worlds that is fascinating (beautiful.) Of course many artists throughout history have created transcendent mathematically inspired/based art: Piero Della Francesca’s Resurrection immediately comes to mind. I remember seeing the piece in Sansepolcro and being overwhelmed, and I assure you I wasn’t thinking about perspective or any other mathematical idea at the time. The raw human emotional drama of the piece subsumed any of the (not trivial) math underlying it.
Ironically, it seems through the development of powerful applied mathematical tools (computation) that new math/art integration struggles ensue, which will make another interesting thing to consider in a future post (preferably not by me.)
Back to obsessing about my basal ganglia.

I was reading the section of Ira’s book, Foundation Processing, called “Motion.” In it, he says:

In many ways, computer monitors are animation machines, as the pixels being continuously refreshed on the screen, at your monitor’s refresh rate, are a form of animation (not a terribly exciting one). This fact that animation can happen in front of your eyes and be wholly undetectable is significant.

I hadn’t really understood, not REALLY, what was at stake in “flickering signifiers” (N. Katherine Hayles, “Virtual Bodies and Flickering Signifiers,” Electronic Culture, ed. Timothy Druckrey). But this passage made sense for me of a phenomenon I encountered a while ago at the library. I’m not sure it’s still true, but, when King Library first put up surveillance cameras, I remember looking at them and seeing the computer screens just behind me. They looked like TV screens used to look late at night, after stations went off the air, lines cycling up over and over again; the quickly moving lines were all that could be seen on a gray screen. Then I would turn around, look at the library’s computer screens with my own eye, and see no lines, no cycling, a perfectly normal desktop picture. I’d turn back to the video camera, and it would still be there. Things are moving all over the place on these screens, but our eyes distill them into still images.

I at first posted about how this idea surfaces in sci fi (and poetry).  More interesting — Ira says “significant,” but it could also be sinister – is to think about the effects of distilling moving, flickering, animated images into still images.  Is anything subtracted from our view in the process of distillation?  Is the process completely “natural,” or informed by ideology, by habits of seeing? It strikes me that the eye’s propensity to render moving images still is very much like the capitalist’s desire to render relationships as commodities.  Not all reduction is bad — some of it is essential to sanity, if nothing else.  But does the eye’s desire to reduce movement to stillness affect our ideas of beauty or even promote the use of Ritalin for ADHD? (sorry for such a huge leap)  Counteracting the sinister is why it is so important to teach code; knowing a bit of it enables looking under the hood.

Here is Ira's challenge: "Can a mission statement be brief, amorphic and semantically mutable: i.e. 'We do research across all disciplines on the impact of interactive technology. (Come play with us!).'"  Also, he wants us to say what we've got that makes us distinctive.

What's distinctive about us?  We can cross disciplines more easily at Miami: "we've got interdisciplinarity." I wanted to transform "interdisciplinarity" or "interactivity / responsiveness / implicatedness among disciplines" from a noun into a verb: "Our Research Mission as transdisciplinary faculty is to explore the implications of our respective fields for each other's fields insofar as their disciplinary processes occur in interactive media."  It's still ugly, and that means something.  How about "Our research mission is to implicate our fields in each other's work through interactive media."  Oiy, what does that mean?

Ira's second challenge: "What if your field is interactive media?"  Ira's proposed revision: "IMS Research is a very, very, very dynamic, sloppy, collaborative, confusing, alogical, passionate, playful, exploratory, terribly exciting and (hopefully) insanely irreverent space. Come play with us."

At Ian's birthday party on Sat. Robin and I forgot our camera (Thanks to Glenn P. for a save here.) It caused me to reflect some on the obsessive need to document our lives and then "replay" them. It's almost as if, without the documentation, we can't be sure an event occured or how we felt during it. I felt guilt and sadness for forgetting the camera; this would be time permanently lost. This idea of capturing time is obviously not new, but we are now capturing far more time (as a culture) than actually passes. Thus, it would take more than a lifetime to (re)view a lifetime. And this hyper-documentation is exponentially increasing, expanding the present and even creating history in the moment.
I don't know what we're going to do with all this time, since unfortunately there is both too little and too much at the same time. Perhaps we need time munchers–little bots that roam around and eat time, but only the time poorly spent. This way our replayed histories could be even more glorious.

This is from a letter I sent to the faculty of the Interactive Media Studies program at Miami, regarding the development of an IMS research mission statement.

A point that I would like to see stressed in the mission is the deeper fundamental impact that computation brings to media studies. I think it is easy to lose the forest for the trees here—aided in large part by the software industry. The tendency, both in and out of academia, is to see computation (the glass half full view) as a facilitating and democratizing tool/force, which in itself is not a bad thing. However, this somewhat superficial perspective I think misses the much more significant potential of computation as a distinctive medium and even alternative "intelligence"*. The tool/force perspective relies on an industrial age paradigm-technology enhances, frees, empowers, etc. Computation fits neatly within this continuum of being another incremental step toward full automation. Again, this is a valid and useful signification. However, it seems also to me to be overly egocentric-the individual remains in control of the machine; ideally it serves his/her wishes (ultimately completely.)

In contrast, computation can be a much less agreeable and cooperative agent. As a tool, it is arguably highly inefficient. Consider the actual costs of system development, operations, training, deployment and maintenance, vis-a-vis work productivity. Of course current human demand for "toys" makes these numbers work, but if we try to separate the fulfillment of actual human needs from wants, I wonder how productive computer technology really is (yeah, yeah, I know this is wimpy lefty thinking.) Considering computation as a medium offers a significant break from the older productivity model. As a medium, computation offers universal mutability; it can model/process/analyze/generate visual, aural, tactile, kinetic, textual, etc. data–it can take the form of (perhaps) everything. Thus, when we segment to: digital media, digital humanities, etc, we are expressing a bias based on older disciplinary boundaries rather than any limit inherent within the medium. This is something to consider seriously. And filtering further to digital video, 3D, multimedia, etc seems highly problematic.

A current problem is how to get a literal grip on/in/around the medium. The software industry has stepped in to categorize/granularize the medium for us, and make a whole lot of moola in the process. They have been very effective in confusing the mechanism for the medium. Epistemology is not a high priority in the engineering process, so our software tools don't ask why only how, and we keep buying up the stuff, even if most of us never use 9/10ths of the features in these bloated tools-yet we dutifully upgrade every cycle. I would argue that to stop this cycle and get a "grip" on the medium we need greater fluency in the actual computation medium–not simply facility in manipulating commercial software applications. And this is best achieved through developing programming literacy. I believe IMS should be at the forefront of this-not to train computer scientists, but rather to provide essential education. If we want our students to be able to parse, interpret, analyze, etc. shouldn't they have that proficiency in perhaps the single most dominant and controlling medium in their lives? And obviously I think IMS research should blaze a path in this area. Let me stress again that this is not about low-level computer science based research, but rather fluency in the computation medium and work/research that reflects this fluency and hopefully helps define our emerging field.

* I'll offer some additional half-baked thoughts on "alternative intelligence" in a future post.