Skip navigation

Category Archives: code & art

I’ve created two (VERY) simple semantic visualizations based on a search for terms defined as positive or negative. I was originally planning on dynamically generating word lists using WordNet or some other dictionary api. However, good-old has a much wider and deeper word well (including returns from WordNet). I looked into programatically parsing the returned url (which I may eventually do), but for now have generated the word lists manually (I know, I know, this is admitting some defeat). The visualizations plot a linear and then radial gradient based on lines containing the pos or neg terms. I keep track of the number of pos/neg terms, should a line contain multiple terms (some do). Each line (or concentric ring) overlaps its neighbors and is translucent, allowing some optical color mixing. Arbitrarily– red is pos and blue is neg. The gray is neutral.


Linear Visualization

Radial Visualization

The next visualization (yes, yes it’s a day late) plots an array of protobits (pixels) based on all the characters in the poem (including spaces). The syntactic elements are the colored pixels in their actual location in the poem. The poem is read (so to speak) by an arthropod-esque bot that moves across the characters. The arthropod’s motion is affected by the respective syntactic elements it crosses. Any characters the arthropod head touches are displayed in the bottom right of the window. The syntactic elements are also displayed in the center and remain there until the next element is reached. The arthropod, built as a series of interconnected springs, is a metaphor for the stream of reading that is affected by syntax, as well as its own inertia.

Link to syntactic visualization

I created my first visualization today for the project. Keeping things simple I plotted word usage count as a particle graph. (I also settled on the term protoBits).

Link to Visualization

My process: All the words in the poem were sorted alphabetically, and duplicate words were counted. I plotted the unique words along the x-axis and the duplicate words along the y-axis. Each particle initially occupies a unique position, but to keep things more interesting I made them dynamic and added a random jitter to their x-position upon impact with the ground. Although there is acceleration along the y-axis, there is no gravity/friction–so the system never stabilizes. Moving the mouse over a particle reveals the word plotted. The higher particle columns represent the more common words. Particles turn orange once they’ve been rolled over.

My goal will be to try to create a unique visualization (of increasing complexity) each day leading up to the conference, (so please stop by tomorrow 😉 )

I’ve been able to get the WordNet API integrated into a simple Java app. One amusing side-note is that I got stuck for a day trying to get the WordNet .jar file to run in my Java app. After spending a few hours of unsuccessful Googling, I picked up my own book, in which I explained (to myself) how to solve the problem. So what I thought originally would be the more time consuming and challenging parts of the project–parsing and semantic relationships–have been (at least initially) fairly straightforward. The larger challenge that looms before me is what the heck I’m going to do with all this data.

The problem is not actually what to do, but rather what to do in the next 2 weeks, prior to MLA. I wish I could just explore this material without the burden of deadline. This was supposed to be how I was going to spend my sabbatical this fall–yeah, right!

My thoughts about the visualization process today are to begin with single cell creatures and work my way up. I’ve been thinking about a name for these fundamental organisms: microbots, micro-protobytes, microbytes, protobits, protobots. My thought for these initial creatures is single pixels that bounce in 1 dimension: distance = word usage. I know this is fairly boring, but I feel like I need to begin simply and fundamentally. I will post a few Processing sketches of these initial tests next.

It’s time this blog is resuscitated.

Fortunately I have something to write about, as I am beginning a very interesting collaboration with Laura on the visualization of 18th century romantic poetry-a subject I am severely ignorant about. Here is a recent note I sent to Laura:

Sent Dec 12, 2007

… Some initial thoughts I want to share:

1. I’ve been thinking and working on parsing:
Thus far I’ve been able to input the poem and generate some relatively simple statistical data about overall syntax and word usage (i.e. number of occurrences of terms). I could (and will) parse deeper and collect phoneme groups, prefixes, suffixes, etc as well. In addition, I really want more semantic “meat”, so I’ve downloaded WordNet ( a “lexical database for the English Language” developed at Princeton). WordNet should (I’m hoping) allow me to query all terms against a simplified semantic interface. For example, I would like to be able to identify any term that relates to birth or death or love or hate, etc. This seems the only logical way to approach mapping semantics. Of course, once I collect buckets of terms based on these more general concepts, finer semantic filtering could occur recursively (man that sounds pretentious-put it on the poster “fer sure”!). For example, all the terms that semantically connect to birth, could be further separated–giving forth of an idea, creating a life-form, heritage, lineage, noun vs verb, etc., etc.

If time permits (hah!) it would be good to find some other dictionary api’s; for example aural data (relating to phonemes), etymology, etc.

Once all this mess of data is collected and statistics are generated, I’ll connect the data to a visualization tool. For now, I’m thinking about using my protobyte forms as sort of a conceptual armature (genus perhaps?). I would love to have the poem visualizations/protobytes motile in 3D (ultimately evolving)-–poetry creating virtual life!!!

Eric asked some provocative questions in his comment on my last post, a discussion of which warranted a new post.

[eric]:"If art exists in the realm of “messing around” (which I agree), than why code? I thought the benefit of coding was to be able to control more precisely what happens on the screen. If you are looking to mess around and randomly create images to find the one that “feels” right, wouldn’t it be easier to do this using a painting program or some other shortcuts that allow easier ways to mess around?"

Yes, it is easier to use a graphics application to “mess around” rather than having to write code. It is also (usually) easier to take a photograph of a sitter, rather than paint a portrait, or to buy pre-rinsed, pre-cut salad greens to prepare a salad, or to listen to an abridged book on tape or to get all your news off the Fox network. However, degree of ease may not be the correct metric in evaluating the creative process. In easel painting, gravity is an important factor; the viscosity of the paint and the amount of pigment load on the brush are directly dependent on it: Not enough viscosity will cause paint to run, but too much will cause clumping. The process of mixing and thinning is a perpetual balancing act between these 2 extremes. In addition, the treatment of the painting surface (the ground) affects the speed brushes can sweep across it. Larger weave patterns and textured gessoes increase brush drag. Per Erics’s question, these craft issues could be seen as inefficiencies in the painting process, and perhaps to be avoided (if possible.) However, these sorts of factors also provoke creative engagement, demanding a multi-modal response by the artist. Even the solitary creative process is therefore collaborative, through the artist’s relationship with her materials.

Writing code has similar inefficiencies and collaborative possibilities. Although, the inefficiencies are less literally material. Rather than brush drag, there is brain drag, as code abstraction remains a perpetual struggle. Code structures, often bundled as modular processing units, have input and output concerns, which can be conceptualized as brush sweeps, having entry and exit points. The issue of thin (running) vs. thick (clumping) in painting also has a parallel concern with regard to economy of encapsulated processing (a code module can do too much or too little). These inefficiencies in coding, similar to the inefficiencies in other creative processes, demand an engaged response by the coder. And focused engagement is at the heart of any creative process.

[eric]:"It also seems like stretching the definition of art to say plugging random numbers into mathematical equations is akin to a creative process. Ira’s process of creation here seems more similar to tossing a stone into a pond and observing the ripples. Yes, each toss would be unique and the artist can observe the ripples and decide which ones are most pleasing. But so what? This is not a human experience that is likely to touch another human being in such a way to be deemed a great piece of art. If it did work as art, it would be more aptly labeled a beautiful natural wonder of the world rather than a man-made masterpiece. Perhaps that is the role of this digital artform - to visualize the aesthetics of math. It reminds me of the artist who photographed the atom and other natural wonders."

The initial part of this argument, “...stretching the definition of art to say plugging random numbers into mathematical equations is akin to a creative process.”, I’ll leave for someone else to address. One friendly suggestion would be to consider when the definition of art is not being stretched and why? With regard to the stone tossing metaphor and a work of art’s relationship to a viewer, I don’t see these problems being distinct to code art. The pejorative tone of the tossing metaphor presupposes a lack of rigor behind the artist’s discernment. Matisse was supposedly able to generate very successful paper cut-outs, late in his life, with surprising alacrity; of course it took him a lifetime of rigorous work to develop this facility. Additionally, while stone tossing is a singular act, coding, as is painting, often is a time-based creative act, with historical layers of interconnected meaning that build up. While a simple random number generator may not produce an image of much interest, hundreds of such structures, twisted and tweaked, organized around an artist’s aesthetic interests, certainly could. And, as I think my protobytes reveal, such “play” doesn’t necessarily always yield geometrically based or mathematically illustrative work. This work has also been well received, which suggests to me that it might be possible to:"...touch another human being..." with code art.

With regard to “...visualize the aesthetics of math", most western art is, to varying degrees, a visualization of mathematical thinking. Whether dealing with perspective, the golden section, color theory, meter or cubism, artists have always incorporated the mathematical thinking of the day within their work. Lest we forget, a painting is most often based on a regular polygon (quadrangle) and most formal decisions within the painting are, at least in part, based on this fundamental mathematical structure. Code is arguably today’s math, but it is also an emergent creative medium. Thus the urge to explore it’s creative potential seems consistent with artistic historical precedent. In addition, contemporary society is hopelessly dependent upon code, storing our memories in digital photos, connecting us through the net, monitoring our health and wealth, etc. Thus it should also be no surprise that code and coding would be considered, by some artists (this one included), very, very relevant content.

I decided to try to reveal my creative coding process with a series of experiments. I wrote the 1st experiment this morning (when I should have been working on my book,) and it’s a pretty big hack job. However, I think it illustrates the basic process of finding/playing with the code (as a creative medium?) The experiment extends through 5 Processing sketches.

Sketch 1:
I wrote a simple triangle plotting algorithm, using basic trig (remember the Unit circle relationships?)
view sketch 1

Sketch 2:
I parameterized the triangle plotting function, giving it more expressive range
view sketch 2

Sketch 3:
I call the parameterized tri function in a loop using the same trig relationships, which now control the x and y position of each triangle as well as the structure of the individual triangles
view sketch 3

Sketch 4:
I just messed with some of the numbers to create something more visually engaging
view sketch 4

Sketch 5:
I structured the spiral plotting into a function with parameters, which internally handles calling the tri function. I also added Processing’s draw function, which starts an animation thread. From the draw function, I call the spiral function passing in random arguments, generating random output results. I also messed around with the numbers to try to make something interesting to watch.
view sketch 5

The last couple of years my brother, father and I have attempted to do an annual male bonding thing. Not the “malest” of groups, we tend not to tote shotguns or camping gear, but prefer a comfortable and clean b&b, near a lake (likely manmade) and of course conveniently (and affordably) located. For 3 Jewish guys from Long Island, we’re easing ourselves into “nature”. The main purpose of these annual trips is to reconnect and maybe get some exercise. Usually an emotionally charged and too frank discussion marks these gatherings, which we all ultimately find stimulating, once the initial anger subsides. This year we met in Ohio and used Oxford as our base.

The 2 full days we spent together included 9 holes of golf, canoeing, swimming and lots of meals out at restaurants; nature was pretty much kept at bay this year, (at least we did canoe on a real river.) As if our involvement with nature paralleled the psychodynamic, we also, for the most part, steered clear of any too provocative discussions, which I now regret. One conversation though, between my brother Eric and I, that did occur (inspiring this post) was about art and programming.

My brother, a tax lawyer turned businessman who retired in his early 30’s, and I have pursued very different paths in our lives. Eric is goal oriented, extremely pragmatic and in control of his emotions. He approaches life strategically and most often makes highly efficient and fiscally prudent decisions. As an aside, although I am more than 2 years older than Eric we both graduated college and grad school in the same years (he also somehow managed to get 4 degrees, while I got 2.) In the past, Eric’s described life as a game, and he is unquestionably the most facile gamer I have ever known. We grew up playing all sorts of competitive games as kids, from pool to ping pong to nearly any game one could play with a pair of dice, deck of cards or rubber ball. We of course always wagered for large stakes, (leveraging our allowances as far as our imaginations would let us.) It is very likely by the time I was 13, I owed my brother tens of millions of dollars, (which I’m thankful he’s yet to collect on.) I wasn’t a bad gamer, but my brother was brilliantly rational and in complete control of his emotions. I on the other hand was ecstatic when I was winning, distraught when I was losing and prone to huge risk taking– with an emotionally based wagering system. It was very common for our games to end in an argument or worse.

Our discussion about art/code grew out of Eric actually reading some of the earlier posts on this blog. He questioned the notion of code as a creative material, because of its lack of direct correspondence. According to him, code could be written many ways and still produce the same output, so the code itself operated, vis-à-vis aesthetics, tangentially to any concept of craft (my words and interpretation here.) Of course code/coding structure encompasses craft with regard to algorithmic application. However, it is easy to show varied correlation(s) between algorithmic efficiency and aesthetics, making the comparison pointless. Thus if Eric is correct, code is not really a primary aesthetic medium, capable of reflecting creative intention or process, but more of a highly developed tool.

In thinking about this problem, I considered a few examples of code-based creativity, outside of computation. When I was in grad school, I studied with Neil Welliver. Welliver was a major figure in the figurative revival in painting in the 50’s who eventually, along with artists such as Alex Katz, Phillip Pearlstein, and Jack Beal, developed a very signature, even formulaic, style of painting. Rather than approaching painting emotionally or romantically (i.e. abstract expressionism), these artists systematically made highly representational paintings. Welliver actually created his large paintings from the top down, working from the top left corner and proceeding row by row to the lower right corner, never going back and reworking the surface of the painting. Thus Welliver worked algorithmically, in the same way a nested for loop processes data in the rows and columns of a table structure. He also worked directly from a smaller study, which he created originally working plein aire.

In considering Eric’s question, would it matter if Welliver’s larger paintings were painted from the bottom up or even side-to-side. The ordering or procedural application of the marks seems almost inconsequential; (although Welliver would have likely disagreed.) In addition, Welliver worked from a very limited palette of (I think) 8 colors, which he mixed beforehand. Thus his color decisions were also somewhat algorithmically predetermined. However, in spite of his system there is obviously a very direct correspondence between Welliver’s hand making each mark and the form experienced when viewing the painting, and to Welliver the nature of these marks was extremely significant in regard to the value of the work.

Another coded art form is musical composition. In composition, notation is purely symbolic and not intended as the public work of art, unlike the notation (marks) in a painting, or the actual musical performance. Similarly code is also not intended (by most people) as the primary expression. However, in composition there is still a correspondence between the notation and the musical form. Of course interpretation, musicianship, the quality of the instrument all influence execution. But changing the order of the notes will always and directly change the essence of the piece. This is not the case with code. Swapping while loops for for loops will have no impact on the (aesthetic) execution of a program, nor will countless other structural changes. Thus the question still remains unanswered. Or perhaps the question itself is problematic.

Is a direct correspondence necessary for a material to be an effective creative medium? If I can code 10 programs that all create the exact same image, does that somehow negate the value of the code as an effective medium? If code is not a medium, but instead a tool, then what is the medium? Is the problem that a specific programming language is too high level and thus includes too many pathways to the same low-level bit processing? Is the bit processing (down at the memory locations) actually the medium?

So I’ll continue hacking away, using code as a creative medium, in spite of my brother’s inspired confusion and doubts. At least I was smart enough not to make a wager with him this time.