DOCUMENTING THE REINVENTION OF TEXT:

The Importance of Failure

by JOHN UNSWORTH

If an electronic scholarly project can't fail and doesn't produce new ignorance, then it isn't worth a damn.

Contrast the spirit of this assertion with the rhetoric that characterizes much of what we say, write, and read about the subject of electronic text, the World-Wide Web, and information technology in general: the trope is one of change, invention, evolution, with overtones of progress and improvement, and with undertones of inevitability and universality. We meet this trope in mass-media news and advertising about computers and communications, in the promotional literature of our educational institutions, in scholarly books and articles about hypertext and digital libraries, and in grant proposals for electronic scholarly projects which aim, or claim, to break new ground, undertake pilot projects, provide models for the future.

Featured Lessons: Collaboration Takes More Than E-Mail A Catechism for Digital Publishing Solving the Dilemma of Copyright Protection Before Its Time Pioneering an Online Newspaper

Elucidated: Under Evolution

Lieb: S-l-o-w Pages

Front Page

My focus here will be on the academic part of what is clearly a larger cultural trend, and specifically on hypertext projects and hypertext theory, as they address the subject of transformative change, but I will be holding these projects and this theory to an extrinsic standard, a standard suggested by the rhetoric of invention, discovery, and progress -- namely the standard of science. I think I can predict the objections to this exercise, but in spite of those, I believe this is a worthwhile experiment, and a worthwhile discussion, because it may help us to sharpen distinctions among different kinds of writing about hypertext, and because it may help us to arrive at some principles for evaluating both theoretical and applied work in this area of research.

I should acknowledge that my remarks here are the direct result of being asked a question for which I didn't have a very good answer, about a year ago. At a conference at the University of Maryland, Neil Fraistat (whose Romantic Circles Web site some of you may know) asked me if there were any writing on specific humanities hypertext projects that was neither promotional nor anecdotal, but that reported and analyzed and theorized the experience of constructing such a project. I could think of a couple of examples, but only a couple, and none perfectly apt. The conversation with Neil progressed to the topic of the importance of reporting and analyzing failure in any research activity, humanistic or scientific, and to the patterns of funding that discouraged such reporting and analysis. I owe whatever illuminations emerge in the following to that conversation, and I take it as an emblematic instance of a research opportunity: a question for which there should be an answer, for which one could imagine an answer, but for which no very good answer was at present to be found.

Certain Limits, Uncertain Cases
At the most basic level, the level of survival, it is a given that resources -- in academia as elsewhere -- are limited, and that we struggle for those resources in the form of institutional support, outside grant funding, and release time. Given those limited resources, we are obviously obliged, for practical as well as intellectual reasons, to argue for our projects and our programs. In short, there is a kind of evolutionary pressure at work in the transformation of the book: some projects will survive, others will not; some theories will flourish, others will wither. If we hope that rationality rather than sheer force might guide that process, then the only rational course, for both the proposers and the funders of such projects, is to declare and defend our evaluative criteria, particularly when we consume or allocate resources that might otherwise go elsewhere.

By the same token, and before going further in my discussion, I would say that any academic or funding activity bears the same responsibility: it is no more a justification to say "it has always been done" than to say "it has never been done." In either case, we need to know why it should be done, and we need to know how we will determine whether we succeeded or failed in the endeavor.

I'd like to begin, then, with two theses from Sir Karl Popper, the founder of the philosophical school known as critical rationalism, a school of thought from which many of the arguments in what follows will be derived:

First Thesis: We know a great deal. And we know not only many details of doubtful intellectual interest, but also things which are of considerable practical significance and, what is even more important, which provide us with deep theoretical insight, and with a surprising understanding of the world.

Second Thesis: Our ignorance is sobering and boundless. . . With each step forward, with each problem which we solve, we not only discover new and unsolved problems, but we also discover that where we believed that we were standing on firm and safe ground, all things are, in truth, insecure and in a state of flux. ("The Logic of the Social Sciences" in The Positivist Dispute in German Sociology, 1976)

Popper's theses, and his writings in general, do a fine job of expressing something that I want to emphasize today, in the context of the Transformations of the Book, namely, on the one hand, the importance -- the utility -- of what we do know and, on the other hand, the ephemeral, contingent, transitional character of that knowledge -- and therefore, the need for experiment, the indispensability of mistakes, and the necessity of recognizing, documenting, and analyzing our failures.

Transformation as Evolution
There is no question that the book, or more properly text technology -- what Jay Bolter calls "writing space" -- is currently undergoing a major transformation. Inasmuch as we think of that transformation as progress, or hope that it will be, those changes are implicitly being treated as evolutionary. It has been observed that

[a]ny theory of evolution is about processes of change. An extra requirement for an evolutionary theory is that purely random and entirely time-reversible patterns are excluded; evolution concerns exclusively change that is, at least statistically, irreversible. To qualify, irreversible change must entail processes that lead to emergence, or at least the persistence, of ordered structure in space and time. (E. Laszlo, The New Evolutionary Paradigm, 1991, xxiii.)

Evolution is our name for a positive, unidirectional change -- an alteration in the direction of something better, where better is defined as more complex, more ordered, more useful, more adaptive, more fit to a particular purpose. The test of whether a transformation qualifies as an evolution, then, is whether or not it improves on what it changes, and does so in a way that external forces are likely to reward and reinforce.

Is Change Improvement?
We know from observation -- of our own aging bodies, for example -- that not all changes are improvements. So if we are advocating a change, or 

"If a statement cannot possibly be proven false, then it can't be considered a scientific statement."

participating in one, we ought to be deeply concerned with evaluative questions. In the case of the transformation of the book, the question could be phrased, "Does hypermedia improve on the book?" And that is a question that ought (in principle) to be answerable, with some combination of empirical evidence and rational argument. But in order to gather such evidence, or make such arguments, we would first need to establish evaluative criteria. What might such criteria look like, in the case of hypertext projects or hypertext theory?

Before attempting to answer that question, I should point out that the criteria by which evidence would be selected and on which arguments would be based will be rather different in these two cases: Theory has one set of responsibilities, and craft has another. But the two are, or ought to be, connected and mutually responsive. Where experimental endeavors are concerned, theory ought to be able to explain, predict, and produce practical results, and practice ought to provide the occasion to test, implement, modify, or falsify theoretical assertions.

Evaluative Criteria in Hypertext Theory
Hypertext theory is a recent but broad and interdisciplinary field: It includes literary scholars of many different periods and specialties, philosophers and sociologists, computer scientists, user-interface and human-computer interaction experts, librarians, publishers, and practitioners. Hypertext theory is still sorting out its relationship to the even broader fields of literary theory, communications and media theory, architecture and design, and many others. In an important sense, then, the task for hypertext theory at this point is to define itself, to describe and understand its constituent parts, and (perhaps most of all) clearly identify the object of its attention. What I have to say here about evaluative criteria is addressed to a narrowly defined "hypertext theory," and even within that, principally to the literary type, but I think it could apply as well to the broader field of media studies in which hypertext theory sometimes finds itself. In addition, I'm going to work with a much narrower meaning of the word "theory" than is usually used in connection with hypertext, and especially in literary hypertext theory. In brief, "theory" here is taken to mean assertions (about the nature or function or design or impact of hypertext) that have the potential to be proven or disproven.

• Can it be falsified?

 

 
 
 
 
 

The first criterion I would propose, in evaluating theoretical statements about hypertext, is borrowed directly from Popper, namely the criterion of falsification. As Popper has it, if a statement cannot possibly be proven false, then it can't be considered a scientific statement: it might be a perfectly legitimate example of some other kind of statement (metaphysical, philosophical, poetic, etc.), but it is not scientific because, for Popper, the distinguishing feature of science is that it proceeds by making assertions that can be falsified, testing them, and perserving, modifying, or discarding its beliefs based on those tests

Obviously, that first criterion raises the question of what we are to call writings on hypertext that don't make claims that could be falsified: "Essays" might be a good choice, in the tradition of Montaigne; appreciations, musings, metaphysics -- all those are open, too. My point is not that all writing about hypertext should take the form of empirical assertions, only that we should have a clear way of distinguishing the genre of writing about hypertext that we are reading, and if that writing calls itself "theory" then we should expect it to provide us with (dis)provable assertions. And when a theorist of hypertext does make claims of a factual nature (such as the claim that hypertext is an improvement over the state of text in printed form), then the person making that claim has obliged himself or herself to support those claims with empirical evidence and rational argument, not to prove the assertion true (something that can't ever be done, even in science), but only to make the best case that can be made, given both what we do know and what we don't.

That first criterion, falsification, is extremely important: If we do think that we are "reinventing the text," if we suppose that we are in fact inventing or doing "research" in any sense of the word, then we must have a theory to guide that research, and it must be possible for that theory to be proven wrong by the evidence. In short, if failure isn't a possibility, neither is discovery.

It should be noted, too, that the possibility of failure is not simply a matter of the nature of our assertions, but also of the climate and terms of our funding: In the sciences and in the humanities alike, the current atmosphere is not friendly to failure largely because of the emphasis on short-term, gainful outcomes (marketable products, if you will). The emphasis on marketable products is obviously an expression of society's desire to "get its money's worth" out of research funding of all kinds. I would argue that, if we really want to get our money's worth, we should make sure that we don't fund "research" that investigates problems the solutions to which are already known, nor should we fund research that selects problems likely to be solved successfully in one funding cycle. Of course, we don't want to encourage failure for its own sake either, but it seems clear that we should favor those projects that stake out difficult territory, have a well-thought out approach to that territory, and can at least define what failure, or in a narrower compass, falsification, would be.

• Is it explanatory?

 

 
 
 
 
 

In simplest terms, the purpose of science (and of knowledge more generally) is to explain. In the sciences, as elsewhere, that is generally a matter of degree, not of absolutes, and one measure of the value of a theory is its reach: All other things being equal, the theory that explains more of the observable data associated with a particular problem area is generally considered a better theory. I see no reason why the same should not be true of hypertext theory, or of theories concerning new media more generally. In reasoning about the transformation of the book, or its disappearance, or the emergence of whatever might replace it, we may proceed from isolated observations, but our conclusions on the larger topic ought to be able to explain more than the individual observations from which they are derived. In other words, theory in this realm, as in others, needs to rise above particulars to generalizations (and, as earlier proposed, those generalizations ought to be testable against evidence, and potentially falsifiable).

• Is it predictive?

 

 
 
 
 
 

That is a difficult one, not only for the humanities, but for social sciences as well. In "Replies to My Critics," Popper paid special attention to the predictive function as a means of distinguishing between scientific and nonscientific reasoning. What he concluded was that

There is a reality behind the world as it appears to us, possibly a many-layered reality, of which the appearances are the outermost layers. What the great scientist does is to boldly guess . . . what these inner realities are like. This is akin to myth making. . . . [and] [t]he boldness can be gauged by the distance between the world of appearance and the conjectured reality, the explanatory hypothesis.

But there is another, a special kind of boldness, the boldness of predicting aspects of the world of appearance that so far have been overlooked but that it must possess if the conjectured reality is (more or less) right, if the explanatory hypotheses are (approximately) true. . . .

. . .[I]t is this second boldness, together with the readiness to look out for tests and refutations, which distinguishes "empirical" science from non-science, and especially from pre-scientific myths and metaphysics.

I do think that the second kind of boldness can be expected, in rare instances, from theories about the transformation of the book, about hypertext, about whatever the object of our discussion may be called: new "aspects of the world of appearance" (of information) will emerge, within our generation and the next and the next, theory could aspire to predict those appearances. The theory that does so could also look for tests and refutations, even before they appear.
• Is it productive?

 

 
 
 
 
 

A good theory should be productive in a number of ways: it should inspire argument, it should give rise to new ideas, observation, and speculation, it should allow us to do things -- things we couldn't do before, things we didn't know we wanted or needed to do, things we hadn't imagined doing. In short, it should be fertile. Again, I see no reason why that criterion should not be applicable in our domain as well as in others, and in fact I expect that quality to be valued above (and sometimes at the expense) of all others, in our domain. Whether or not we believe Marx or Freud as explainers or predicters, we in the humanities still value them highly because they have been and continue to be productive, productive of discourse, above all.

• Is it persuasive?

 

 
 
 
 
 

In measuring the persuasiveness of a theory, I can think of no better metric than that proposed under the heading of "conformity" by the Principia Cybernetica project. In this remarkable Web, the Conformity node begins by noting that "the more people already agree upon or share a particular idea, the more easily a newcomer will in turn be infected by the meme." The author of the node (Heylighen) notes that "conformity pressure is mostly irrational, often rejecting knowledge that is adequate because it contradicts already established beliefs," but he goes on to point out that

Conformity pressure is an expression of "meme selfishness." As memory space is limited and cognitive dissonance tends to be avoided, it is difficult for inconsistent memes to have the same carriers. Cognitively dissonant memes are in a similar relation of competition as alleles: genes that compete for the same location in the genome. Memes that induce behavior in their carriers that tends to eliminate rival memes will be more fit, since they will have more resources for themselves.

Clearly, one would not want to privilege persuasiveness, or successful meme selfishness, above other criteria for evaluating theoretical proposals, but inasmuch as the evolution of the book is a co-evolution, proceeding in a complex relationship with ideas about the evolution of the book, we should recognize that in that case there is a material interaction between theory and its object, and that a successful theory may achieve its success -- even on predictive grounds -- as a result of its persuasiveness.

Evaluative Criteria in Hypertext Projects
As I noted earlier, the evaluative criteria appropriate to hypertext theory and to hypertext practice are likely to be different. Whereas the criteria I would apply to theoretical statements turn largely on the claims implied or expressed at an epistemological level, the criteria I would apply to hypertext projects have more to do with the implementation of theory, and thus with the results themselves, or with the goals expressed for the particular experiment. We should be able to say whether a particular project's goals proceed from some implicit or explicit theory or theories, and we should be able to say whether those goals seem to us to be worthy, and why, but we do not, and should not, on the whole, expect a particular project to focus its energies and resources on elaborating or defending its theoretical superstructure: it is enough, I think, that it should provide evidence for accepting or rejecting a theory, produce a useful product, and/or raise interesting new problems or solutions.

• Does it declare the terms of its own success or failure?

 

 
 
 
 
 

It is fair, I think, to require new projects in the area of electronic texts, digital libraries, hypermedia editions, to declare the terms of their potential success or failure. If I can't tell you that much about what I propose to do, then I don't know what I'm doing, or why. If I do know what and why, then I know what will constitute success or failure, and I ought to articulate that. Granted, it may be difficult to provide a clear and immediate formula that will really make sense of the extrinsic measurements one could gather: Hits on a Web site? Citations in the scholarly literature? Acceptance at the high-school level? At the intrinsic level one ought to be able to establish milestones for production and functional specifications for use, at the very least. Frankly, the only metric that is likely to matter to the universities that sponsor such projects is their success in attracting outside funding, but scholars, designers, and funding agencies ought to care about more than those simple intrinsic criteria. That is not to say that failure to meet those goals should be considered sufficient reason for abandoning the project, but if the initial functional and production goals of the project are not met, then that ought to be the occasion for an analysis of failure, which in some cases might be the most valuable thing to come out of the project.

• Does it formulate a methodology for solving the problem it addresses?

 

 
 
 
 
 

That is a criterion that applies in rather different ways to the beginning, the middle, and the end (if any) of a project. At the beginning, a problem-solving methodology ought to be required, but it shouldn't be regarded as a failure if that methodology is revised in the process of completing the project, since we assume (if this is research) that there will be some sort of feedback loop between the problem and the solution, and as the problem is progressively analyzed and considered, the methodology for solving it will also be refined. In the middle of a project, if there has been no change at the methodological level, then I would suspect that the problem selected was not really a problem at all. If, at the end of the project (and I haven't seen the end of one of those projects yet), the methodology couldn't be formulated in general terms, then I would suspect that nothing much had been learned from the experience of tackling that problem. In fact, I think that successful hypertext projects are continually reformulating their methodology, and their only failure, on the whole, is the failure to document the stages in and reasons for their methodological evolution. That is a very real failure, though, since we could learn a great deal not only from their product, but also from their process.

• Does it address (or generate) unsolved problems?

 

 
 
 
 
 

In Conjectures and Refutations (1960; 1968), Karl Popper notes that

Every solution of a problem raises new unsolved problems; the more so the deeper the original problem and the bolder its solution. The more we learn about the world, and the deeper our learning, the more conscious, specific, and articulate will be our knowledge of what we do not know, and our knowledge of our ignorance. For this, indeed, is the main source of our ignorance: the fact that our knowledge can only be finite, while our ignorance must necessarily be infinite.

That passage gives us, I think, a very compact, elegant, and persuasive criterion for deciding whether a real problem has been addressed, and solved -- namely, the test of whether the solution of that problem has raised new problems. All of my personal and pedagogical experience strongly inclines me to agree with Popper that acquiring new knowledge means discovering new ignorance. Given that, then hypertext research projects should be expected to address unsolved problems (otherwise their problems belong to the arena of production rather than that of research), and the proof of their having done so should be that they culminate in a new plateau of ignorance, a new set of unsolved problems.
• Can its solutions be generalized?

 

 
 
 
 
 

Finally, on the topic of evaluative criteria for hypertext projects, I would suggest that the solutions a project does arrive at -- notwithstanding the new, unsolved problems it should raise -- ought to be generalizable to other work in other disciplines and other contexts. That principle is, at the applied level, very like the principle, at the theoretical level, that says a theory should be broadly explanatory. The practical experiment that produces the greatest number of tools, methods, errors, or insights that can be generalized to other projects, other disciplines, other contexts, will be the most successful experiment, at least as research (mind you, it may not be the most popular on the Web, or the most marketable). I'd go even further, and suggest that at this early stage in the evolution of our methods and this medium, we should give the highest priority to projects that clearly demonstrate a potential for generating generalizable solutions -- provided, of course, that they can say why those solutions are needed and how they might be arrived at.

Conclusions
We are in an important evolutionary moment: an important transformation is taking place, and we are a part of it. Many things that we take to be trivial, or embarrassing, or simply wrong, will be of interest to our peers in the future. Our first responsibility, therefore, is to document what we do, to say why we do it, and to preserve the products of our labor, not only in their fungible, software-and-hardware-independent forms, but also in their immediate, contemporary manifestations. The greatest mistake we could make, at this point, would be to suppress, deny, or discard our errors and our failed experiments: We need to document them with obsessive care, detail, and rigor. Our successes, should we have any, will perpetuate themselves, and though we may be concerned to be credited for them, we needn't worry about their survival: They will perpetuate themselves. Our failures are likely to be far more difficult to recover in the future, and far more valuable for future scholarship and research, than those successes. So, if I could leave you with a single piece of advice, it would be this: Be explicit about your goals and your criteria, record your every doubt and misstep, and aspire to be remembered for the ignorance that was uniquely yours, rather than for the common sense you helped to construct.

This paper was presented October 25, 1997, at a conference on the "Transformations of the Book," an event in the Media in Transition series.

---

The Journal of Electronic Publishing
December, 1997   Volume 3, Issue 2
ISSN 1080-2711   http://www.press.umich.edu/jep/03-02/unsworth.html

Página creada y actualizada por grupo "mmm".
     Para cualquier cambio, sugerencia,etc. contactar con: fores@uv.es
     © a.r.e.a./Dr.Vicente Forés López
      Universitat de València Press
    Creada: 15/09/2000 Última Actualización: 18/06/2001