Anatomy Of An Idea

People often ask me about my research techniques. You would think this would be a relatively straightforward question, but the truth is that I have to keep changing my answer, because my techniques are constantly shifting as new forms of search or discovery become possible. Right now, I'm in that thrilling stage of writing-while-still-researching my next book, and I just went through a little episode of discovery that I think might be worth mapping out, as a case study of how ideas come into being, at least in my little corner of the world.

The subject matter of the book is not all that important here, but suffice it to say that I am currently working on an introductory bit that contrasts old, bureaucratic models of state organization with some new network structures that are currently on the rise. So my mind has been primed for anything that seems thematically relevant to those topics.

This particular thread begins with a random encounter on Twitter: checking out my @ mentions a few weeks ago (vanity will get you everywhere), I stumbled across someone mentioning my book to a friend, and also recommending something called "Seeing Like A State." (I can't track down this tweet, so can't give proper credit here.) I wasn't fully sure what "Seeing Like A State" was, but it sounded up my alley, so a quick Amazon search revealed that it was, in fact, a very promising-sounding book written by James C. Scott, about the methods of state organization and control in modern history, and so within a matter of minutes, I was reading it on the Kindle iPad app. (I'm sure it is mentioned in many books that I've read already, but somehow I had missed it over the years.)

The book turned out to have a small discussion of the design of the French railway system in the early 19th-century, which reminded me of a map my old mentor Franco Moretti had showed me two decades ago in grad school, contrasting the heavily centralized French system with the more chaotic British rail lines. So that sent me into a quick exploration of French engineering history that culminated in downloading two PDFs of academic essays on the topic, each of which provided some key historical texture that was missing in Scott's book.

In the meantime, I was continuing to devour Seeing Like A State. Feeling a little guilty about missing a book that should have come across my radar before (it had been published in 1998), I googled around to see what responses the book had generated. As it happened, one of the top-ranking results was a blog post by the political theorist Henry Farrell, with whom I have been discussing the ideas in my new book for many years now. His post was part of a larger debate about the Scott book with the economist Brad DeLong, who had penned a detailed review on his blog. One of DeLong's main criticisms is the way in which Scott ignores the insights of Hayek, which sent me back to an essay of Hayek's that I'd been meaning to read for my book, but hadn't yet got around to. The Hayek essay opened up a whole approach to what I was writing about that I suspect will generate at least a dozen pages of material in the final book.

As I was reading the Scott book, I was storing my highlights from it in the new service, findings.com, which we launched a month or so ago. Findings lets you organize important quotations from eBooks or the Web, but it also allows you to follow other users' quotations. (My introductory post about it is here.) One of the fascinating things it lets you do is see what quotations other readers found interesting in the books you've read. And so when I was reviewing my quotes from the Scott book, I discovered a few other Findings users were also reading it, and one them had picked out a quote that I had somehow missed, a quote that perfectly described the logic of state organization. That turned out to be the quote from the Scott book that I ended up using in my own chapter.

Sometime in the middle of this, I gave a talk at Google, and the speaker before me was the Internet legend Vint Cerf. Listening to Cerf talk about the origins of the Internet -- and thinking about the book project -- made me wonder who had actually come up with the original idea for a decentralized network. So that day, I tweeted out that question, and instantly got several replies. One of those Twitter replies pointed to a Wired interview from a decade ago with Paul Baran, the RAND researcher who was partially responsible for the decentralized design. When I clicked through the link, I discovered that the interview had been conducted by my friend and new neighbor, Stewart Brand, with whom I was having lunch that week. So when I saw Stewart I got to ask him about Baran, and try out this little hunch I was working on about the contrast between the French rail system and the design of the Internet. Meanwhile, one of the other Twitter replies had pointed me to Katie Hafner and Matthew Lyon’s Where Wizards Stay Up Late, released more than a decade ago but also available on Kindle, where I found a detailed history of the Internet's early days. And reprinted in that book was an early sketch by Baran of the network model that beautifully contrasted the centralized model of the French rail system, and the map that I had seen so many years ago as a grad student.  

And so after all that meandering, my vague introduction had turned into two distinct stories, with two perfectly contrasting diagrams to anchor them visually.

What's the moral of this story? I think there are a few:

1. The discovery process is remarkably social, and the social interactions come in amazingly diverse forms. Sometimes it's overhearing a conversation on Twitter between two complete strangers; sometimes it's the virtual book club of something like Findings; sometimes it's going out to lunch with a friend and bouncing new ideas off them. It's the social life of information, in John Seely Brown and Paul Duguid's wonderful phrase -- we just have so many more ways of being social now.

2. I find it interesting that there are certain kinds of questions that I now send out by default to Twitter, not Google. The more subtle and complex the question, the more likely it'll go to Twitter. But if it's simply trying to find a citation or source, I'll use Google. So trying to figure out who wrote Seeing Like A State was a Google query, but wondering about the origins of the Internet made more sense on Twitter. (I should add that the responses I'm looking for on Twitter are links to longer discussions, not 140 character micro-essays.)

3. Priming is everything. All these new tools are incredible for making rapid-fire discoveries and associations, but you need a broad background of knowledge to prime you for those discoveries. I'm not sure I would have jumped down that wonderful rabbit hole of the French railway design if I hadn't seen that map in grad school two decades ago. Same goes for the Hayek and the internet history as well. I had enough pre-existing knowledge to know that they belonged in the story, so when something about them got in my sights, I was ready to pounce on it.     

4. Very few of the key links came from the traditional approach of reading a work and then following the citations included in the endnotes. The reading was still critical, of course, but the connective branches turned out to lie in the social layer of commentary outside of the work.

5. It’s been said it a thousand times before, by me and many others, but it's worth repeating again: people who think the Web is killing off serendipity are not using it correctly.

6.  Finally, this simple, but amazing fact: almost none of this--Twitter, blogs, PDFs, eBooks, Google, Findings--would have been intelligible to a writer fifteen years ago.

Comments

[Lyttleton]

A very interesting read. I just finished your most recent book and passed it on to a friend. As a city dweller, I was especially drawn to the discussion of how city life is a boon for creativity and innovation.
What I've always appreciated about your books (and I think I've read all but maybe one) is your consistent optimism and wonder at how technology and interconnectivity improves our lives. It's a very exhilarating contrast to the constant nihilistic storm of commentators both off and online.

[Jaredran]

While this is all incredibly cool, do you believe that it's making your books better? Ever richer in serendipity and an even longer zoom than you were able to achieve 15 years ago? Or are you simply accessing the same pool of information faster?

[Life Insurance Laura]

I would think he is able to access a whole new pool of information with key word searches and the information Google is able to assemble. Its really quite astounding when you think about the sheer numbers.

[dien thoai di dong]

No offense, but if there's a facebook like button, it'll be much easier for me to share.

[discount jordan shoes]

rabbit hole of the French railway design if I hadn't seen that map in grad school two decades ago. Same goes for the Hayek and the internet history as well. I had enough pre-existing knowledge to know that they belonged in the story, so when something about them got in my sights, I was ready to pounce on it.

4. Very few of the key links came from the traditional approach of reading a work and then following the citations included in the endnotes. The reading was still critical, of course, but the connective branches turned out to lie in the social layer of commentary outside of the work.

5. It’s been said it a thousand times before, by me and many others, but it's worth repeating again: people who think the Web is killing off serendipity are not using it correctly.

6. Finally, this simple, but amazing fact: almost none

[Charming Charlie]

It should also be said that at every step of the way you're able to filter out leads that would not be productive. Or perhaps that filtration happens as the narrative is being written about the research.

[calendar for android]

Without his subconscious inspiration, you would've appeared much more attractive, and less Gingrich-esque.

[Mccormicktim]

> "people who think the Web is killing off serendipity are not using it correctly."

The discovery process you describe in your article is impressive and inspiring -- like your recent book Where Good Ideas Come From, which I read and enjoyed.  However, it is hardly representative of most Web users' access to or use of information.

The serendipity concern is based on what the 99% do, not you: evidence shows that most people use relatively few news sources, rarely switch away from accustomed tools (e.g. their search engine), Twitter posts come largely from 20k or so accounts (cf Yahoo Research "Who Twitters?" study by Wu, Watts, et al), and people increasingly get info from social networks of people who are often like-minded.

To suggest that people articulating the serendipity problem, e.g. Eli Pariser, generally do so because they personally don't know how to use the Web correctly isn't plausible or very generous.  Your readers would be well served by a more fair hearing of the idea and its reasoned proponents.

@mccormicktim

[Sparkarch]

Serendipity is the subject of the 2008 Darwin College Lecture Series. [http://sms.cam.ac.uk/collection/74]
I think your point 3 is dead on for serendipity. As a concept it rests on the sagacity part of its definition as much as chance. As the boy scouts say "be prepared." You can't be lucky if you don't see opportunity in the stream of chaos around you.

[Ugg boots clearance]

OMG! How cute is this? I am so enjoying these projects that you are sharing. Nothing else is getting finished because I am too busy making projects!
thank you for sharing!

[ugg boots for sale]

discovery process you describe in your article is impressive and inspiring -- like your recent book Where Good Ideas Come From, which I read and enjoyed. However, it is hardly representative of most Web users' access to or use of information.

The serendipity concern is based on what the 99% do, not you: evidence shows that most people use relatively few news sources, rarely switch away from accustomed tools (e.g. their search engine), Twitter posts come largely from 20k or so accounts (cf Yahoo Research "Who Twitters?" study by Wu, Watts, et al), and people increasingly get info from social networks of people who are often like-minded.

To suggest that people articulating the serendipity problem, e.g. Eli Pariser, generally do so because they personally don't know how to use the Web correctly isn't

[ugg 5815 boots]

There are other, more sensible ways to eliminate odors, the first of which is to clean up whatever's causing.

[KulaginDanila]

Эффектная реклама на радио и в прессе.

[quang cao web]

And Geneine, if you're going to comment, at least attempt a debatable point, unless you don't have one.

[UGG Boots]

should add that the responses I'm looking for on Twitter are links to longer discussions, not 140 character micro-essays.)

3. Priming is everything. All these new tools are incredible for making rapid-fire discoveries and associations, but you need a broad background of knowledge to prime you for those discoveries. I'm not sure I would have jumped down that wonderful rabbit hole of the French railway design if I hadn't seen that map in grad school two decades ago. Same goes for the Hayek and the internet history as well. I had enough pre-existing knowledge to know that they belonged in the story, so when something about them got in my sights, I was ready to pounce on it.

4. Very few of the key links came from the traditional approach of reading a work and then following the citations included in the endnotes. The reading was still critical, of course, but the connective branches turned out to lie in the social layer of commentary outside of the work.

5. It’s been said it a thousand times before, by me

[UGG Boots]

everything. All these new tools are incredible for making rapid-fire discoveries and associations, but you need a broad background of knowledge to prime you for those discoveries. I'm not sure I would have jumped down that wonderful rabbit hole of the French railway design if I hadn't seen that map in grad school two decades ago. Same goes for the Hayek and the internet history as well. I had enough pre-existing knowledge to know that they belonged in the story, so when something about them got in my sights, I was ready to pounce

[kirpik bakımı]

Twitter posts come largely from 20k or so accounts (cf Yahoo Research "Who Twitters?" study by Wu, Watts, et al

[estetik cerrahi]

And Geneine, if you're going to comment, at least attempt a debatable point, unless you don't have one.

[kurumsal seo]

There are other, more sensible ways to eliminate odors, the first of which is to clean up whatever's causing.

[ugg 5815 boots]

Folks are starting to get medieval on one another. Or maybe gang violence is not what it used to be in southern California.

[Samad Aidane]

It is great to see your statement about writing-while-still-researching.

I am following this approach right now in my research on what we can learn from latest brain research that we can apply to developing leaders. I stumbled on your book "Mind Wide Open" in the process and, by the way, found the last chapter "conclusion" some of the best writing on brain research I read so far.

Great to see that writing-while-still-researching can be achieved.

[Natalija]

Hi Stephen,
I saw your TED speech on Where Good Ideas Come From, and I dare to say you got it all wrong. I hope you don't mind hearing opposing thoughts on your "ideas". You missed the whole point, because in all of your speech and blog you never once mentioned the word intuition or gut feeling. Word hunch doesn't do it. The best ideas don't come from thinking or from intellect. They come from Higher consciousness to which we have access by means of intuition. Even better conditions for such great intuitive ideas than in cafeterias are in nature, meditation, temples or anywhere else where it is easy to connect to the Higher Cosciousness.
I suggest you reading this http://thefreepenguin.nl/DavidIckeNewsletters/Don't%20Think%20It%20...%20Know%20It%20-%20David%20Icke%20Website.pdf. Here David Icke explains the difference between thinking and knowing. Also, there are numerous popular science books that can inform you on this, linke Blink by Gladwell, Gut feelings by Gigerenzer (Max-Planck institute)...not to mention all the other books on intuition. Try to open your mind to it and reconsider.

[Ed Manlove]

A couple more paths that might lead you to where you are going...

A book entitled "Old Man Thunder, father of the bullet train" by Bill Hosokawa

The concept of "designing for informality" as talked about by Adam White and his cohorts at www.groupshot.com

[sanusi]

thats very good story but you did put the pics

[download Mac keylogger]

This is my first time i visit here. I found interesting things to many in your blog, mostly to the debate.