Well, they’re old. I’m not going to read the really old stuff but some of them were dog-eared (pages bent over to mark them).  I couldn’t remember why I did that so I had to read the article again. I came across this set of “stats” in the May 5, 2008 Information Week issue about analytics (page 6):

“The Information Week 500 recognizes the most innovative business technology companies. So, what defines an innovative organization? A look at the 500 companies that made the list last year finds:

• 54% have deployed AJAX development tools

• 43% use business intelligence to boost productivity

• 27% communicate with customers via wikis, blogs, and social networking.”

Given our interest in Enterprise 2.0 developments, these are interesting stats. I hope they are meaningful and related to what we are doing or otherwise I am just having “happy ears” and hearing what I want to hear…

(Note: Dear InformationWeek – if there is a link to this article/item on your site, I’d be happy to use that rather than type out the text – I couldn’t find it though).

The Top 100 Alternative Search Engines by Written by Charles Knight, AltSearchEngines editor / January 29, 2007 2:34 AM

To be very honest with you, I don’t use nor have I tried all of these different engines.  I think they actually validate my other points, that you should use the engine and technique for the problem at hand.  Some of them are fun to play with (although Ms. Dewey is a bit loud for work) and they (for the most part) variously demonstrate the basic items about search – user interaction and visualization, keywords, ranking, relevancy, clustering, relationship tracking, social search, etc.

I think I will have to add in a whole posting thread about the engines listed; not to duplicate what Mr. Knight has done but hopefully to extend it.

Take a look and see for yourself.  Enjoy.

The commodity systems that are available out on the market have been designed with a leading purpose in mind. One question then, is to divine just what that purpose was and to see if it matches your search problem. As a former consultant, I find it interesting that many people have approached the problem as a “feature/function” comparison and then selected an offering that really doesn’t match the need. I don’t think the fault is entirely with the search engine folks but rather in the buyers just not understanding their own problem. There’s something inherent about search engine functions and features that makes it difficult to match them to the problem and that probably lies in a poor understanding of what people are doing in their job execution overall that forces them to look for information. (Now that part of the problem does belong to the search engine teams).

I’ll set that thought aside for now and move to taking a look at the nature of the use cases around search. A friend of mine has suggested that that are something like 5 different use cases around search:(Feel free to argue with me on this point).

1. Simple File System search
2. eMail Archiving
3. Government Data Warehousing
4. Generic Content Management
5. Legal Discovery

… but I think this is more like a spectrum of use cases to deal with and looks something like this:

I think the biggest takeaway on this topic is that the amount of metadata is always increasing (when going to the right) and it also becomes more user defined (adhoc) and less pre-defined (a priori). Notice also how the nature of the search question is changing across the spectrum. I think this is the crux of the problem on the buyer’s side – they don’t understand their own question well enough in the context of general search and then the seller’s get into it and voila! you get a mis-match on the system.

Shouldn’t be a surprise to anyone.

The other thing that I’ve observed is that most people, including architects, designers, product managers, etc., don’t really think through the question of how the users will invoke search. What are these people searching for? Why are they searching for it at all? How will they approach the job at hand?

As one of the stakeholders at CM&A around search, indexing, analytics, and visualization, I get asked a lot of questions about search and what we’re planning to do, about it. I also get to ask questions back like, “What kinds of search do you do?” and, “How well does it work?” Most of the time the answer is that it doesn’t work very well and “I prefer to search the internet as opposed to my own intranet”. It’s an interesting statistic that about 50% of all keyword searches fail. That is, the user has to modify the query keywords and try again. And that’s just the interfaces like Google, which are pretty minimalistic.

I admit it, I’m not a big fan of PowerLink and according to most customers I talk to, they aren’t either. I think that one reason is in how customers and employees approach the question of search and how it was implemented is a disconnect. So that’s today’s topic.

I started a set of research which I call “the psychology of search” and is really focused on how people approach the problem/question. I think there are 3 kinds of search:

Recovery, which is looking for a specific thing in a specific place (“Honey, have you seen my car keys?”);

Discovery, which is about looking something up (“Okay, what’s on the calendar for today?”); and

Exploration, which is akin to information research (“Hmm, what are the competitors doing?”).

The graphic below might sum this up:

These are not the academic definitions – I will refer you to Broder’s Taxonomy for that – but customers seem to get the picture for this pretty well.

These generalized categories arose from this slightly more detailed list of search types we’ve seen people execute:

• Specialized Search -Seeking specific information whose characteristics are tightly defined
• Generalized Search -Unrefined, broad, loosely defined. Not concerned with completeness of results…
• Legal Investigation
  • Defendant – an exhaustive search of all sources and assets for specific information. Generally tightly defined. Completeness of results is essential. Results must bear intense scrutiny from opposing council. Must provide transparency into idiosyncrasies.
  • Plaintiff – a detailed examination of assets for specific information. Typically more loosely defined. Resembles Knowledge Discovery in that plaintiff’s attorney’s will want to “cast a broad net” for information.
• Knowledge Discovery (KM) -R&D, Patent Analytics. Generally an unstructured search with a few keywords
• Business Intelligence -Data mining, trolling, pattern recognition, trending analysis. Usually performed on historical data archives to refine operational behaviors or processes
• Internet Search -The main categories for such queries are shopping, finding various web-mediated services, downloading various type of file (documents, etc), accessing certain data-bases (e.g. Lexis-Nexis type data), finding servers (e.g. for research ) etc.
• Situational Awareness -Watching or monitoring specific data sources for activities, trends, events. Generally accomplished by text mining of sources like LexisNexis, etc.
• Navigational -Characterized by knowing you have something but not remembering where you put it. Could have very high query rates if widely deployed. People prefer categorical navigation.

I think once you realize that there are all kinds of search types and that these need to be supported well, you will start thinking about the kinds of search and indexing technologies you will use to help your customers. Until this happens, though, search will be a very dissatisfying experience.

I was putting together my deck for EMC World and thought that maybe it would make good blog material. People are always asking me about search, classification, analytics, and visualization anyway, so I thought, why not?

I started looking into the history of search engines and I was very surprised to find that this is really new to the technology scene. I mean, full text is really only 18 years old. When I think about it, for me, search was always there. I worked in an advanced research shop in Mountain View, CA back in the early 80’s and was always on the internet. All these arcane applications that most people never heard of: Gopher, Finger, Archie, etc. (Unless you are really old, like me and Dave Reiner). I was using Netscape before it was a company. Okay, I’m old. I’m not a web native. I’m a web immigrant. But full text apparently got started just before I started at this R&D company, so to me, search was just natural. I thought it was always there. Little did I know.

A bit of a historical perspective:

• In 1945, Vannever Bush (Wikipedia lists him as, “A leading figure in the development of the military-industrial complex and the military funding of science in the United States, Bush was a prominent policymaker and public intellectual (”the patron saint of American science”) during World War II and the ensuing Cold War, and was in effect the first presidential science advisor”).
• In the 1960’s, Gerard Salton invented the SMART Information Retrieval System. Many important concepts in information retrieval were developed as part of research on the SMART system, including the vector space model and relevance feedback.
• In 1960, Ted Nelson created Project Xanadu with the goal of creating a computer network with a simple user interface. It didn’t succeed. http://en.wikipedia.org/wiki/Project_Xanadu
• In 1963, Ted Nelsoncoined the phrase, “hypertext“.
• In 1972, ARPA invented the ARPANet which, “was the world’s first operational packet switching network, and the predecessor of the global Internet.
• In 1990, Alan Emtage creates the first search engine, “Archie” (short for Archives”)
• 1991: Tim Berners-Lee combines with TCP/IP and invents the WWW. First WWW site, http://info.cern.ch/, posted August 6, 1991.
• 1994: Berners-Lee founds the World Wide Web Consortium (W3C) at MIT

Today: We know we have too much data.

I find it interesting to look at some old presentation materials from User Conferences (Documentum folks call these User Conferences “Momentum”; EMC’ers call it EMC World. Same thing).

Here’s a Momentum presentation by a customer (Nortel Networks) back in 1998:

The meat of the discussion: back in 1998, 5 years after the invention of the internet, Nortel already has the “too much data, not enough information, and I can’t find any of it” problem. Wow.