Twitter and search sound like two peas in a pod, but it’s actually not the marriage made in heaven that you’d think it is. When you think of search, you think of a search engine, like Google, where the world’s information is seemingly at your fingertips. You feel confident that when you Google something, you won’t miss the important information.

The secret is that it’s Google’s algorithm that makes search work, not the fact that it indexes everything in the world. In fact, most people don’t get past the second page of search results, so we’re not even utilizing all of the data that Google collects. When I speak to people about Twitter search, they seem to want the same thing: “access to every tweet ever tweeted.” That sounds fine on paper, but in actuality, you really don’t want access to every tweet — just the really good ones.

That’s the issue that Twitter is tackling these days, figuring out which tweets to serve up when you search for a word, phrase topic or hashtag. If you were to search for “#grammys” on Twitter, you’d find a whole lot of junk and spam and your experience wouldn’t be a very good one. Sure, we all want to know what our buddies said five years ago when they were drunk, but that’s not how Twitter search works right now.

Last week, the company announced that it would be introducing “older tweets” into search results, with not much more information than that. Here’s what the team said at the time:

Previously, Twitter search results displayed Tweets going back about a week. We’ve developed a way to include older Tweets, so you can see content that goes beyond the more recent Tweets.

Pretty vague, I’d say. But the crux of that statement is that Twitter is definitely looking backwards as far as the content that its accumulated since launching in 2006. There’s a lot of great information to be had from tweets that happened during events like the uprising in Egypt, political elections, the day that Michael Jackson died and just about every natural disaster that’s happened since Twitter launched.

I sat down with Sam Luckenbill, an engineer on the Twitter search team, to discuss what the company has in mind for its search experience. Luckenbill joined Twitter after having been a Ruby on Rails consultant. When he joined the company full time after grad school, there were only 20-30 employees. In 2008, Twitter acquired a search company called Summize, leading many to believe that Twitter would turn into the next Google but just with tweets. That wasn’t the case, as a simple search and result experience isn’t engaging in a real-time environment:

Twitter is mostly real-time and mostly will be, people overestimate how valuable older stuff is going to be to them. I think in general, for a search product to be great, you have to cover the long tail.

What does a Twitter long tail look like if it’s so real-time? That’s what the company has to grapple with as far as what the search experience will morph into over time. I imagine an experience that takes me back to a moment in time, much like Facebook’s Timeline, where I can search for something like “Egypt” and then am able to relive a very serious event from the most popular tweets sent at that time. Twitter is a long way away from that, but with the introduction of older tweets, it’s a natural progression.

The queries that Twitter gets are very different from what Google and Bing see as well. “In particular, the queries change very quickly, and it obviously matches what’s going on in the world,” Luckenbill tells me. “That’s part of the reason why realtime is hard – the corpus is constantly changing.”

Google relies on its own News product to fill up its search results with “real-time” information, whereas everything that gets tweeted, in essence, could be huge news. Twitter is learning from its users about how they search, what they interact with once they get results and then tinkering with its approach on the fly, much like a real-time company would do.

When you have as much information as Twitter has, you can’t just dump out an entire database of ramblings on your users in a search result. You have to pass it through a few filters to make sure that you’re serving up the best, most interesting and relevant content.

So will Twitter change the way search looks in the future? Luckenbill only said “The UI isn’t in its final state.” That would be a huge yes.

As the flock has grown, the search team has grown, and people are starting to carve out specialties for themselves. Search is going to be a speciality for sure, but how will it end up looking? What content will we want to see? Do we really want to see every tweet that’s ever been published? No, of course not, but when things aren’t trivial, all we can do is attempt to trivialize them. In an attempt to feed our appetites for “all the tweets,” Twitter rolled out its downloadable archive feature last year, and I think it’s a safe bet that people searched once or twice and then stopped. We think we wanted it, but did we really need it?

This problem is in Twitter’s court, so we’ll have to wait and see. Figuring out what we need over what we think we want is no simple task. Since there’s never been a service quite like Twitter, we don’t have anything to compare it to as far as relevancy and search experience. So it’s something that we’re just going to have to trust with the little birdies down on Market Street.

It’s not so much about “search.” It’s about “discovery.”

[Photo credit: Flickr]