Oh, please, you just have to load it onto a blade server and have it cycle thru hashtag and word counts. That gens a file you use to index it, along with user and date time stamp. Be nice to store geocoding, but twitter handles are mobile.

I mean, just because they haven't done it doesn't mean it can't be done. We sequence entire genomes every day of the week.

and just because they could do it doesn't mean they should. I know I'm supposed to believe that Twitter caused the Arab Spring, but really it just feels like the hashtag-bloviating of a few million self-obsessed trivial minds.

I just don't give a shit and want it to go away.

This is the kind of stuff I tease my niece about: "My generation put a man on the moon. What has your generation done? Oh, yeah, you invented Twitter. Good job."

@1 According to LoC they have 170 billion tweets so far which require about 17 terabytes of storage. Searching data sets of this size is not something that your average "blade server" is capable of, unless you want simple searches to take hours to complete.

It sounds like they thought they'd be able to pipe everything through MySQL and run searches. But working with big data sets like this requires some dedicated thinking. Still, given the regularity of the data and the fixed record sizes I would think they could come up with something.

It would probably be a good task to set some graduate students at. Or, maybe hand out parts of the data set and have a prize for anyone who can efficiently index and search the data.

Or, just hire Google or Amazon to figure it out.