Sunday, September 2, 2007

Loader

I've got the basic loader working. It is actually a two stage process. It first loads all news stories from the various sites into a flat table. This stage is quick (just getting raw data).

Step two takes that info, processes it, and creates the new stories in the db (with 1 vote for each). It actually follows some of the links back to the social site, and gets the real story link. Which is a pain and will break, but it's all I've got for some sites.

This second step can take a while, as it's having to process a lot of info.

No comments: