Goldkin's Journal: Musings of the Shiny Lizard

Code: http://pastebin.com/s5bkVEuV

For the longest time, I'd been hurting for a good, topic-based organization scheme for all of the feeds I consume in Google Reader. This may be useful to anyone that has this same problem with their own RSS reader, so I figured I'd publish this for the Greater Good™.


My problem is: I consume (not necessarily read) over 500 RSS-syndicated articles a day. Because of this, I need a good, fast indexing scheme that tells me exactly what the most common topics happen to be so I can organize my time effectively. I have less time for obscure things, though if I'm targeting those to give myself a rest from the tedium of daily news, I'd like to be able to see them at a glance, too.

Doing this on a source-by-source basis falls apart when common issues (like SOPA) transcend sources that might otherwise contain very focused content. So, I needed a good organizational scheme that was topic-oriented, source-agnostic, and binned all content by its most common keywords and key phrases. Further, it needed to target what the articles are actually saying, instead of what the articles think they're saying (via, say, tagging the article).


The easiest organizational scheme I could think of was to present feeds similar to new posts in an Internet forum. So, I did. Technical babbling follows on how the algorithm does that.

Because titles usually contain the most relevant content for a particular item, each article's title receives its own topic category automatically, according to very simple rules for word and phrase relevancy. "Relevancy" is determined by how often words and phrases appear in the title, subtracted by how often each component word appears in the message body of all articles. In other words, very simple, quorum-based voting, emphasizing well-organized articles that put their most relevant information in the title and save all of their flavor text for the body.

End technobabble.


What I've found is this simple scheme works surprisingly well. It allows me to take my complete collection of feeds and articles and extract, at a glance, the most relevant ones that I should be reading. And I like that; it gives me good, high level insight that I can use to cut out much of the noise in favor of juicy, juicy signal. And it's so useful that I'm left to wonder why more RSS readers don't do this effectively, in idiomatic, simple, and well-organized ways.

So, as a grassroots effort at improving everyone's online reading experience, I figured I'd just release the code. It's pretty technical, for those that don't like playing with Python code and bending it to their will.

But for everyone who do: would you share this, improve it, and get it submitted to applications that should be using exactly this sort of organizational scheme? I'd greatly appreciate it, if only for the joy of knowing I helped make this little place we call the Internet that much easier to work with.

For the past several years, I've made my trade as an information worker. Even with my year-long hiatus after moving (which I've come to respect as a Good Thing), I kept busy while searching for what is now my present job.


And now I realize why, as a Floridian, my experience is so strange -- in a city that's home to so many diverse cultures.

It isn't my accent. It's not my manners. Nor is it my stature, my education, my experience, my phrasing, my looks, my cynicism, my quirky habit of saying all the wrong things at all the right times, or my manner of keeping current with the world.

I'm simply not used to being around smart people.


I say this without a hint of elitism. I'm just a person that needs to be around people smarter and more experienced than I -- to learn, and to help organize my thoughts.


It's one of those realizations that, cast backwards over the years, makes perfect sense of what was an otherwise confused journey:

* It explains why I saw no fundamental value in Computer Engineering when I was in college. In truth, U. Miami wasn't the best college for an engineering degree of any sort. When prompted by the Assistant Dean of Engineering's heartfelt plea that my class not graduate as engineers, I diverted to a (Computer Information Systems) degree co-hosted by the Business School.

* It explains why, during that degree and the next, I spent the majority of my time learning to program from books and online, on my own and separate from my degree. I maintained this without missing more than one lecture per class, never dropping or retaking a single class, and meeting the minimal requirements for prestige, graduating both degrees with an A-.

* It explains why I spent my time hiding in Second Life from 2004 onward. Imagine trying to come to grips with this reality with no one to talk to. Now imagine being given the perfect, malleable little world to play with for similar misfits. It was a match made in heaven.

* It explains why, after a year and a half of my job at an undisclosed anti-virus company, I found myself quickly bored and depressed. I once again lacked peers, beyond a friend who quickly became my silent coworker.

* And it explains why, now, I find myself hungry when I'm only just settling into my new job. I've already hit my stride, surpassed my peers, and promptly relapsed into my studies.


These are realities that, cast in the relief of new information, make perfect sense in hindsight.

For anyone else, being surrounded by intelligence might have happened naturally, from college onward. It didn't for me; I just wish I'd figured this out sooner.

So, to continue my education, I'll need to surround myself with intelligent people where I work. This all but ensures I'll be sending applications to Microsoft, Google, and Amazon again in the near future.

Though perhaps, not too soon. The brief peace in my career has been rather nice.