Dan Guido’s Favorite Food? (A script to search reddit comments)

CSAW CTF was fun. My team (ACMEPharm) solved all the challenges but network 400, which was a dumb challenge anyway :P

One of the other challenges we struggled with was a recon one: “what is Dan Guido’s favorite food”? There was also a hint that said something like “A lot of our users use reddit”. Since we had already solved all the other recon challenges and none required reddit, we were fairly certain this is where to look. Looking at dguido’s page there are tons of links- he’s part of the 5 year club.

Reddit has a robots.txt that tells search engines not to search it, and also a user’s comments aren’t indexed so they aren’t searchable using it’s search. This was the motivation for me to scrape a user’s comments so I could search through them locally.

#!/usr/bin/python

import urllib
import sys
import time

class AppURLopener(urllib.FancyURLopener):
    version = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1"

urllib._urlopener = AppURLopener()

class redditScrape:
    def __init__(self, startpage):
        self.thispage = startpage
        self.counter = 1

    def getInfo(self):
        while 1:
            print "Fetching ", self.thispage
            f = urllib.urlopen(self.thispage)
            data = f.read()
            self.saveHtml(data)
            self.getNextPage(data)
            #reddit asks for only one request every two seconds
            time.sleep(2)

    def saveHtml(self, data):
        f = open(str(self.counter), "w")
        f.write(self.thispage + "\n\n")
        f.write(data)
        f.close()

    def getNextPage(self, data):
        index = data.find("rel=\"nofollow next\"")
        if index == -1:
            print "Search done"
            sys.exit(0)
        else:
            hrefstart = data.rfind("href", 0, index) + 6
            hrefend = data.find("\"", hrefstart)
            self.thispage = data[hrefstart: hrefend]
            self.counter += 1

a = redditScrape("http://www.reddit.com/user/dguido")
a.getInfo()

Then I would

grep -Ri "cheese" .
grep -Ri "pizza" .
...

Unfortunately the answer turned out to be in another person’s comment so my script missed it, but someone else on my team found it not long after… in a thread I was blabbering in.