Dan Guido’s Favorite Food? (A script to search reddit comments)
October 5, 2012 Leave a comment
CSAW CTF was fun. My team (ACMEPharm) solved all the challenges but network 400, which was a dumb challenge anyway :P
One of the other challenges we struggled with was a recon one: “what is Dan Guido’s favorite food”? There was also a hint that said something like “A lot of our users use reddit”. Since we had already solved all the other recon challenges and none required reddit, we were fairly certain this is where to look. Looking at dguido’s page there are tons of links- he’s part of the 5 year club.
Reddit has a robots.txt that tells search engines not to search it, and also a user’s comments aren’t indexed so they aren’t searchable using it’s search. This was the motivation for me to scrape a user’s comments so I could search through them locally.
#!/usr/bin/python import urllib import sys import time class AppURLopener(urllib.FancyURLopener): version = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1" urllib._urlopener = AppURLopener() class redditScrape: def __init__(self, startpage): self.thispage = startpage self.counter = 1 def getInfo(self): while 1: print "Fetching ", self.thispage f = urllib.urlopen(self.thispage) data = f.read() self.saveHtml(data) self.getNextPage(data) #reddit asks for only one request every two seconds time.sleep(2) def saveHtml(self, data): f = open(str(self.counter), "w") f.write(self.thispage + "\n\n") f.write(data) f.close() def getNextPage(self, data): index = data.find("rel=\"nofollow next\"") if index == -1: print "Search done" sys.exit(0) else: hrefstart = data.rfind("href", 0, index) + 6 hrefend = data.find("\"", hrefstart) self.thispage = data[hrefstart: hrefend] self.counter += 1 a = redditScrape("http://www.reddit.com/user/dguido") a.getInfo()
Then I would
grep -Ri "cheese" . grep -Ri "pizza" . ...
Unfortunately the answer turned out to be in another person’s comment so my script missed it, but someone else on my team found it not long after… in a thread I was blabbering in.