Dec 29 13:03:10 mopey-macky kernel: [72355.838027] Free swap = 0kB
Dec 29 13:03:10 mopey-macky kernel: [72355.838031] Total swap = 5855684kB
Dec 29 13:03:10 mopey-macky kernel: [72355.866431] 1048576 pages RAM
Dec 29 13:03:10 mopey-macky kernel: [72355.866436] 38328 pages reserved
Dec 29 13:03:10 mopey-macky kernel: [72355.866440] 9361 pages shared
Dec 29 13:03:10 mopey-macky kernel: [72355.866444] 1000493 pages non-shared
Dec 29 13:03:10 mopey-macky kernel: [72355.866451] Out of memory: kill process 6730 (run-mozilla.sh) score 665297 or a child
Dec 29 13:03:10 mopey-macky kernel: [72355.866556] Killed process 6734 (thunderbird-bin)

Yes, that was fun, randomly killed processes because I’m out of memory. some instances of nikto were taking 2gb of memoy and spidering infinitely over these dynamic pages.

To fix, I added a stupid watchdog script.

#!/usr/bin/python

import subprocess
import time

#percent of memory the nikto is taking
MAXMEMPERCENT = 13
#time is in hours
MAXTIME = 1
#time in seconds to check
SLEEPYTIME = 60
lfile=open("./nikto_wd.log", "a")

while 1:
  p1 = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE)
  p2 = subprocess.Popen(["grep", "nikto"], stdin=p1.stdout, stdout=subprocess.PIPE)
  output = p2.communicate()[0].split("n")

  for line in output:
    #print line
    thisline = line.split()
    try:
      if ("/usr/bin/perl" in thisline[10] and thisline[3] != "" and thisline[9] != ""):
        memusage =  float(thisline[3])
        hours = int(thisline[9][0])
        #process needs to be killed 
        if int(hours) > MAXTIME or float(memusage) > MAXMEMPERCENT:
          print "die, zombie scum", thisline
          lfile.write("die, zombie scum " + str(thisline) + "n")
          subprocess.call(["kill", thisline[1]])
    except IndexError:
      pass
  lfile.flush()
  time.sleep(SLEEPYTIME)