Count number of lines in a file

find . -type f -exec cat {} ; | wc -l

and if you don’t want repeats

find . -type f -exec cat {} ; | egrep \S | wc -l

I even thought about cat, but I ended up doing something like:

total=0
for i in $( find -H . -type f ); do
  temp=$( wc -l "$i" | cut -f 1 -d  )
  if [ $temp > 0 ]; then
    total=$(($total+$temp))
  fi
  echo $total
done
echo $total

Bash Error Checking

I was reading an oriley bash scripting book, and they had an entire chapter dedicated to error checking in bash.  For me, this was a little weird since I think the way they handled it made the code cluttered.

They have cascaded statements like

if [ $ -ne 0 ]; then
#handle error
elif…

For me, this is strange.  I think in a bourne shell, the best way is to just set the -e flag, which according to the man page

-e errexit If not interactive, exit immediately if any untested command fails. The exit status of a command is considered to be explicitly tested if the command is used to control an if, elif, while, or until; or if the command is the left hand operand of an “&&” or “||” operator.

This can be accomblished by

set -e

in your script.

Error handling is a balancing act between doing too much and handling everything (thus making the code unreadable) and not checking enough.  Normally, if I need much more error checking than that, time to pull out the python.

sorta captcha breaking thing

game 2 of hackthissite.org’s programming challenge

The directions (you have 30 seconds to upload the code)

“The pixels in the above image are numbered 0..99 for the first row, 100..199 for the second row etc. White pixels represent ascii codes. The ascii code for a particular white pixel is equal to the offset from the last white pixel. For example, the first white pixel at location 65 would represent ascii code 65 (‘A’), the next at location 131 would represent ascii code (131 – 65) = 66 (‘B’) and so on.

The text contained in the image is the answer encoded in Morse, where “a test” would be encoded as “.- / – . … -”"

Sample image:

Solution:

#!/usr/bin/env python
from PIL import Image

#load some image information
imagefile = Image.open('./PNG.png')

key = {  ".-":"A",
         "-...":"B",
         "-.-.":"C",
         "-..":"D",
         ".":"E",
         "..-.":"F",
         "--.":"G",
         "....":"H",
         "..":"I",
         ".---":"J",
         "-.-":"K",
         ".-..":"L",
         "--":"M",
         "-.":"N",
         "---":"O",
         ".--.":"P",
         "--.-":"Q",
         ".-.":"R",
         "...":"S",
         "-":"T",
         "..-":"U",
         "...-":"V",
         ".--":"W",
         "-..-":"X",
         "-.--":"Y",
         "--..":"Z",
	 ".----":"1",
         "..---":"2",
         "...==":"3",
         "....-":"4",
         ".....":"5",
         "-....":"6",
         "--...":"7",
         "---..":"8",
         "----.":"9",
         "-----":"0",
         }

#imagedata = imagefile.load()
#width,height = imagefile.size
#print imagedata[0,9]

imagelist = list(imagefile.getdata())

plaintext = ""

lastval = 0
thislettercoded = ""
for i in range(0,len(imagelist)):
  #we only care if the spot is white
  if imagelist[i] == 1:
    thischar = chr(i - lastval)
    if thischar == '.' or thischar == '-':
      thislettercoded = thislettercoded + thischar
    else:
      thislettercoded = thislettercoded.strip()
      plaintext = plaintext + key[thislettercoded]
      thislettercoded = ""
    lastval = i

print plaintext

modular exponentiation python program

This is a simple – not efficient – but doable way to do modular exponentiation

#!/usr/bin/python

def modexp(base, pow, mod):
  exponent = 1
  i = 0
  while i < pow:
    exponent = (exponent * base) % mod
    i += 1
  return exponent

if __name__ == "__main__":
  import sys
  print modexp(int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]))

Example usage

$ modexponent.py 2 1234 789
481

For large numbers this method isn’t practical. The below program implements the improved python algorithms for modular exponents.

#!/usr/bin/python
#Let x be an integer written in binary, y and n be integers
#I'm sure this isn't the best, but this is only called once per program
def binconvert(n):
  barray = []
  if n < 0: 
    raise ValueError, "must be positive"
  if n == 0:
    return 0
  while n > 0:
    #barray = n%2 + barray[:]
    barray.append(n%2)
    n = n >> 1
  barray.reverse()
  return barray
#this is the intuitive but slow way
def modexp(base, pow, mod):
  exponent = 1
  i = 0
  while i < pow:
    exponent = (exponent * base) % mod  
    i += 1
  return exponent
#y**x mod n
def modexp1(y, x, n):
  #convert x to a binary list
  x = binconvert(x)
  
  s = [1]
  r = x[:]
  for k in range (0, len(x)):
    if x[k] == 1:
      r[k] = (s[k] * y) % n
    else:
      r[k] = s[k]
    s.append ((r[k]**2)%n)
  #print s
  #print r
  return r[-1]
#similar to modexp1 except uses the bits in reverse order
def modexp2(y, x, n):
  a = x
  b = 1
  c = y
  while a != 0:
    if a % 2 == 0:
      a = a/2
      c = (c**2) % n
    else:
      a = a -1
      b = (b * c) % n
  return b
if __name__ == "__main__":
import sys
  print modexp(int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]))
  print modexp1(int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]))
  print modexp2(int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]))

Using the above program, some interesting things are easy to demonstrate. For example, here is a test that shows for 100,000 tries, the 2 algorithms get the correct answer.

import prob23
import random
import sys

BASEMAX = 60000
EXPMAX = 60000
MODMAX = 100000
ITERATIONS = 100000

print "beginning program"
print "This will take awhile... you need", ITERATIONS, "of these dots to complete."
p
for i in range (0, ITERATIONS):
  if i % 10 == 0:
    sys.stdout.write(".")
    sys.stdout.flush()
  r_base = random.randint(1, BASEMAX)
  r_exp = random.randint(1, EXPMAX)
  r_mod = random.randint(max(r_base, r_exp), MODMAX)
  trueval = prob23.modexp(r_base, r_exp, r_mod)
  testval1 = prob23.modexp1(r_base, r_exp, r_mod)
  testval2 = prob23.modexp2(r_base, r_exp, r_mod)
  if trueval != testval1 or trueval != testval2:
    print "One of these exponent things doesn't work"
    #sys.exit(1)
print "Done, all of these answers were the same."

Running this program several times with various parameters verifies the correctness of the algorithm (like how an Engineer verifies things).

Also, it’s interesting to time these algorithms when compared to the brute force method (multiplying every number). All three algorithms were timed to find a solution. Below is some example code that accomplishes this (this particular code is for the brute force version, but it is almost identical to the other algorithms – the only change is the modexponent call).

import modexponent

for i in range(1, 65538):
  thistry = modexponent.modexp(3, i, 65537)
  if thistry == 2:
    print i
    break
  if i % 15000 == 0:
    print "trying", i

Running all three algorithms, the results are shown below.

me@Opteron:~/math/chp3$ time python bruteforce.py
55296
real 11m28.131s
user 11m18.710s
sys 0m1.170s
me@Opteron:~/math/chp3$ time python algorithm1.py
55296
real 0m3.285s
user 0m3.290s
sys 0m0.000s
me@Opteron:~/math/chp3$ time python algorithm2.py
55296
real 0m2.621s
user 0m2.610s
sys 0m0.010s

This is a huge speedup – from 11 minutes in the case of brute forcing to under 4 seconds for the two algorithms.

gnu readline – python

This is the very start of our cryptanal program frontend

#!/usr/bin/env python

import readline

"""The shell class is the front end for cryptanal"""

class shell:
  def __init__(self, filename=None):
    print"""
WWW         WW eEeEeEeE LL        CCCCC    OOOO    MMMM    MMMM  eEeEeEeE
 WW         W  EE       LL       Cc      OOO  OOO  MM MM  M  MM  EE
  WW       WW  EeEeE    LL      CC       OO    OO  MM  MMM   MM  EeEeE
  WWw WW  WW   EE       LL       Cc      OOO  OOO  MM        MM  EE
   WWW  WWW    eEeEeEeE LlLlLlL   CCCCC    OOOO    MM        MM  eEeEeEeE

                             TO CRYPTO-SHELL
  (Useful for deciphering what little Susie is writing to little Billy)
"""

    self.filename = filename
    self.crypto = None
    #if self.filename != None:
      #self.crypto = freqcount.subCryptAnal(self.filename)

    #setup the tab completion information here
    self.commands = ["help", "printfreq"]
    readline.set_completer(self.completer)
    readline.parse_and_bind("tab: complete")

  #completer funtion for tab complete
  def completer(self, word, index):
    matches =
    try:
      return matches[index] + " "
    except IndexError:
      pass

  #this is the main event loop
  def mainloop(self):
    while 1:
      command=raw_input('> ').lstrip()
      if command.lower().startswith('help'):
        self.help(command[4:].lstrip())
      else:
        print "Error: command not recognized"

  def help(self, args):
    print "HELP"

if __name__ == '__main__':
  thisshell = shell()
  thisshell.mainloop()

Golay G24

This is a dirty implemenation of Golay correcting code using python.

This is a solution to 18.13 problem 1 from Trappe and Washington’s Crytography book. To run this, you need bash, python, and the numpy libraries. To run, run golay.sh.  The algorithm is located in golay.py

golay.py

#!/usr/bin/env python
 
from numpy import *
import sys
 
geng24 = array([[1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,1,1,1,0,0,0,1,0],
                [0,1,0,0,0,0,0,0,0,0,0,0,1,0,1,1,0,1,1,1,0,0,0,1],
                [0,0,1,0,0,0,0,0,0,0,0,0,1,1,0,1,1,0,1,1,1,0,0,0],
                [0,0,0,1,0,0,0,0,0,0,0,0,1,0,1,0,1,1,0,1,1,1,0,0],
                [0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,1,0,1,1,0,1,1,1,0],
                [0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,1,0,1,1,0,1,1,1],
                [0,0,0,0,0,0,1,0,0,0,0,0,1,1,0,0,0,1,0,1,1,0,1,1],
                [0,0,0,0,0,0,0,1,0,0,0,0,1,1,1,0,0,0,1,0,1,1,0,1],
                [0,0,0,0,0,0,0,0,1,0,0,0,1,1,1,1,0,0,0,1,0,1,1,0],
                [0,0,0,0,0,0,0,0,0,1,0,0,1,0,1,1,1,0,0,0,1,0,1,1],
                [0,0,0,0,0,0,0,0,0,0,1,0,1,1,0,1,1,1,0,0,0,1,0,1],
                [0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,1,1,1,1,1,1,1,1,1]])
 
B = array([[1,1,1,0,1,1,1,0,0,0,1,0],
           [1,0,1,1,0,1,1,1,0,0,0,1],
           [1,1,0,1,1,0,1,1,1,0,0,0],
           [1,0,1,0,1,1,0,1,1,1,0,0],
           [1,0,0,1,0,1,1,0,1,1,1,0],
           [1,0,0,0,1,0,1,1,0,1,1,1],
           [1,1,0,0,0,1,0,1,1,0,1,1],
           [1,1,1,0,0,0,1,0,1,1,0,1],
           [1,1,1,1,0,0,0,1,0,1,1,0],
           [1,0,1,1,1,0,0,0,1,0,1,1],
           [1,1,0,1,1,1,0,0,0,1,0,1],
           [0,1,1,1,1,1,1,1,1,1,1,1]])
 
errorvector = array([0,0,0,0,0,0,0,0,0,0,0,0])
 
def weight(obj):
  wt = 0
  for i in obj:
    wt += i
  return wt
 
geng24_transpose = geng24.transpose()
 
#r = array([1,1,0,1,0,0,0,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,1,0])
#r = array([0,1,0,0,0,0,1,1,0,1,0,1,1,0,1,1,0,1,0,0,0,0,1,1])
#r = array([0,0,1,0,1,0,1,0,1,0,1,0,0,1,0,0,1,1,1,0,1,1,0,0])
#r = array([1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1])
 
r = []
for i in sys.argv[1].split(','):
  r.append(int(i))
r = array(r)
 
s = (dot(r,geng24_transpose) % 2 )
 
sdotb = (dot(s,B) % 2 )
 
#e = array([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])
e = zeros( (1,24), dtype=int8)
e = e.flat
 
if weight(s) <= 3:
  print "ERROR CORRECTION 3"
  e = array([s, array([0,0,0,0,0,0,0,0,0,0,0,0])])
  e.shape = (1,24)
  e = e.flat
 
if weight(sdotb) <= 3:
  print "ERROR CORRECTION 4"
  e_new = array([array([0,0,0,0,0,0,0,0,0,0,0,0]), sdotb])
  e_new.shape = (1,24)
  e = (e + e_new) % 2
  e = e.flat
 
for j in range(12,24):
  if weight((geng24[:,j] + s)%2) < 2:
    print "ERROR CORRECTION 5"
    e[j] = 1
    e_new = array([(geng24[:,j]+s)%2, array([0,0,0,0,0,0,0,0,0,0,0,0])])
    e_new.shape = (1,24)
    e = (e + e_new) % 2
    e = e.flat
 
for j in range(0,12):
  if weight((sdotb + B[j,:])%2) <= 2:
    print "ERROR CORRECTION 6"
    e[j] = 1
    e_new = array([array([0,0,0,0,0,0,0,0,0,0,0,0]),((sdotb + B[j,:])%2)])
    e_new.shape = (1, 24)
    e = (e + e_new) % 2
    e = e.flat
 
print "\ne =  [",
for i in e:
  print i,
print "]\n"
 
c = ((e + r)%2)
m = array(c[:12])
 
print "r = ", r
print "c = ", c
print "m = ", m

Here is golay.sh

#!/bin/bash
 
echo "Running with Example Problem"
echo "==========================="
./golay.py 1,1,0,1,0,0,0,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,1,0
 
echo "
Running with Problem1"
echo "==========================="
./golay.py 0,1,0,0,0,0,1,1,0,1,0,1,1,0,1,1,0,1,0,0,0,0,1,1
 
echo "
Running with Problem2"
echo "==========================="
./golay.py 0,0,1,0,1,0,1,0,1,0,1,0,0,1,0,0,1,1,1,0,1,1,0,0
 
echo "
Running with Problem3"
echo "==========================="
./golay.py 1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1

Follow

Get every new post delivered to your Inbox.