wget login pages

how do you scrape a page that you have to login to get to? Well, one way is to save the cookies and use –post-data, though this depends on how the session is saved.

$ wget http://site/login/index.php –post-data “username=user&password=pass” –save-cookies=cookies.txt –keep-session-cookies

then to grab other pages

$ wget –load-cookies=cookies.txt http://login/someotherpage/index.php

Bash Bomb

So my buddy Greg pointed me to what he called a ‘bash bomb’.  It looks like:

:(){ :|:& };:

Anyway, all it does is recursively fork.  http://www.cyberciti.biz/faq/understanding-bash-fork-bomb/ gives a good explanation.  I do like it, because of its simplicity and obscurity. I have to deal with recursively forking things all the time (thanks operating system class with students experimenting with fork for the first time).

A simple pam hard limit on the number of processes can mitigate against this.  Put it in /etc/security/limits.conf.

My applicable limits are (fairly liberal):

*               soft    nproc           225
*               hard    nproc          300

So far, nothing has crashed the system with these, but I keep having to tweak them, so I may restrict them further in the future.

Really global environment variables for ssh

<mopey> how do I export a variable in pvm?  I add it to my .bashrc or .profile but it ignores it.
<mopey> an environment variable
<mopey> Because I get this error:
<mopey> The value of the $PVM_ROOT environment
<mopey> variable on compute-0-1 is invalid (“”).
<mopey> Use the absolute path to the pvm3/ directory.
<mopey> but if I ssh into compute-0-1, echo $PVM_ROOT it is set correctly
<staynalive_> mopey: I don’t know much about PVM
<staynalive_> but I would check to see if it gets set for non-login sessions
<staynalive_> by doing (in one command) “ssh compute-0-1 printenv”
<staynalive_> Yeah, I just tested it and that’s the issue.
<mopey> good call, it’s not being set for some reason, although it’s “being set” in ~/.bashrc
<mopey> where would I set it, if not bashrc?
<staynalive_> Umm
<mopey> my .profile calls bashrc btw, although that shouldn’t really matter since that’s only on interactive logins, right?
<staynalive_> Yeah
<staynalive_> I think I actully changed a ssh flag to carry the environment variables through to the new machine in a ssh session.
<staynalive_> “PermitUserEnvironment yes”
<mopey> oh.  well that’s handy.
<staynalive_> That way if users set something up funky they can carry it to the nodes.
<staynalive_> But the manual warns of some possible security issues…
<mopey> if someone is on my frontend node, it’s only being used on the compute nodes, so it shouldn’t be that big of a deal.
<mopey> since they are basically thin
<mopey> aaah, ssh has env variables all of it’s own…
<mopey> I remember telling you that at one point.  I guess I’m losing my marbles

<mopey> has anyone ever gotten sshrc to succesfully set ssh environment variables?
<mopey> it *should* be straightforward
<mopey> the sshd man page says:  8.  If $HOME/.ssh/rc exists, runs it; else if /etc/ssh/sshrc exists, runs it; otherwise runs xauth.  The “rc” files are
<mopey>  given the X11 authentication protocol and cookie in standard input.
<mopey> is this not run (when I do ‘ssh compute-0-0 env’) because it is too late in the process?
<mopey> because my $HOME/.ssh/environment _is_ run, and I can set them that way.  Except that I want to do it for all users and that seems to be a lame solution.
<mopey> It seems like the command should be executed *after* the rc files are read (it’s step 9)
<mopey> So I wonder why the hell it’s being ignored…
<mopey> staynalive, you said you use “PermitUserEnvironment yes”.  So do you just set a $HOME/.ssh/environment for each user?

<mopey> haha
<mopey> for those who care, pam overrides my ssh variables for the most part
<mopey> so you can define variables in /etc/security/pam_env.conf
<mopey> god, that took forever to figure out why my /etc/environment variables weren’t getting set over ssh
* Vog-work has quit (“ChatZilla 0.9.79 [Firefox 2.0.0.10/2007111504]”)
<twinprism> thanks for sharing, mopey, I care…
<mopey> weird.
<mopey> :)
<_sera> I don’y
<_sera> sheesh… don’t
<mopey> Normally it probably doesn’t matter I’m sure.  Like if you get a bash shell and can actually execute profile/bashrc
<mopey> But if you have a crippled pvm shell or something, it’s way important
<mopey> plus I think pam_env is how PATHs and junk get set on login – at least on ubuntu
<mopey> *gdm/kdm/xdm login

Build a Bridge and Get Over it

In my environment, the best kind of firewall is the bridge kind of firewall. Meaning: a transparent kind of firewall. Here’s how I generally set them up.

So, I don’t have control of the gateway routers. I have IP addresses all over the place in my little IP pool. For example I have one little server room with a 134.50.7.24 a 134.50.7.88 and a 134.50.7.244. What I want is a firewall complete with IDP to limit access to stuff in a centralized way. BTW, I also tend to have firewalls on every machine, I guess for the double layer of protection. Firewalls are sort of like condoms. Double layering doesn’t seem to help that much, but it couldn’t hurt. And either way, once you’re penetrated, you’re fucked.

To clarify, what I want is something completely invisible to the outside. As far as the outside is concerned, there is nothing there. In fact, this could be the case as far as the inside is concerned. However, you may want to give the bridge an IP address and let something through so you can ssh in to fix whatever problems.

So:

Network —– Bridge ——switch —Internal Network of various IPs

The cool thing about a bridge is you can stick it anywhere and it won’t change how any of your other hosts are hooked up.

For this task, I chose everyone’s favorite os, Linux. I also chose a minimalist version of Ubuntu since I heart ubuntu. This is actually a pretty trivial thing to set up.

# apt-get install bridge-utils

Then you want to create a new bridge device and add your ethx to it. Your ethx will obviously probably be eth0, eth1, eth2 or whatever network devices you want on the bridge.

# brctl addbr br0
# brctl addif br0 eth0
# brctl addif br0 eth1
# ip link set br0 up

Now guess what. You’ve now turned your $300 machine into a $10 hub.

Next if you want to give your bridge an ip address and a route:

# ip addr add x.x.x.x/x brd + dev br0
# route add default gw x.x.x.x dev br0

Where x is your ip and your subnet and gateway.

I find as I’m writing this I’m just repeating what else is out there. Go and read http://www.linuxjournal.com/article/8172 and it will show you the way.

Now that you’ve got your bridge working you can filter traffic using ebtables (on the hardware level) and iptables (on the ip level) and snort and whatever. There are tons of examples of this online, so I won’t bore you here. Awesome.

Put all this in rc.local or whatever boot up script so that your system remembers everything and you’re golden! To all those people who say “just use smoothwall and don’t worry about iptables” in your face! smoothwall/ipcop/monowall are all pretty cool, but being a transparent bridge isn’t one of the things they are capable of out of the box. They are mostly meant to be gateways or whatever.

Matching Regular Expressions that don’t end with…

Regular expressions do not mix well with syntax that requires memory, such as XML. I was trying to add a <br /> tag to every line that did not have a </p> tag. so for example I can print the strings I want with grep -v ‘^.*</P>’

Anyway, this turns out to be a bear, because (?!expression) just isn’t working for me with sed, although I think google says it should.

So what do I do? I make two!

s/(w.*</p>)/1<br />/g

adds a <br /> to every line with a word.

s/(.*</p>)<br />/1/g

take off the <br /> for lines that have a </p>

The good thing about this is it should work with all standard regular expressions, unlike that look ahead stuff which may only work with certain utilities.

You could run this with sed, vim, perl, python, whatever, and it will work.