du vs df differences: the untold story

/dev/mapper/VGsys-LVroot
                       14G   12G  1.7G  88% /
root@SERVER:/> du -hxs /

5.9G    /
root@SERVER:/>

What the hell? why would df report such a different size from du for a filesystem?

Well, there are a few answers, many of them are common.

1) deleted filehandles held open by a process!
That's not the case here.  Server had been rebooted a few times.

2) Sparse files!
Not the case here either.  There are a few sparse files, but nothing anywhere near that big

3) Filesystem corruption!
This is seriously what I thought it was.  But nope.  Filesystem was cherry.

So, do you know what it was?
Two coworkers suggested I remount or bind mount root to another place.
# mount /dev/VGsys/LVroot /mnt2

# df -h /mnt2
                       14G   12G  1.7G  88% /

# du -hxs /mnt2

12G

So, do you know what it was?
There was a subdirectory that had 5g of crap in it that was also a mountpoint.
The du command I ran restricted to one file system, and saw the mountpoint, and ignored anything under it.  Too bad there was actually stuff there.

This was the craziest thing i've seen in a while.

Understand what commands do.

 It can save your ass (or make you look impressive.)


So this morning, one of the noc folks tells me he's been working on and off on a stupid problem thats' been going on for about 6 months.

A nagios check for DELL hardware had been failing for over 100 days.
Part of the check runs a command called "omreport -?" to make sure the omreport binary supports all the features my check needs.

It works EVERYWHERE except this one host.

I spent all afternoon on it. Things finally broke after I straced the whole shebang.

For some reason, the check, instead of running "omreport -?" for some reason was running "omreport -1"
WHY???
I logged into the box, and everything worked fine; but nagios wouldn't work.
So, here's my troubleshooting process (at least all the processes that mattered)

1) What user does nrpe run nagios checks as? Nobody. So lets log in as nobody and try that command.
$ omreport -?

(clunk)

wait. Why does it work on another host? I strace it and i see something weird. Thats where I see that even though I type "omreport -?" i "omreport -1" is getting run.

$ echo -? 
-1
WTF???
Wait.
What does a "?" mean in shell expansion? "one or more"
Is there anything weird in nobody's home directory (which is /)?

$ ls / 
-1 
foo 
blah

ARE YOU SERIOUS

So, a lesson is crazy unix.
when I typed (or tried to run) omreport -? ? is a shell expansion character that'll expand one or more filenames that match -(plus one character) Root's homedir had a file called -1 in it, so it expanded -? to -1. Voila! I removed that stupid file and everything worked.

Truecrypt is dead. Long Live Truecrypt (with LUKS on linux!)

Since there's lots of speculation on the safety of Truecrypt binaries, if you're on linux, you can just forget about it.  LUKS supports truecrypt devices and containers with its own independent implementation.  Here's the TL;DR:

* Open a truecrypt container:

# cryptsetup tcryptOpen cryptfile mapper_device_name

* open a truecrypt container with a keyfile or keyfiles:

# cryptsetup tcryptOpen --keyfile your.keyfile cryptfile mapper_device_name

You can open a hidden or system by using the --tcrypt-hidden or --tcrypt-system switches.

EXAMPLE:

# cryptsetup tcryptOpen --keyfile bunnies.txt omgsecret.tc secretmapper

# mount /dev/mapper/secretnapper
 

Power Management via udev

If you are running Linux on a laptop, there are times you'll want certain settings, or events to happen when your laptop is running on battery, vs AC.  I have a bunch of settings to extract as much battery life as possible via battery, and extract as much performance as I can via AC.  Many existing tools that do things like this automatically are way too general, so there's plenty of ways to do this kind of thing on your own, very easily.

There are plenty of instructions out there on how to run certain actions when your linux laptop is running on battery vs. AC using pm-utils.

Problem is, pm-utils is depreciated, and is being removed in favor of udev/systemd hooks.  Fedora 20 was one of the distros that did this recently.  Here's the simplest way to replace that functionality if you already have pm-utils scripts, or want to develop your own using udev rules.

First, you'll want to create your udev rule.  mine's /etc/udev/rules.d/99-powermiser.rules

**EDIT**  I have a githup repo with these now, with updates: https://github.com/c0t0d0s2/Powermiser

---begin udev rule---

SUBSYSTEM=="power_supply", ENV{POWER_SUPPLY_ONLINE}=="0", RUN+="/usr/local/sbin/powermiser battery"

SUBSYSTEM=="power_supply", ENV{POWER_SUPPLY_ONLINE}=="1", RUN+="/usr/local/sbin/powermiser ac"

---end udev rule

Now, here's my basic powermiser script:

---begin script---

#!/bin/bash


buslist="pci i2c"




case $1 in
true | battery)
echo "Enable powermiser"
echo '1500' > '/proc/sys/vm/dirty_writeback_centisecs';
for x in /sys/class/scsi_host/host*/link_power_management_policy; do
echo min_power > ${x}
done;
echo '0' > '/proc/sys/kernel/nmi_watchdog';
# System bus
for bus in $buslist; do
for x in /sys/bus/$bus/devices/*/power/control; do
echo 'auto' > ${x}
done
done
# USB powersaving
for x in /sys/bus/usb/devices/*/power/control; do
echo auto > ${x}
done
echo '1' > '/sys/module/snd_hda_intel/parameters/power_save';
# cpu
for x in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do
echo powersave > ${x};
done


;;
false | ac)
echo "Disable powermiser"
echo '500' > '/proc/sys/vm/dirty_writeback_centisecs';
for x in /sys/class/scsi_host/host*/link_power_management_policy; do
echo max_performance > ${x}
done;


echo '1' > '/proc/sys/kernel/nmi_watchdog';
for bus in $buslist; do
for x in /sys/bus/$bus/devices/*/power/control; do
echo on > ${x}
done
done
for x in /sys/bus/usb/devices/*/power/control; do
echo on > ${x}
done
for x in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do
echo performance > ${x};
done




;;
esac

---end script---

This script will change power settings for SATA devices, pci devices, and set the cpu scaling governor to either powersave or performance.  You cna easily add functions as you wish.

HELP!  My script doesnt work!
Make sure /usr/local/sbin/powermiser is owned by root, and chmod u+x -ed
Also make sure your udev rule is named ##-rulename.rule (e.g. 99-powermiser.rule) and also owned by root.

SSH socks port forwarding and YOU.

There are times that you'll want to create an on-the-fly socks proxy server for various reasons.  A common one, is because you're behind a restrictive firewall, and want to get outside.  If your restrictive firewall allows you to ssh out, you can get around these restrictions easily.

If you are on a UNIX box, you can start an ssh session like this:

$ ssh -D localhost:9876 username@shell.example.com

Where "shell.example.com is a host outside of your firewall.

If you're on a windows host, using putty, set up your session like normal, then browse to SSH--> tunnels.  Then set the "dynamic" option like in this example then click "add":

Now, to use this SOCKS proxy, configure your client software to use localhost, port 9876 as a socks 4 or 5 proxy.

This works great for viewing websites while at work, if your web usage is restricted or monitored, for instance...

Ubuntu - Encrypted LVM's and Keyfile on a USB stick

I stole howtwos from two places for this.
This post is more to remind me how I did it, so I can do it again.
Source documents:
http://www.debuntu.org/how-to-encrypted-partitions-over-lvm-with-luks
http://ubuntuforums.org/showthread.php?t=555513
http://www.debianhelp.org/node/6797

The steps I took.

First, i installed the system as normal. I always install an Ubuntu system with LVM's from the alternatives CD, so I have flexibility to change partitions easily if needed.

so, I am only encrypting the home LVM. Not too worried about the rest.

1) move the contents of /home to /oldhome.
2)

apt-get install cryptsetup lvm2

lvm2 should already be installed, but i'm paranoid)
3) modprobe dm-crypt
4) My LVM's were already set. I'll be blowing away /home, so:

badblocks -c 10240 -s -w -t random -v /dev/PVNAME/homelvm

Now, let's set up the encrypted device:

5)

# cryptsetup -y --cipher aes-cbc-essiv:sha256 --key-size 256 luksFormat /dev/PVNAME/homelvm

6)

root@ubuntu:~# cryptsetup luksOpen /dev/PVNAME/homelvm encryptedhome
Enter LUKS passphrase: [your passphrase]
key slot 0 unlocked.
Command successful.

7) format the partition with whichever fs you want:

mkfs.ext4 /dev/mapper/encryptedhome

8) Modify your /etc/crypttab:
encryptedhome /dev/SHAZBAT/home none luks,retry=1

YOU ARE NOT DONE YET:

You'll need to add modules to /etc/initramfs-tools/modules
9)

aes-i586
dm-crypt
dm-mod
sha256
# these for the automated keyfile coming up.  if you arent using it, skip:
nls_cp437
nls_iso8859_1
vfat

10) edit fstab, modifying your mount for home like mine:

/dev/mapper/encryptedhome /home         ext4    errors=remount-ro 0     1

11) update initramfs!!

update-initramfs -k all -c

You now have an encrypted home lvm. Move your stuff back, and test it by rebooting. You should get a prompt at boot to type your passphrase.

but passphrases are icky.
Let's now set the machine up so you can boot unattended if a keyfile is present on a USB key:

12) find a USB key. I use a general one formatted with vfat. Create a keyfile:

sudo dd if=/dev/urandom of=/root/keyfile bs=1024 count=4 

13) Create a keyscript. mine is slightly modified from the instructions avove, mounting by uuid and not device.

#!/bin/sh
modprobe usb-storage #load usb mass storage driver
sleep 5 #wait for recognized devices to settle
mkdir /keydev 1>&2
mount -t vfat -o ro,umask=077 UUID=deadbabe /keydev 1>&2
cat /keydev/key
umount /keydev 1>&2

(if you need to find your uuid, use the "blkid" command)

14) add your keyfile to the encrypted lvm:

cryptsetup luksAddKey /dev/PVNAME/homelvm /your/keyfile 

15) change your /etc/crypttab to this:

encryptedhome   /dev/SHAZBAT/home       none    luks,keyscript=/sbin/keyscript
encryptedhome   /dev/SHAZBAT/home       none    luks,retry=1

This way, if you forget your keyfile, you can still use your passphrase.

16) reboot!

Random Solaris Tricks: reading tagged 802.1q packets

This works on Solaris 10. Dunno about what other Solaris-es it works on.

If you have an environment where you have various tagged vlan packets, and you want to create subinterfaces that attach to each vlan, you can do that in solaris 10.

For example:
You have three vlans on one port, each tagged differently. In this example, we have vlan 100, 120 and 134.

You configure your interfaces like this:

ifacename = (tag# * 1000) + interface #

So for instance, if you have hme1, you'd configure hme1 3 times:

hme100001
hme120001
hme134001

Each of these three interfaces will be managed the same way solaris interfaces normally are (i.e. each has an entry in /etc/inet/hosts and /etc/inet/ipnodes, and each interface gets its own /etc/hostname.interface file)

(hat tip Jarett)