This was the craziest thing i've seen in a while.

Understand what commands do.

 It can save your ass (or make you look impressive.)


So this morning, one of the noc folks tells me he's been working on and off on a stupid problem thats' been going on for about 6 months.

A nagios check for DELL hardware had been failing for over 100 days.
Part of the check runs a command called "omreport -?" to make sure the omreport binary supports all the features my check needs.

It works EVERYWHERE except this one host.

I spent all afternoon on it. Things finally broke after I straced the whole shebang.

For some reason, the check, instead of running "omreport -?" for some reason was running "omreport -1"
WHY???
I logged into the box, and everything worked fine; but nagios wouldn't work.
So, here's my troubleshooting process (at least all the processes that mattered)

1) What user does nrpe run nagios checks as? Nobody. So lets log in as nobody and try that command.
$ omreport -?

(clunk)

wait. Why does it work on another host? I strace it and i see something weird. Thats where I see that even though I type "omreport -?" i "omreport -1" is getting run.

$ echo -? 
-1
WTF???
Wait.
What does a "?" mean in shell expansion? "one or more"
Is there anything weird in nobody's home directory (which is /)?

$ ls / 
-1 
foo 
blah

ARE YOU SERIOUS

So, a lesson is crazy unix.
when I typed (or tried to run) omreport -? ? is a shell expansion character that'll expand one or more filenames that match -(plus one character) Root's homedir had a file called -1 in it, so it expanded -? to -1. Voila! I removed that stupid file and everything worked.