Yeah, I've read them too. Lists of shell tricks you already know - pstree (wow!) bc (bash already has built-in math), and a dozen commands you see in every Linux site, book, and training course.
Here's a list of damn useful commands you haven't heard before.
1. A Simple way to Send Output and Errors
Want to send output and errors to the same file?
command &> file
Maybe you're troubleshooting a problematic app with strace, and want to see the system calls at the same time as the apps errors?
strace badapp &> errors_and_output
Benefit: Easy to remember, and simpler than 'send errors to output, and then send that to a file'
Works in: any currently supported Linux
2. Parallelize Your Loops
Almost every Linux network administrator knows the power of the for loop: the way to do something for one, one hundred or one thousand users, files, machines, processes, or whatever else. Most people set their loops in sequence – so that each jobs is finished before moving onto the next.
But the jobs command can be used to background each loop, so you don't have to wait for it to complete before continuing with the next.
Here's an example running an apt-get update:
for HOST in $(< ListOfHosts); do ssh $HOST 'sudo apt-get update' & done
Maybe you need a bunch of SSH tunnels running simultaneously:
for HOST in $(< ListOfHosts); do ssh -C -N -R 80:localhost:80 $HOST & done
Sometimes you probably don't want to see all the output as it happens – in that case, save a file on each machine and use another loop to collect it later.
Benefit: Saving a metric shitload (2/3rd of an imperial shitload) of time waiting on stuff to finish
Works in: any currently supported Linux
Drawbacks: Bash probably has some limits of the amount of concurrent jobs. But I've yet to run into them.
3. Catch Memory Leaks By Using Top via Cron
Memory apps are rare in Linux, but they do happen, particularly when using beta distros or home grown software. A lot of time the identitty of the app with a memory leak isn't that apparent. Linux has an Out-Of-Memory app built in to identify and kill these apps, but by the time it eventually kicks on your system may have been unusually slow for a while – and that's if your patience hasn't already worn thin and you've rebooted.
The normal way to find an apps memory consumption is by running top (or one of it's graphical equivalents, like System Monitor) and check the Resident Set Size (called Res or RSS) of the processes you care about (you can ignore figures for how much memory the app has allocated – memory leaks come from usage, not allocation, and apps can assign bucketloads of memory they don't use without hurting your system). Most people aren't aware top can be run non-interactively, which means you can use cron and top to generate a simple report of an apps usage over time.
- Run top.
- Use the < and > keys until processes are sorted by RES (resident memory usage).
- Hit W to write your config out to a file
- Add a cron job:
crontab - <<< '*/15 * * * * top -n 1 -b'
You'll now get an email every 15 minutes with the top output.
Benefit: way less complicated than adding software like SAR
Works in: any currently supported Linux
Drawbacks: Has some limitations of the amount of concurrent jobs.
4. Standard in directly from the command line
Wondering what the hell the <<< above was? Bash allows you to send programs stdin directly from the command line.
Benefit: Let's you write your command on the goddamned commandline, even for weird creepy programs that want you to do everything via standard in. Shakes fist at MySQL.
Works in: Bash 3 and newer.
Drawbacks: Still quite a few Bash 2.x systems out there.
5. Set a Random Initial Password, That Must be Changed
There's a lot of organizations who have nice, secure policies for passwords. Passwords stored on Windows machines. Linux is either not covered by the policy or the policy is routinely violated – people have idea about Linux authentication (most people don't quite understand PAM, and Linux admins don't often realize Linux can quite happily authenticate to Active Directory) and once upon a time, the OpenSSH developers didn't like PAM (that's since changed).
To set password that must be changed upon first login.
umask u=rw,go=
openssl rand -base64 6 | tee -a PasswordFile | passwd –stdin joe
chage -d 0 joe
The password is saved to PasswordFile , which only your own account can read. Then contact via some medium you consider relatively secure – like a phone call or encrypted email and them tell their initial password.
Benefit: Ensures users aren't using your default password forever
Works in: any currently supported Linux where OpenSSH has been updated (if your users use SSH to do their first login). Red Hat still say this doesn't work in the RHEL 3 / 4 documentation, but with their own updates applied, it's AOK.
Drawbacks: None
6. Add Your Public Key to Remote Machines the Easy Way
In order to perform key based logins to a new machine, you need to get a copy of a public key to the remote machine yourself. Sure, you could do this manually – which gets a bit boring after a while (why doesn't SSH have an authorized_keys.d anyway?), but why waste time when SSH comes with it the tool to do it?
Just run:
ssh-copy-id -i .ssh/id_rsa.pub hostname
After being prompted to enter your password for the last time, SSH will say:
Now try logging into the machine, with "ssh 'hostname'", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
Try it. No more passwords!
7. Extract an RPM without any additional software
This one isn't necessary on Debian based distros, as .deb are merely ar archives. Every Red Hat guide ever written mentions using rpm2cpio (which comes as part of the default rpm package), but frankly I can never be bothered remembering the weird syntax to cpio, the ancient archive format used by …uh, pretty much just rpm.
The following command installs a package to a temporary directory, and but doesn't modify your RPM database (just one in the temporary directory, whose contents you can delete afterward). Since the temp directory doesn't have any othr software in it, we also disable dependencies and scripts.
mkdir /tmp/deleteme
rpm -ivh –root /tmp/deleteme –nodeps –noscripts package.rpm
8. See How a File Has Changed from Factory Defaults
This is a simple troubleshooting tool when you're not sure how a file has changed from its defaults. First identify the package that owns the file:
dpkg -S /etc/foo/foo.conf
rpm -qf /etc/foo/foo.conf
Then extract the original package either with tar (DPkg) or the rpm trick above (RPM) and run:
diff /etc/foo/foo.conf /tmp/deleteme/etc/foo/foo.conf
And see the differences.
Benefit: Faster troubleshooting of bad config files (note strace is also handy in these cases)
Works in: any currently supported Linux
Drawbacks: You have more time free at work, to spend reading Digg.
9. Undo Your Network Screwups After You've Lost the Connection
Messing with a firewall or a network over a remote connection? Nerve wracking isn't it? Change the wrong setting and you'll be locked out, unable to fix it.
So why not undo your mistake? Schedule a job to run at a later time that undoes what you're about to do.
at now + 5 minutes <<< 'cp /etc/ssh/sshd_config.old /etc/ssh/sshd_config; service sshd restart'
If you screw up, the job will run and restore things to the way they were.
If you your change works, then just run atq to check the queue of upcoming at jobs, and atrm jobNumber to remove it.
Benefit: Gets you back in after you lock yourself out.
Works in: any Linux provided atd is enabled - which is usually the case.
Drawbacks: Remembering to do it before you make the risk change.
10. Check a Port is Open
Want to check whether a network service is running before you use it? Netcat can be used to easily connect to ports, and has a rather handy wait -w option to tell it how long to wait for.
nc -w 3 server ssh <<< ' '
Would connect to the ssh port on a machine called server, and wait for up to 3 seconds before sending it, er, nothing, and closing the connection. Whether the port was open will be reflected by nc exit status.
if nc -w 3 localhost 22 <<< " &> /dev/null
then
echo 'Port is open'
else
echo 'Port is closed'
fi
Here's a few bonus tricks, which you may already know…
Bonus Trick 1: The Easy Way to Extract Tar Archives
New to Linux? Don't feel like using File Roller to extract archives? Despite everything you rad on the internet, tar (the tape archiver - that why you have to tell it you want to use a file with -f) doesn't need you to specify archive file formats any more. To extract a file, simply:
tar -xf archive.tgz
But keep reading…
Benefits: No stupid messages from a computer asking for information the computer can determine on its own.
Works in: Recent distros (last year) only
Drawbacks: It's still to early to use this in a lot of distros.
Bonus Trick 2: Use Math Shells
Most of you already know this, but since digg.com keeps linking to articles about 'bc', it's worth pointing out that Bash comes with it's own math shells. These can be invoked like an ordinary subshell, but by using two round brackets. Say you have a n script that needs to figure out disk space.
SIZE_IN_KB=204535848
SIZE_IN_GB=$(( $SIZE_IN_KB / 1024 / 1024 ))
echo $SIZE_IN_GB
Benefit: No need for an extra process like bc every time you need to work with number
Works in: any currently supported Linux.
Drawbacks: If you want to do floating point or other advanced math, you'll probably want to bust out python.
Bonus Trick 3: Never reboot a system for NFS failures
OK, this isn't a shell trick. It's more of a general thing for NFS that not enough people know. We've included it here because VentureCake loves you.
At some point every Linux admin has had a problem with a computer using a hard-mounted NFS export, where the connecton to the server has been lost - perhaps the network had a problem or the server went down. Any processes which check the status of filesystems - df, rpm, etc. - will hang, waiting on the storage to respond. Next time, you'll want to mount using the intr option (not soft - see the Linux NFS FAQ). This time, run:
killall -KILL rpciod
rpciod (the kernel process that handles NFS IO), will instantly respawn, sending errors to processes waiting for NFS IO, causing them to respond. If you're mounting exports from multiple NFS servers and only wish to time out a single connection, you can do so with:
iptables -A OUTPUT -d nfsserver -j REJECT
Within about a minute, the NFS client will decide the server is unreachable. Again, the processes start responding.
You can now unmount the NFS server. No need to reboot.
Benefit: No need to reboot when an NFS mount fails.
Works in: any Linux.
Drawbacks: You can't disable an individual NFS export, just all the exports from A particular NFS server. Still beats rebooting though.
Bonus Trick 4: Encourage Others to Use & Contribute to Your Scripts
If you want to improve your scripting skills, it pays to be kind to your peers.
- Use self-explanatory uppercase names for your variables. In particular, this means not using 'i' as a variable name, so when your fellow scripter is eighty lines down your twelfth, nested, for loop, they don't have to scroll up and work out what the hell 'i' means now, when it's much easier to come to your house and kill you with an axe.
- Keep your loops and conditionals indented
- Putting your functions at the top of the script, and check input.
Bonus Trick 5: Shell Sites That Don't Suck
There's a lot of sites on Linux shell commands. Very few are Bash specific, so you'll be missing out on a lot of the good stuff. They also tend to be non-task oriented - if they need to show you grep, they'll show it using some weird thing about animals, rather than, say how to strip comments and blank lines from a file.
grep -vE '^$|^#' /etc/foo.conf
…by the way. Anyway, here’s a few of our personal favorites:
Tips from an RHCE - Part of Red Hat Magazine, but useful even if you’re not into Red Hat.
SHELLdorado - Not Linux specific, and a little out of date, but very practically oriented- lots of sample scripts you can pillage and plunder.
Handy Sed One Liners - Another ancient document, but the examples covered in it show 99% of what you want to use sed for.
