How To Adopt 10 'Good' Unix Habits 360
An anonymous reader writes to mention an article at the IBM site from earlier this week, which purports to offer good Unix 'habits' to learn. The ten simple suggestions may be common sense to the seasoned admin, but users with less experience may find some helpful hints here. From the article: "Quote variables with caution - Always be careful with shell expansion and variable names. It is generally a good idea to enclose variable calls in double quotation marks, unless you have a good reason not to. Similarly, if you are directly following a variable name with alphanumeric text, be sure also to enclose the variable name in square brackets ([]) to distinguish it from the surrounding text. Otherwise, the shell interprets the trailing text as part of your variable name -- and most likely returns a null value."
FP? (Score:5, Funny)
Re: (Score:2, Funny)
Re: (Score:3, Informative)
-Lasse
welll.. (Score:5, Funny)
I seriously doubt reading this article is going to get anyone to start showering on a regular basis.
Re:welll.. (Score:5, Insightful)
Ie there is NOTHING bad about piping cats. While you might indeed get a ~30% performance increase if you skip the cat, the complexity increases. We often sacrifice performance in order to increase abstraction and understanding.
What makes unix so powerful is its modularity, the fact that you can pipe any output from any application to any applications stdin. This makes it possible to use common tools app1 | app2, app1longoutput | grep thingsIwant. The possibility to mix and match common elements that (arguably) makes unix powerful.
Advice that says "stop piping cats" is akin to "stop using helper functions, they overload the stack, instead do everything in one function"
--
A better articulated article on the programmers intellectual ability vs proper abstraction techniques:
http://www.acm.org/classics/oct95/ [acm.org] - Dijkstra, Edsger - "Go To Statement Considered Harmful"
Re:welll.. (Score:5, Funny)
PETA would disagree.
Re:welll.. (Score:4, Funny)
PETA would disagree.
Oh? Just imagine the cacophony of a bunch of cats playing bagpipes while "singing" along.
Re: (Score:3, Insightful)
Re:welll.. (Score:5, Insightful)
Further, the assembly line abstraction of cat as 'input the contents of these files into the beginning of my pipeline' is predictable, simple and very clear and readable. Using the filenames in the commands means you have to be certain each command will take filenames, and if you replace the first step (from a grep to an awk, for example), you have to rethink your input method semantics again.
Any typing speed gains and performance improvements you may get will probably get shot the first time some command does something unexpected, or by the extra steps of thought.
And if performance really was a serious concern you probably shouldnt be writing it as a shell script...
Piping cats (Score:3)
$ <file.txt grep foobar
Few people know that input-redirection can be established before the command name
Re:welll.. (Score:5, Informative)
That list was fairly arbitrary, but the piping cat thing is something that basically only annoys the most anal of anal, and they probably do it sometimes too.
Its common for me to do cat foo and then hit the up arrow and append a pipe to another command instead of editing the whole command line. Computers are pretty fast, and real anal people would use fgrep instead of grep, but again I always use egrep, because I never know when a regular expression will be edited into a more complex one, and to me all of the speeds are the same.
My #1 habit to tell people, although it is not a habit, but just where to start it to learn your shell. No science guys, csh is not a worthy shell in 2006. If you have to suffer with the wacky behavior of a csh variant, at least use tcsh.
My #2 thing to learn is a text editor.
As far as habits go. First and foremost, unalias cp, mv, rm to have the -i flag. In my opinion, that is a BAD habit to start. You WILL lose files sooner or later, and the more painful the better so that you will think so you will stop doing it. the -i flag will NOT stop you from redirecting into a file, and the most dangerous is the -rf flag with rm will override that -i. Remote copies via rcp or scp will not honor the -i flag. Unarchiving an archive will not honor the -i flag. There are tons of ways to lose files, and you will lose them. Its a much better habit to universally save yourself from yourself to not lose them by testing with -i, working off of a copy, and thinking before you hit return, creating new directories to eliminate clobbering a file, NEVER, EVER, do tar cf foo.tar
Actually, this top 10 list is pretty lame, and should be ignored.
Re: (Score:2, Informative)
a.txt: foo
b.txt: foo
c.txt: foo
whereas cat ?.txt|grep foo produces
foo
foo
foo
I've also seen Unixes where their shells are linked against spectacularly broken libc's. Under Tru64's Bourne and Korn shells, for example, a multithreaded program foo fork bombs when run as foo < z.txt, but works fine as cat z.txt|foo (foo < z.txt under Bash works, though, because Bash is linked against the GNU
Re:welll.. (Score:5, Insightful)
I always use 2 or more args for grep (Score:3, Insightful)
Re: (Score:3, Insightful)
Re:welll.. (Score:4, Informative)
First you are telling cat to output the entire file, and then you are telling grep to go through the entire output of cat. If you're working with gigabytes of data here, that can quickly be a frustrating exercise! Folks who are in the mentality of using cut | grep and even a visual editor like vi instead of sed are up the creek when they find themselves needing to manipulate and get portions of very large data sets.
Re: (Score:3, Funny)
Square or Curly brackets? (Score:4, Informative)
Re: (Score:2)
I think somebody needs glasses ;)
Re: (Score:2)
I think somebody needs glasses
Anyway...Curly brackets? Nyuk nyuk nyuk!
Re:Square or Curly brackets? (Score:5, Informative)
Typo (Score:3, Informative)
mkdir (Score:3, Insightful)
$ cd tmp/a/b/c || mkdir -p tmp/a/b/c
If the directory exists you end up in the directory, if it does not it creates the directory but leaves you where you first started. Hence you don't know which directory you will be in after the command is executed!
Comment removed (Score:5, Informative)
Re: (Score:2)
( A=tmp/a/b/c cd $A || mkdir -p $A && cd $A )
But this is silly: mkdir -p succeeds even if the directory already exists, so this suffices:
mkdir -p a/b/c && cd a/b/c
I also resent the use of zillions of little command options to "be more efficient". The -C to tar is just plain silly:
( cd a/b/c ; tar xf $ELSEWHERE/foo.tar )
does the same thing, withouth having to look up the fricking -C each time you want to use it. Three guesses which technique has th
Re: (Score:2)
The latter also works nice
Even that's wrong / Ich bin Grammatiknazi (Score:2)
Only that isn't portable. Some flavors of unix zcat will try to uncompress $1.Z ; clearly there wouldn't be a arc.tar.gz file.
Fortunately, there's an easy fix for that: Use redirected stdin rather than a filename for zcat :
zcat <arc.tar.gz | ssh user@foo 'cd tmp/a/b/c ; tar xvf -'
Some people call me a 'Grammar Nazi' for being so picky about how people use English. I get frustrated
Preview is my friend! (Score:2)
Re: (Score:3, Interesting)
gzcat will never be broken in the way described, hence the following is fine and portable IME:
gzcat arc.tar.gz | ssh user@foo 'cd tmp/a/b/c && tar -xvf -'
HOWEVER, I find that even vaguely modern CPUs are much faster at gunzipping than typical internet speeds. So, I would use this myself:
cat arc.tar.gz | ssh user@foo 'cd tmp/a/b/c && gzcat | tar -xvf -'
On the otherhand, I would never actually write that,
Re:If I wanted to upload binaries... (Score:4, Insightful)
1. Never use csh or any derivative thereof.
2. Know the portable behaviour of your Unix tools.
3. Learn to use ed, one day you'll be glad you did. You can also use ed and ex from scripts or from a command.
4. A shell command is a small program. If you are unsure about a command, test it first, like you would any program.
5. Learn to use the standard shell on your system.
6. Learn useful nonstandard extensions of utilities, but use them with care.
7. Never rely on an extension to the point that you forget how to do it portably. The definition of "portably" is up to you.
8. Learn to use csh enough that you can make do in an emergency, and learn *why* you shouldn't use it.
9. If your standard shell is Bash, learn Korn too. And vice versa. Learn both, how they differ, and how they differ form your standard shell.
10. Sometimes a real C program or a script in a different language is better than using shell.
-Lasse
Re: (Score:2)
mkdir -p tmp/a/b/c; cd tmp/a/b/c
Re:mkdir (Score:5, Insightful)
- scripts
- commands that take long enough that you go have a coffee.
This makes sense:
make install && lilo && reboot
This doesn't:
cd tmp/a/b/c || mkdir -p tmp/a/b/c
If you fail the first part, well, you typed " || " instead of pressing enter.
If you succeed the first part, you typed " || mkdir -p tmp/a/b/c" without a bloody reason.
Type first part. Press enter. Observe result.
If necessary, type the second part, otherwise correct the first without baggage of the second one hanging around.
Re: (Score:2)
First, let me be clear that this list is bogus. These 'Good' habits are not that advantagious at all, and I've been using UNIX for quite some time now, and I'm pretty good at it.
Now, being pedantic, the cd a/b/c || mkdir -p
Re: (Score:2)
His example of good habit with mkdir did not convince me
$ cd tmp/a/b/c || mkdir -p tmp/a/b/c
If the directory exists you end up in the directory, if it does not it creates the directory but leaves you where you first started. Hence you don't know which directory you will be in after the command is executed!
Actually, the example is a good habit.
If the change directory fails, the script will not make the directory in the wrong place. It is a good practice.
Unix is more than just a shell (Score:2, Insightful)
(plus he didn't mention my favourite shortcut: shell history)
How about being more inclusive and expanding this to deal with security features (surely the single biggest benefit?) and the ease of working on remote boxes?
Re: (Score:2)
Mod parent 'redundant'.
Shell code in linux based devices (Score:2)
Hey, FON [slashdot.org], are you reading [datenbruch.de] this? Sadly, even shell code in comsumer devices is often deeply flawed. Which is a good thing, if you want to install a new firmware - and a bad thing, if an attacker does it for you.
This article... (Score:5, Informative)
Re: (Score:2)
Agreed, the article is incorrect regarding use of tar's -C switch.
This works: tar -C
This doesn't: tar xvf -C
I agree that '&&' is a better practice to encourage than ';'. Chaining commands without heeding their exit status can make a real mess and cause much confusion.
Aside from that I don't think it's a bad top-ten. I think there should've been something in there about command completion. If productivity is important to this
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
This article is about Unix, where there is no such option (see the specification [opengroup.org]).
You may be thinking of GNU tar. I suggest you look up what the acronym GNU stands for, with particular reference to the N.
(This is not to dispute that the example given in the article is broken.)
Re: (Score:2)
Well, at least Michael Stutz compensated for the lacking -z in tar by adding a superfluous -1 option to ls at the beginning of a pipe. I think most ls implementations check whether stdout goes to a terminal to decide whether output should be columnized.
-Lasse
Re:This article... (Score:4, Informative)
If you are encountering data created by untrusted users, don't forget the strange consequences of filenames that contain newlines.
Failing to use -0 is dangerous malpractice.
Very helpful (Score:5, Funny)
I am so glad that he showed what a difference can make, because I was *really* getting annoyed at having to wait that extra
Re: (Score:3, Funny)
More to the point (Score:2)
That's why I'm of the v7 Unix philosophy. Make a tool that does one thing, and does it well.
If I later need to optimize for speed, then I'll read the man page.
Re: (Score:3, Informative)
When timing commands, it's best to repeat the command several times and see if the times change significantly.
Re: (Score:2)
I am so glad that he showed what a difference can make, because I was *really* getting annoyed at having to wait that extra .084 seconds.
But if in perl, python or java??? Worse yet, for for loop over 1 million files?
Things I had to learn the hard way (Score:5, Interesting)
#rm -r -f / tmp/dir
when "all" you wanted was
#rm -r -f
instead do this:
#(cd
or even better use sudo if you have it:
$(cd
2. When logged on as root or when using sudo on a production system think things over
at least twice before hitting enter.
3. Make sure at all times you're on the right machine, logged on as the right user in the right directory.
Set up your shell prompt to look like this user@host
Re: (Score:2)
% cd ~
% cd somejunk
% ls
just junk, junk...
% rm -Rf *
somejunk: permission denied
% su -
Password:
# rm -Rf *^H^H^H^H^H^H^H^Hpwd
/
Re: (Score:2, Interesting)
You are forgetting one thing - there's no solution for stupidity and lack of common sense. While tips like these are generally useful, the person who's going to screw up a system will ignore you, and the zillion other tips that people have taken efforts to write. I've seen people who've run a rm -rf /bin to clean the recycle bin and then wonder what happened.
Its hard to bring in any improvement in the average unix admin,
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
This is exactly why I prefer the US/UK keyboard for unix stuff, even if I live in Finland, Finland, Finland, the country where slash needs a shift.
There are similar annoyances with a few other punctuation characters in the Fi keyboard. Besides, it's easy to setxkbmap when you need to type with your native language.
Re: (Score:2)
absolute drivel (Score:4, Informative)
Re: (Score:2)
Mostly BS (Score:2)
Anal Unix Guy (Score:2)
These things he is stating is like when an other Unix guy is looking over your shoulder and you type in a command that works perfectly fine but has to correct you to do it his way, to save the extra millisecond in CPU while wasting about 5 Seconds to retype the command.
Secondly depending what you are doing on Unix you may or may not need to use these commands often enough to remember all the parameters for every tool. like the mkdir -p, While I know mkd
Re: (Score:3, Insightful)
Yes -- and habits is what people desperately need. The people I know primarily need three habits: RTFM when they don't understand something; adjusting their behavior based on the FM; and managing their use of the current directory (i.e. you don't have to cd into a directory to use a file which lives there).
Re: (Score:2)
Interesting... I'm used to using mkdirhier for the same effect. I thought it was simply an alias to mkdir -p, but it turns out to be a shell script (part of X11 imake package) that uses plain mkdir. I better update my scripts to the more portable version now...
Eh? There's more to Unix than shell scripting (Score:3, Informative)
"10 good habits that improve your UNIX command line efficiency" would probably have been a better title.
The title did however bring back fond memories of Eric Raymond's The Art of Unix Programming [catb.org]. The book is available online, and if you were hoping for something a bit more substantial as well, then the section Basics of the Unix Philosophy [catb.org] might be worth a read.
Re: (Score:2)
tar comment (Score:3, Informative)
tar xvf
without the dash. (E.g., tar -xvf)
While that does work, I prefer to add
the dash as it makes it more consistent
with the other commands. So I consider
that a bad example. tar is one of the
older commands like dd that have weird
command line syntax.
Actually useful hints (Score:5, Informative)
In scripts, prefix dangersous commands with an 'echo' for a test run (So you can catch all those rm -rf
Single quotes are the best quotes for plain strings. The only reasion to use double quotes is if you need to quote a variable or a single quote.
Completion is fun, but using wildcards is more flexible (though you'll only want to use benign commans like cd, less, etc):
nano
nano
Note that the use of subpaths reduces the amount of flexibility.
cd
cd
nano
Finally, as a comment for the article, using:
test -e $DIR || mkdir -p $DIR
is much better than their suggestion and probaly faster anyway. Though I'd just do "mkdir -p $DIR" and maybe "&>/dev/null" under most circumstances anyway.
That's all I can think of at this point. Anyone else have tips?
No actual habits in the article (Score:5, Insightful)
Some good, some bad, some plain pointless. (Score:3)
mkdir -p is a convenience people should be aware of, but telling people to start getting overly creative with the shell expansion behavior is asking for mistakes/trouble. Also, having a mkdirhier script in case the example isn't supported on all shells is an indication that you shouldn't get overly cozy if you are going to be dealing with a lot of different systems/users with different default shells. The amount of time a lot of people take to figure out the 'clever' way in terms of how to phrase the expansion so the shell will expand it right is often longer than just typing the two lines more that the less thought takes. Not saying this isn't useful, but in my experience too many people mess things up too frequently or take too long to think up the expressions to tell them trying to be clever ends up taking more time than they think they are saving.
Change the path instead of the archive is not that dire to do normally, but if you avoid it, to me it's just easier to be in the target directly and use full path to the archive.
On combining commands, I second that ; can be dangerous and && as a default will make the chain more ready to break, but again I say not trying to be so clever as to put all you want on one line. Some things go wrong that aren't reflected in return codes, doing it one at a time let's you think of those. True, though, that the && never assume the first command works, while your fingers may keep moving and hit enter on next command before your brain realizes the command failed, so && may have merit, but then again taking your time may have more merit.
On the quotation thing, true enough, you must understand how quoting works to do remotely complex things, particularly nested circumstances (i.e. ssh to a system to run a command, where the output will be parsed by two shells.)
On the breaking up long lines thing, in a shell script it may be more necessary, but on an interactive command line it could also indicate you are trying too hard to do things in one chunk. I admit sometimes it does get too wide, but particularly less experienced admins should consider if there were a simpler way to do it in smaller chunks they won't screw up.
Grouping commands is important to know, and harmless (better than repeating the same pipe over and over and more powerful).
I will say xargs is way way over-rated. Too many people, particularly dealing with directory trees containing spaces, get into trouble piping the output of find into anything when IFS causes something like "/tmp/Monthly Report" to be parsed as two different files. find has a competent filtering mechanism (-type, -iname, -name, etc...) and it's own -exec. find is well aware of the state of each file. You could assign IFS to try to avoid it, but using find's built-ins where possible alleviates it.
When you are talking about interactive shell operation, picking the
much larger demonstration that would have been useful, but this just makes people mock the example. In any event, this seems like an okay thing to convey, but I dunno if it would've made my top 10.
Probably a more valid point about using awk, and a common trap I do see people stuck in.
On piping cat, that seems like more an annoyance than anything constructive. Some people use the cat | grep construct because it is so unambiguou
bad ibm no cookie (Score:4, Informative)
Let's say I'm a sysadmin and I'm running as root, trying to remove all the files in the
find
User 1001 has a directory in
Can you guess what happens?
find prints:
xargs sees: ["/tmp/tmp43cc91","/tmp/haxor","","/tmp/haxor","/
Oops!! You just hosed your system!
The correct way to use xargs is to use the -0 switch, which will separate the input by null characters, which cannot appear in filenames. find has a handy -print0 option which will output the correct output:
find
And your system is safe.
Incorrect benchmarks! (Score:3, Interesting)
> ~ $ time grep and tmp/a/longfile.txt | wc -l
> 2811
>
> real 0m0.097s
> user 0m0.006s
> sys 0m0.032s
> ~ $ time grep -c and tmp/a/longfile.txt
> 2811
>
> real 0m0.013s
> user 0m0.006s
> sys 0m0.005s
> ~ $
Re:Don't use shell (Score:4, Insightful)
Re: (Score:2)
Anyone?
Hello...???
Re: (Score:3, Insightful)
No, don't mod up anybody in this thread. Perl and Python are abominations. Pure, unadulterated Bourne shell is for the true, seasoned *nix user. Just like Java is an answer to a question nobody asked in the GUI world, so too is Perl and Python in the command line world.
Re: (Score:2, Interesting)
#!/usr/bin/perl
system("blah");
system("blah");
if(perl code perl code) {
system("blah");
}
etc.
why?
1. because i can't remember the awful syntax of the bash if statement. isn't it something like
if[[""$X$$"" == ""$Y""]];;
2. how about accepting command line arguments in bash? in perl it's just $ARGV[0]. nice and simple and like C++ (except for the offset by one) so i don't want to have to bother le
Re: (Score:2)
Re: (Score:3, Insightful)
2. how about accepting command line arguments in bash? in perl it's just $ARGV[0]. nice and simple and like C++ (except for the offset by one) so i don't want to have to bother learning another one.
Command line args? $1 $2 etc or $* for all of them.
Re: (Score:2)
Re:Don't use shell (Score:5, Insightful)
Or a concise Python script that opens up a text file of URLs, and extracts the files listed in the URLs:
#!/bin/sh
for a in $( cat file | awk '{print "'\''" $0 "'\''"}' ) ; do
wget $a
done
Python has it place, and is far better for medium to large projects, and projects where the code needs to be maintainable. Shell, however, works a lot better for automating UNIX tasks than Python does. Not to mention embedded systems: I can compile Busybox to have both a good shell and all of the commands that one would run from shell scripts (including grep, cut, sed, and, yes, awk) in only about 300k. A Python binary is about a megabyte big, and you need about ten megabytes to fit all of the libraries Python 2.4 comes with.
Re:Don't use shell (Score:5, Funny)
Re:Don't use shell (Score:4, Informative)
This code is not pure shellscript : it uses awk and wget to get the job done...
A Python equivalent might be :
It's not that much longer, it's much easier to read and less error-prone (especially the awk part), and it uses fewer external utilities.
To me, the *only* advantage of shellscript is that it's the only language that you are sure to find on any Unix system.
Re:Don't use shell (Score:4, Insightful)
No shit, Sherlock! You have clearly never worked in a large organisation, where - believe it or not - you, as a standard user, do not actually get to insist that the already-overworked IT department jump through bureaucratic hoops to install your favourite bloated scripting language, unless you have a damn good business case for it. And probably not even then.
Hint: if the task you want that scripting language to accomplish is trivial to achieve with a simple shell script, you don't have a good business case.
* This doesn't apply to wget, obviously, but if your platform really has no standard alternative, you are more likely to persuade IT to install something small and simple like wget, fetch, curl, etc. than a complete programming environment like Python.
Re:Don't use shell (Score:5, Insightful)
Re: (Score:2)
And, for what it's worth, I once had to write an HTTP fetch utility using nothing but SYSVR4 bourne shell, and stuff that shipped stock with a bare-naked install of Solaris 2.5.1.
Why? I needed stuff. NOW. I couldn't wait for next week for an 'admin' to install a web browser, I would have missed my go-live date.
Oh, yes, the box also had an old version of pine on it. I remember, because after HTTP fetching whatever it was, I had to de-base64-it.
I also once wrote an LR(1) parser for arithmeti
Re:Don't use shell (Score:5, Informative)
wget -i filename
Or have I missed something?
Monkeyboi
Re: (Score:3, Funny)
wget options to read urls from a file (Score:2)
Re: (Score:2)
This can be worked around using the undocumented -e flag. What "-e" does is make it so the script stops running if a program called in the script (except in if and a couple of other places) fails (returns a non-zero exit code).
I think it's really bad that this flag hasn't been documented; it's one of those "inside secrets" that UNIX is nortorious for having. And, yes, Python has really great documentation.
From 'man bash':
Seems reasonably well documented to me.
Re: (Score:2)
Try writing a moderately complex pipeline in perl using native popen() calls. Not only do you have to deal with lots of file descriptors explicitly, but you'll have an unmaintainable mess if you have frequently changing requirements, such as adding or removing extra filters at various places.
Always use the right tool for the job. If you want to do complex logic, by a
Re: (Score:2)
When possible, avoid shell scripting: the language is flawed by design. Use a modern programming language, such as Python.
How did you get mod'ed up to 5 I don't know.
Python may not be on the system your managing.
Python, while not as heavy as Java, is an elephant in a china shop compared to shell script
Shell scripting is an essential basic admin skill, good admins don't need Python at all.
Java, Python, Perl -- sheshh... they have their places but not for everything. If any one tool could effectively d
Re: (Score:2)
Re: (Score:2, Informative)
*nix is a highly modular component-based software system with a standard interface (flat byte streams) between components, and a basic set of standard components (given in the POSIX standard) that can be relied upon to always be present.
Re: (Score:2)
http://www.networkworld.com/community/?q=node/976
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:2, Informative)
Re: (Score:3, Informative)
This isn't the first time I've seen this, but it will result in a file MYJPG.JPG being called MYjpg.JPG, ${i//JPG/jpg} would be better as at least the it would end up with the
Again, there's lots of ways to do it. To use the trivial JPG -> jpg, example, yes, you're correct in that using the shortest match at the end would be a better approach (excluding other issues). I just wanted to illustrate the redundant
Re: (Score:3, Informative)
Why is the perl script so hard? The command line would have a bunch of sed going on, whereas the perl script only requires running perl.
I'm guessing... perl -e 'for(`ls`) { chomp; $n = lc $_; system("mv $_ $n"); }'
Re:lowercase uppercase (Score:4, Informative)
Break it down into its constituent parts: Iterate, rename. Whoo! Simple now, eh?
# cd cameraDir
# find . -type f -prune | while read file
> do
> mv "$file" "`echo \"$filename\" | tr '[A-Z]' '[a-z]'`"
> done
#
Re: (Score:2)
Re: (Score:2)
Everyone is talking a different language. No wonder why nobody but nobody uses this operating system.
To biased and pig headed to.
I always stress to new to xNIX types, learn the basics first before getting wrapped up in a perl/java/python or some other cult. Often these other languages are like taking elephants into china shops. And besides, they are not always on a system -- ditto vi versus emacs.