EFF's Logfinder 169
clonebarkins writes "EFF has just released a new software tool called "logfinder" to help server admins find (and delete) unnecessary log files on their boxen. "By finding unwanted log files, logfinder informs system administrators when their servers are collecting personal data and gives them the opportunity to turn logging off if it isn't gathering information necessary for administering the system.""
I just made one, too (Score:4, Funny)
Re:I just made one, too (Score:2)
Re:I just made one, too (Score:2)
Re:I just made one, too (Score:2)
Re:I just made one, too (Score:2)
Thanks EFF! (Score:2)
Re:Thanks EFF! (Score:2, Insightful)
Re:Thanks EFF! (Score:2)
don't give them root/admin access - i.e. don't give them permission to delete the logs.
Re:Thanks EFF! (Score:5, Insightful)
InnerWeb
Re:Thanks EFF! (Score:3, Funny)
Yeah, sure. You guys are so paranoid, next you're going to be telling me that the flesh-reanimation technology I've been working on can be used for evil too.
Re:Thanks EFF! (Score:2, Funny)
Yeah, sure. You guys are so paranoid, next you're going to be telling me that the flesh-reanimation technology I've been working on can be used for evil too.
Sure can. My wife just used your invention to bring back to life her dead mother. That is nothing but pure evil, believe you me.
Re:Thanks EFF! (Score:1)
Well.. (Score:2)
Re:Thanks EFF! (Score:1)
neat (Score:1)
Re:neat (Score:5, Funny)
Is a new tool really necessary? (Score:3, Insightful)
LK
Re:Is a new tool really necessary? (Score:2)
Can't subpeona what doesn't exist? (Score:5, Insightful)
Re:Can't subpeona what doesn't exist? (Score:3, Interesting)
Re:Can't subpeona what doesn't exist? (Score:5, Insightful)
If an admin is just using this tool to destroy potentially incriminating logs, then they are using it poorly. Like trying to pound a screw in with a hammer.
The use this has for an admin is to survey (or for the less experienced admin, to discover) what logs the system is currently, so that the admin can decide as a policy which logs should be active or not, and with what level of detail. The itch this tool scratches is that many systems as a default keep more logs than perhaps are necessary. A good admin will shut off whatever is deemed unnecessary, based on multiple criteria (security, system load, user/company privacy).
Forbidding the use of log destruction tools (rm?) is moot. Destroying evidence is illegal. Now, laws (or court orders) mandating a level of logging are a completely different matter.
Re:Can't subpeona what doesn't exist? (Score:4, Informative)
I do exactly that with logs for my company. Once a month I clean out everything we don't need, including "email logs" and other stupid shit MS piles up in various places in the operating system. If/When the lawyers/cops come knocking, I can point to the policy and scheduled reminder and say "sorry, dont have that".
Logs are not the only place stuff resides and piles up, but it's one easy fix and keeps my servers and machines clear of unnecessary disk-space robbing files.
Re:Can't subpeona what doesn't exist? (Score:2, Funny)
> logs" and other stupid shit MS piles up in various places in the operating
> system. If/When the lawyers/cops come knocking, I can point to the policy
> and scheduled reminder and say "sorry, dont have that".
"...but if you'd have come yesterday you could have had 30 days worth".
I think I prefer the policy apparantly in place at www.cryptome.org, which is to delete all your logs every 24 hours.
Re:Can't subpeona what doesn't exist? (Score:1)
Great, get busted for rotating my logs......
Re:Can't subpeona what doesn't exist? (Score:1)
Re:Can't subpeona what doesn't exist? (Score:2)
If your admin is competant enough (Score:2)
Re:Only if you don't do backups. (Score:3, Interesting)
Also, for security, the random keys should then be passed through a public key encryption prior to being written to the
Re:Only if you don't do backups. (Score:2)
What's with database backups, genius?
I appreciate the effort but... (Score:5, Insightful)
a) the sysadmins are competent enough to handle this themselves. I would think that a sysadmin would know how to use some sort of local file search.
b) the EFF understands that it's not always up to the sysadmins to determine the amount of time to keep logs that might be used against an individual.
Re:I appreciate the effort but... (Score:4, Informative)
Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over
In short, competant *nix admins will know most of the many location where their important daemons are storing logfiles. NT admins on the other hand, many not even know what daemons are running on the machine anymore, let alone where they store their log files!
P.S.
Hey wait! This is a python app. I guess NT admins will just have to keep on googling.
Re:I appreciate the effort but... (Score:3, Interesting)
Re:I appreciate the effort but... (Score:2)
wtf? What distro are you running?
Re:I appreciate the effort but... (Score:2)
Re:I appreciate the effort but... (Score:5, Informative)
I don't think you understand *nix logging, or you've been working with poorly-designed systems.
Locations for log files has been pretty well standardized by Posix and the LSB. Logs generally go in
Logfiles which end up in
Now, compare this to a Windows 2003 Server running Exchange 2003, where the log files in c:\windows c:\Windows\system c:\Windows\system\Logfiles c:\Windows\system\security
C:\Program Files\Exchsrvr\ C:\Program Files\Exchsrvr\MDBDATA C:\Program Files\Exchsrvr\mtdata . Many of the logfiles are not viewable with a text viewer. Some of the log files really aren't "Log files", but are "Transaction Logs", which is a different thing in my book.
Some of this makes sense, some of this does not. But I'm not a windows admin, and I didn't design this network here, so maybe this is the result of a poor configuration.
Re:I appreciate the effort but... (Score:2)
Apache:
Syslog:
Apache-ssl:
Samba:
LPD:
FTP: User-defined in config
Email programs:
Not really sure what system the grandparent was using, but all my logs have generally been easy to find too. The only time logs go somewhere else is if *I* want them to, usually in the event that somebody else is hosted on my machine and I want them to have access to their own logs but
Re:I appreciate the effort but... (Score:2)
The
Re:I appreciate the effort but... (Score:3, Insightful)
I didn't bother to read a description of the tool, but there's nothing to say that a competent admin might not want something like this if it eased his burden. Also, there's the matter of incompetent admins. Many of us wear multiple hats. I do development, support, and administration on linux and windows for a small office, mostly by myself. Suffice to say, nobody can be perfect at everything. I'm always looking for tools that help me auto
Re:I appreciate the effort but... (Score:2)
Yeah, well I don't think that Unix distributions should include the find command!
I would seriously hope that the sysadmins are competent enough to do a recursive ls and pipe that into grep when they want to locate a file with a particular name!
(Seriously, what's wrong with providing toolsets to administrators that would like to use them?)
Serious Administrators (Score:2, Funny)
Oh, yeah (Score:5, Funny)
Re:Oh, yeah (Score:2)
Poor, naive admin. You have much to learn. Fuser is your friend.
Re:Oh, yeah (Score:4, Insightful)
In an ideal world every system would be administered by a well trained and experienced system admin, or a trainee admin being mentored by one, who had plenty of time to investigate and maintain the machine. In practice most system admins are people in other roles (developers, DBAs, desktop support or even receptionists) who have been handed the task of managing half a dozen white box Wintel servers (with maybe a SCO or Linux box or even an aging Sun box in the mix) and probably a Netware server doing file and print, most were built and installed by someone one of the manager's knows or have been inherited third hand from another company. If they're lucky they get a training course where they'll learn a few of the GUI screens, more likely they'll be given a few dozen pages of handwritten notes (aka 'the manual') and told to go to the nearest Waterstones/Borders/Whatever and buy a book if they need more.
That was pretty much my first job. I had trained as a C programmer; then I found myself managing 70 desktops running various versions of Windows, a dozen or so White Box Intel based servers running Windows NT 3.51 and 4.0, a SCO OpenServer box, an Alpha running VMS, a 3 member VAX cluster running VMS and an RS6000 running AIX. All with no usable documentation or training. A little later they added in DBAing the Oracle databases and managing the network (a variety of devices from 3Com, Cisco and Bay), at the time I only knew a bit of SQL and wasn't really sure of the difference between a router and a switch. After spending a lot of money on books then a lot of time reading them (I didn't have web access at the time, when I did I started reading websites as well) I eventually learned what I needed to know.
This script is a separate issue. Inpractice I don't expect those sorts of admins to run it, they probably wouldn't know what to do with the information if they did. Where I think it would be useful is for the professional admin who suddenly inherits a bunch of machines (maybe they've moved companies or their company has merged with another). Put this script on them and run it for a few days then see what it turns up. No matter how wonderful and professional you are unless you built and installed a machine yourself and can guarantee that no-one else has ever had the root/admin password to a box you can't be 100% sure that there's not some process running somewhere that is quietly logging something somewhere. No-one who manages a non-trivial number of machines has time to check every machine to make sure that there are no new or unexpected services that have snuck in (and remember it's not something you could do once and then not again, you'd have to keep on doing it). That's why you need scripts that look for anything that could point to unexpected activity. Not just looking for anything that looks like a log on a box but also ports that shouldn't be open (I've lost count of the number of times I've found a box with port 25 open when I know I've disabled SMTP, only to find that someone has re-enabled it without telling me) or unexpected activity on a switch or firewall port. Not only do we have too many machines to manage but also users who delete files they shouldn't which then must be restored from backup, managers who constantly demand reports on system availabity stats and projects that we have to keep an eye on to make sure they don't run wild and break every standard we have.
Stephen
"Boxen" (Score:5, Funny)
Re:"Boxen" (Score:2)
Re:"Boxen" (Score:1)
Plurals... (Score:3)
"Boxen" is fine. If the plural of ox is oxen, then pluralizing box as boxen seems perfectly acceptable to me. It also helps to understand that somebody is talking about a bunch of computers as opposed to a bunch of cardboard boxes.
But I swear that the next person who tells me (in person) that virii is not correct is getting a punch in the
Re:Plurals... (Score:2)
The plural of "virus" is "Virusen".
Re:Plurals... (Score:2)
As opposed to a suave ladies-man like yourself. Nothing's cooler than trolling slashdot.
Re:"Boxen" (Score:2)
What is the plural of virus? [google.com]
In case you actually know any Latin, there is some dispute over whether virus is a second-declension noun, like amicus, "friend," plural amici, or a fourth-declension, like status, plural statuses. Personally, I believe it was fourth-declension, but it was rarely used in Latin at all, anyway.
WTF is Boxen? (Score:2, Interesting)
Re:WTF is Boxen? (Score:2)
Mysteriously, admins don't seem use "mouses", or "hice", or "meese" (plural of moose)
(Of course, then there's the ones who will vehemently argue that "its" is not a word because special cases are bad and "it's" is the proper possessive)
Re:WTF is Boxen? (Score:2)
Re:WTF is Boxen? (Score:2)
Well, the German plural of ox (die Ochse) is "Ochsen," and in German, you do have multiple "Boxen."
If the person who wrote the submission was German, I'm willing to forgive them. However, as "Carl" claims to have been born in Philadelphia (presumably the newer one in the US, and not the older one in the Middle East), he needs beating with at least two wet fish. Maybe it was the fault of the apes, although the spell in Elbonia probably didn't help.
-- Steve
Re:WTF is Boxen? (Score:2)
Re:WTF is Boxen? (Score:2, Interesting)
It's a running joke [catb.org]. See also this [catb.org].
Re:WTF is Boxen? (Score:2, Insightful)
Re:WTF is Boxen? (Score:1)
Unfortuniatily, the term has found its way onto Dictionary.com [reference.com].
Re:WTF is Boxen? (Score:3, Informative)
And yes, you are seriously behind the the times. The oldest copy of the Jargon File I have is from the early '90s and that contains the word boxen.
It can be quite useful, since boxen are always computers, while boxes can be the
All your boxen belong to ux (Score:2)
Well, maybe he's so behind the actually he's ahead. I don't know anyone but lamer kiddies and old farts trying desperately to be hip that really use the phrase anymore. It's almost as if it's become a marketing buzz word, and if you know anyone who uses it frequently to refer to computers is probably reaching out for acceptance or hoping that you'll think they'r
Re:WTF is Boxen? (Score:1)
Here's the transcript...
Plurals were hard, too.
"Brian, how do you make a word a plural?"
"You put a 's'...put a 's' at the end of it."
"When?"
"On weekends and holidays."
"No, Brian. Let me show you." So she asked this kid who knew everything. Irwin. "Irwin, what's the plural for ox?"
"Ox. Oxen. The farmer used his oxen."
"Brian?"
"What?"
"Brian, what's the plural for box?"
"Boxen. I bought 2 boxen of doughnuts."
"No, Brian, no. Let's try another one. Irwin
EFF would sell more copies if... (Score:1, Funny)
is this stupid? (Score:4, Insightful)
lots are crucial for many reasons:
1. Hacking attacks (how else do you track them, and prevent them)?
2. Abuse problems (spammers, credit card fraud)
3. aggregate statistics (what percentage of my customers are based in Europe?)
I can't see why someone would shoot themselves in the foot and use this.
Like log files are really intrusive anyway.
Re:is this stupid? (Score:3, Insightful)
Note, this is why large companies have email retention policies -- because having to do discovery or comply with a subpeona on email records going back years is expensive. So doing this type of thing isn't anything new or sinister.
Re:is this stupid? (Score:2)
I know that webalizer keeps these statistics itself. You don't need to keep the rotated logs for that long. I would assume that you would use utlities that keep that extract that data and use it away from the actual logs themselves.
Like log files are really intrusive anyway.
Depends on what's being logged I guess.
Re:is this stupid? (Score:3, Insightful)
Redhat (Score:2, Informative)
The last time I checked out redhat (about version 8 I rekon) they inluded this nice little utility called "logviewer". And, I though, wow a text viewer how novel, Linux doesnt have many text viewers.
So not only is this a text viewer, but it also finds all those logs hidden in /var/log/*, it must be hard to find anything in /var/log/* ...
Just as an example... (Score:4, Insightful)
Re:Just as an example... (Score:1)
Interesting Motive (Score:4, Interesting)
I was suprised to see the EFF seems to have a totally different motivation. It seems their real motivation is that the government can't demand logs that don't exists, or more specifically you can't get in trouble for not providing what you don't actually have.
Not sure what I think of that...
Re:Interesting Motive (Score:2)
The much more common case is a civil suit where logs are requested in discovery. Woefully, failing to produce logs for a particular period can weigh heavily against your side of the case in a civil matter. I know of several companies that keep email forever for example, only because defending themselves in a suit might rely on being able to demonstrate that actions were taken at a particular time. Saying "yeah, we did that then, but w
Re:Interesting Motive (Score:2)
Re:Interesting Motive (Score:2)
Re:Interesting Motive (Score:2)
Re:Interesting Motive (Score:2)
But how will that help defeat the Evil Terrorists ?
Actually, we love the evil terrorists, so we'd never want to defeat them. We must - after all, we're making so many of them.
Re:Interesting Motive (Score:2)
Same thing libraries did to get around having to turn over patron's reading habits to the police.
Of course the next step the government will take is passing a law requiring everyone to keep their logs for 5 years.
interesting... (Score:5, Informative)
So, no, its not just "locate log" that somone suggested, nor is it "find
As for the comment about competent site-admin. This is a bit more than that too, its also about users and active software, peoples IRC logs, various ftp clients that clobber up and log passwords along with everything else in their config dir. And so on and so forth.
Re:interesting... (Score:3, Informative)
"... We have created a program called logfinder as a sample means of locating files that might be logs on an existing system. logfinder uses regular expressions to find local files with "log-like" contents; you can customize those expressions if necessary to meet your needs. logfinder requires Python 2 or greater and finds logs in text files on a POSIX-like system. (It might also find some log-like data in binary files if the binary files represent that data in textual form.)
log
We have that already. (Score:2, Funny)
Re:We have that already. (Score:1)
Good work, but... (Score:2, Interesting)
Two months later, "they" subpoena your logs to find no trace of evidence. Suspecting log-alteration, th
Re:Good work, but... (Score:2)
I don't think that makes sense. (Score:2)
How would it be "evidence tampering" if you didn't even know about the existence of an investigation until 2 months after you edited the logs? For you to be tampering with "evidence",
Re:Good work, but... (Score:3, Insightful)
There is a difference between evidence tampering (illegal) and system administration (legal). If you remove data because it may be incriminating, you are tampering with evidence. It would also be illegal to delete data after you receive a subpo
Didn't the MPAA do something similar? (Score:2)
Log Retention Policy (Score:2, Insightful)
My experience with logfinder (Score:3, Informative)
Re:My experience with logfinder (Score:1)
Could be moderately useful (Score:2, Informative)
This tool could be moderately useful, especially in an environment where the administrator can't be expected to know all of the ins and outs of third-party add-ons.
I was once assigned to a dotcom that used a third-party component to allow for credit card transactions. What the admin didn't realize was the default configuration left the component in debug mode, placing all user-submitted credit card data in plain text files on the web server
We only found the log file accidentally while performing an unr
anybody else have this lock up their system? (Score:3, Interesting)
This system is rock solid, in use for hours/day with the exact same mix of programs running constantly (evolution, mozilla, ssh/rxvt windows to external systems, etc.)
comments?
No offence to all the MOCKARY here.... (Score:1)
Just what I needed.... (Score:2)
Re:Just what I needed.... (Score:2, Funny)
Server search? (Score:2)
Re:Excellent (Score:2)
Umm, that's where logrotate [debian.org] is for. Yeah I know, there are some stupid vendors who don't support this out of the box, but it's not difficult to make it work with a random application.
As a sidenote, this should be modded 'clueless', not 'informative'.
Re:Excellent (Score:3, Insightful)
1. It took me around 3 minutes to find out that this thread applies to POSIX-like systems only (ie. won't work on this winXP). The fact should really have been mentionned in the summary. I only say this because recently, some summaries seem to have been "hastily" written.
2. I am myself wary of huge, hidden log files that either winXP itself or other programs create. As the only user and sysadmin on this system and keen to mini
Re:Excellent (Score:2)
Re:Excellent (Score:2)
Re:Excellent (Score:1)
Don't want to accuse you of having your own agenda (*cough* Mr. lin ux.com *cough*), but reinstalling every week would seem to be a little extreme. XP may have vulnerabilities, and may not be as stable as Linux servers in the long run, but a properly configured XP box is relatively stable. Add in the requisite free software (firewall, malware catchers, et al: www.pcw orld.com/reviews/article/0,aid,116456,00.asp), and even if it's not as secure as Linux, it should still not need to be re-g
Re:boxen (Score:2)