Analyzing Many Log Files?

Popular sites have a problem: log files become very large very quickly! It is necessary to set them aside in some way when they become too large to be kept in uncompressed form or kept on the hard drive at all.

Most sites solve this problem by periodically compressing and setting aside old log files. Wusage 6.0 can analyze compressed log files automatically. If Wusage is being asked to analyze every file in a directory, care must be taken to remove files which contain the same data wusage has already seen, but under a new name, such as a compressed verison of a log file wusage has already analyzed. Otherwise it is possible to get doubled results.

"My server generates a new log file for each day, so I have dozens of log files already and more on the way. How can I analyze these logs with wusage?"

If you have many uncompressed log files, just use the logdir option to tell wusage where to find them all, and keep them all in a single directory until they have been analyzed. You can keep them there indefinitely if you wish, but wusage runs faster if only the data not yet analyzed is present in the log directory. If you wish to keep your log data compressed and not uncompress it for disk space reasons, use the well-known free program gzip to compress your files. (This is not the same as pkzip or zip.)

Wusage can analyze gzip-compressed log files directly without the need to "gunzip" them first.

"I have many mirror sites, so I have a collection of log files that all contain entries from the same period in time. Can wusage cope with this?"

Just use the logdir option to tell wusage where to find the log files you have collected. If you have more than 20 log files that cover the same period in time, for instance 50 log files from 50 mirrors of the same site, wusage may slow down considerably. This does not occur with non-overlapping log files, even if there are hundreds of them. If your needs cannot be met with a limit of 20 overlapping log files, please contact us with information about your specific needs. (If the above link does not work for you, send email to Contact Us instead.)

"This is all well and good, but I already have several old compressed log files that were not compressed with gzip. How do I analyze them with wusage?"

The Unix cat and zcat commands are extremely useful to reconstruct a single log file. Even better, thanks to the -l option of wusage, you can avoid creating an actual combined disk file of your log entirely. Consider the following Unix command line:

cat oldlog1 oldlog2 oldlog3 | wusage -l - -c conffile.conf

The special filename - (a single dash) signals wusage to read its log entries from standard input, which is piped in from the Unix cat program. For more information about cat, try the Unix command man cat.

Important note: it is important to feed log files to cat in ascending order. An older log file should precede a newer one.

"What if I have compressed data?"

The following Unix pipeline will send several compressed log files to wusage:

zcat oldlog1.Z oldlog2.Z | wusage -l -

And the following pipeline will send a combination of compressed and uncompressed log files to wusage:

zcat oldlog1.Z oldlog2.Z | cat - oldlog3 oldlog4 | wusage -l -

Note that both cat and wusage are being fed information from the output of the preceding program, using the special filename -.

Because the log information is arriving at "standard input" as a single stream of data, wusage cannot keep track of its position in each file. Take care not to present the same data more than once to Wusage 6.0 unless you are deliberately regenerating old reports.

"What about non-Unix platforms?"

The MSDOS-related operating systems support the the type command, which can be used to output several files and pipe that output to wusage:

type oldlog1.dat oldlog2.dat oldlog3.dat | wusage -l -

"I ran wusage on an old log file, and now I have zero accesses for the last two months! What happened?"

Wusage normally creates reports through the most recent complete day or week, and will not generate those reports again. You can override this behavior using the -b and -e command line options, which are used to force wusage to start re-generating reports at an earlier date, or to stop well before the present date. If you inadvertently produce empty reports for the most recent several weeks or months, just use the -b option to specify a date from which wusage should start re-generating those reports, and specify the more recent logfile as well using the -l option.

Table of Contents

Boutell.Com, Inc - PO Box 63767, Philadelphia PA, 19147, USA
Phone/Fax +1 206. 658. 8176