Automate web log compression using Analog logfile analyser tool

Write a script which compresses a series of web log files and confirms that basic data has in fact been correctly preserved.

**Background:** We use the [Analog][1] logfile analyser tool to parse our web logs, and its "[cache file][2]" mechanism to compress ancient web logs. As noted in that documentation, this approach archives some, but not all, of the web log data. We need a script which automates the creation of these cache files, and then tests to confirm that the basic data which is supposed to be preserved has in fact been done so correctly.

The intent is that we will (manually) gzip the cache files and remove the original files once this script confirms that all data has been correctly preserved. This script should *not* gzip or remove anything itself!!

All web logs were created by Apache using standard methods.

**Caution:** All syntax used below is from software written six years ago. Some things may (or may not) have changed. Please read the above documentation carefully to ensure that you understand current syntax.

## Deliverables

The script

**[url removed, login to view]** (**m**ake **an**alog **ca**che files) will do the following:

Initialize counts of successful and failed compressions.

If a set of one or more filenames is passed in the script's command line, loop through all of those files whose filename actually begins with "access." and perform the following steps:

Otherwise, loop through all files in the current directory whose filename begins with "access." and perform the following steps:

In this discussion, "$tail" indicates the portion of the filename which follows the initial component, "access.".

Write to the standard output, "Processing access.$tail\n".

Note that some, but not necessarily all, of the access.* files are compressed using gzip. Analog can parse these directly, and does *not* require you to gunzip them! e.g., A typical set of files might include:

# ls [url removed, login to view]{0,1}*

[url removed, login to view] [url removed, login to view] [url removed, login to view]

[url removed, login to view] [url removed, login to view] [url removed, login to view]

[url removed, login to view] [url removed, login to view] [url removed, login to view]

[url removed, login to view] [url removed, login to view] [url removed, login to view]

[url removed, login to view] [url removed, login to view] [url removed, login to view]

[url removed, login to view] [url removed, login to view] [url removed, login to view]

If anca.$tail already exists, write, "anca.$tail already exists\n". Do not overwrite it, but do continue processing after the creation step below.

Otherwise, create anca.$tail using a command similar to:

analog +C"LOGFILE access.$tail" +C"OUTPUT CACHE" +C"CACHEOUTFILE anca.$tail"

and write, "Created anca.$tail\n".

The following steps currently use a configuration file named [url removed, login to view] with the following content:







If you can do the following *just* using command line parameters, without using a configuration file at all, that'd be far easier, as it'd keep the script "self-contained".

Generate two reports from analog, one from each file, saving the output to temporary files, using commands similar to:

analog +g/web/logs/[url removed, login to view] +C"LOGFILE access.$tail" >>/tmp/manca.before.$$

analog +g/web/logs/[url removed, login to view] +C"LOGFILE NONE" +C"CACHEFILE anca.$tail" >>/tmp/manca.after.$$

This will yield reports which contain lines similar to this:

month: #reqs: pages: kbytes:

--------: -----: -----: ------:

Oct 2005: 14453: 1265: 29981: ++++++++++

Nov 2005: 41690: 3600: 92838: +++++++++++++++++++++++++++++++

If all of the numbers (i.e., three numbers per month) match for the two reports, write, "Compression confirmed.\n". Otherwise, write, "Compression FAILED!\n". Increment the appropriate count.

After the loop is complete, write, "### files processed: ### confirmed, ### failed.\n".

| ![][3] | PGP is **not** required for this project. |

| ![][4] | An NDA is **not** required for this project. |

We will retain full legal rights to the code and/or data upon completion. All code and/or data is and becomes *our* intellectual property! You will *not* have any rights to the code and/or data in any fashion.

**Thank you for your consideration and for bidding through RentACoder!**

* * *Copyright ? 2005, RentACoder Buyer

[notecom][5]. **All Rights Reserved.** Reproduction without permission is strictly prohibited.

* * *

1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done.

2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables):

a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment.

b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request.

3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement).

## Platform

Linux (required) and Solaris (optional)

Квалификация: Техника, Linux, MySQL, Perl, PHP, Архитектура ПО, Тестирование ПО, Solaris, Веб-хостинг, Управление сайтами, Тестирование веб-сайтов

Показать больше web creation software, typical nda, temporary to hire, temporary hire, standard.pl, prohibited fashion, original rentacoder, nda its, intent to hire form, intent to hire, web log compression, apache web server software, write some software | perl, write some software perl, solaris, PGP, nda legal, manca, log files, linux solaris

О работодателе:
( 137 отзыв(-а, -ов) ) Wyoming, United States

ID проекта: #3097296