• Increase font size
  • Default font size
  • Decrease font size
Home Projects Managelogs Introducing managelogs

Introducing managelogs

E-mail Print
Read : 215 times
(0 votes, average 0 out of 5)

The primary idea behind managelogs is to provide an efficient protection against ' file system full' failures. Here is the typical scenario a lot of Apache administrators already enjoyed :

  • You install a log rotation system (rotatelogs for instance) to rotate your Apache logs.
  • As you don't want your log data to grow unconstrained, you complement the log rotation mechanism with an externally-scheduled purge operation (shell script, logrotate,...). Note that nothing prevents your log file system to fill up between two consecutive purge operations but we suppose that it is large enough...
  • In a perfect world, both mechanisms would remain perfectly consistent forever. Unfortunately, experience shows that, sooner or later, either the purge job is erroneously inactivated, or a modification introduces a difference in both configuration (such as a change in paths)
  • At this time, the purge operation becomes ineffective
  • And it remains undetected until Apache crashes (usually in the middle of the night) because the file system where it writes the logs has no free space left.
  • At this time, you investigate the problem, detect the error, fix it, and everything starts again, up to the next failure.

managelogs' primary job is to make this scenario impossible. As it includes both the rotation and purge mechanisms, it ensures that the size limits you define will never be exceeded, even temporarily.

But this is not the only reason to use managelogs. Other features include :

Rotation and purge on several criteria : Several parameters define when managelogs will rotate log files or start removing its oldest data. Size-related and time-related parameters are available and can be combined together to define the rotation/purge logic.

Rotation on an external signal : a rotation can be externally triggered with a signal sent to the managelogs process. managelogs maintains a pid file in the directory where logs are written, so that you easily determine which pid to send your signal to.

Run a command upon rotation : each time a rotation occurs, managelogs can run an external command in background. This command is typically used to gather statistics in an external system (such as awstats), but it can do anything, copy, backup, analyze, and even delete the rotated log file. As it runs in background, the command is not limited in CPU time and does not suspend managelogs' execution.

Integrated on-the-fly compression : log data can be written in gzip or bzip2 compressed form. This allows to store a lot more data in the same space and is much more efficient than usual externally-scheduled rotate/purge/compress operations.

Symbolic links on log files (active and/or backup) : managelogs can maintain a set of symbolic links to the log files, allowing external tools to access them through a set of known paths.

Multiple channel output : managelogs can write the log data it receives in parallel to an unlimited number of outputs, each one having its own rotation/purge logic and parameters. A typical use is to maintain two channels : one, uncompressed, with a limited global size limit, for day to day 1st-level operations, and another one, highly compressed, in another directory, with a higher size limit, as an archive.

Create files with non-root owner/group : the managelogs process can optionally switch to a given uid/gid. In this case, the log files it creates are owned by this user/group. You can also set the mode files will be created with. This is interesting, among others, in virtual hosting environments : each virtual host can write its log to a different location with the rights of the application's owner. Then, you delegate the maintenance of this log data to a non-root user/user group. These non-root users not only have total control on 'their' log data but can also control 'their' managelogs process with signals.

Rotations always occur on line boundaries: a buffering mechanism ensures that the log lines written to log files are complete. This is important for log analysis software, which expects well-formed lines and can fail if it is not the case.

Maintains its state across stop/starts/restart operations : unlike most log rotation software, when stopped and restarted, managelogs recovers the state it was in when it was interrupted, and reopens the log file it was currently writing into. This way, stop/start/restart operations don't have any visible effect on the way log files are managed. You can even modify the managelogs parameters (like the rotate and purge sizes). When restarted, managelogs will first apply the new policy and purge the log files if needed. All of this is possible thanks to a status file managelogs maintains in the log directory.

Well, that's all, congratulations to those who are still with us...

Just a last word : I have been using managelogs for several months now on a number of Apache servers and it made my life really easier. I hope it will be the same for you.

For questions or requests, please ask me.


Please login or register to add a comment