The latest release of Eucalyptus has introduced a new user user console. The user console is written in Python and is using the logging module that can easily be setup to work with rsyslog. It even uses it out the box! Unfortunately, the user console is sending a lot of verbose and useless information into /var/messages which I don’t like. /var/log/messages is the main log for my system so I’d like to make sure that I can find important messages easily. The Eucalyptus user console also uses the Python ConfigParser module so we are able to easily configure the console which includes the logger. Unfortunately for this feature as well, the logging configuration does not seem to take effect but we can fix this easily!
So, in this blog let’s first fix the logging so that it is configurable (I’ll put in a bug for this soon), write the verbose log information to a file other than /var/log/messages and finally setup logrotate to rotate the log on a daily basis.
Before starting make sure that you have installed and configured the Eucalyptus console following the instructions found in the Eucalyptus 3.2 User Console Guide. Make sure that the console is working by starting it up (service eucalyptus-console start) and verifying that you can login (https://my-host-or-ip:8888). This way if the configuration is trashed you’ll know that it was previously working. As always, I suggest you make backups of any files that are edited just in case.
When configuring the Eucalyptus user console in /etc/eucalyptus-console/console.ini you might have noticed the logging section. If you attempt to change the formatting for example in this section and then check /var/log/messages you’ll notice that nothing changed. It turns out that the Python logging module is directly called in /usr/bin/euca-console-server with the default rsyslog settings and the configuration is never honored. The configuration file also uses the StreamHandler in the logging module but there is nothing in the console that uses the stream so the information is not captured.
To first fix these issues we’ll need to stop the Eucalyptus console from using the SysLogHandler with the default settings. Open up /usr/bin/euca-console-server and either comment out or remove the following two lines:
handler = logging.handlers.SysLogHandler(address = '/dev/log') logging.getLogger().addHandler(handler)
With these lines removed, the logger will be started using the configuration in the config file later on in the console’s execution.
Next we’ll configure the logging section in /etc/eucalyptus-console/console.ini to use the SysLogHandler when printing out the logging information. We’ll also make an edit to the formatting string to make it easier to filter out the lines from the Eucalyptus console. Make the logging section of your console.ini file look like the following:
##
# These sections are used to configure the logger. For info, see docs for the python logging package
##
[loggers]
keys=root
[handlers]
keys=eucahandler
[formatters]
keys=eucaformat
[logger_root]
level=INFO
handlers=eucahandler
[handler_eucahandler]
class=handlers.SysLogHandler
level=NOTSET
formatter=eucaformat
args=('/dev/log', handlers.SysLogHandler.LOG_SYSLOG)
[formatter_eucaformat]
format=eucalyptus-console: %(levelname)s %(message)s
datefmt=
##
# End logger config
##
The first two changes occur in the [handler_eucahandler] section. Here the handler is changed from using the StreamHandler to using the SysLogHandler so that logging messages are sent to the rsyslog process. The arguments are changed to include the device to send the logging information to, /dev/logger, as well as which log to send the information to (See Python logging module – SysLogHandler for more information on the SysLogHandler).
The change to the format of the logs makes it easier to filter the log lines produced by the Eucalyptus console without needing to send information to one of the LOG_LOCAL* handlers which could cause issues with other custom log setups (check out Python logging module – SysLogHandler). Since rsyslog automatically adds the time when lines are printed, remove the %(asctime)-15s section of the format line. Adding “eucalyptus-console:” will allow rsyslog to easily filter lines by looking for “eucalyptus-console” in the lines.
Now, create a file in /etc/rsyslog.d/ with any name you’d like as long as it ends in .conf. In the example, /etc/rsyslog.d/eucalyptus-console.conf will be used. Now add the following to it:
# Send the eucalyptus-console output to a separate file :programname,isequal,"eucalyptus-console" /var/log/eucalyptus-console.log &~
This rsyslog configuration file will search the programname column in any log lines sent through rsyslog for “eucalyptus-console” string added above to the format string. If the string is found then it will send that line to /var/log/eucalyptus-console.log. The &~ tells rsyslog to stop evaluating rules for this line so that it only makes it into this file.
Finally, setup the logrotate configuration file for the Eucalyptus console. Create the file /etc/logrotate.d/eucalyptus-console with the following:
/var/log/eucalyptus-console.log
{
rotate 5
daily
compress
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
This file tells logrotate that for the file /var/log/eucalyptus-console.log do the following:
- rotate 5: Remove the logs after 5 days
- daily: Rotate every day
- compress: Compress the rotated log files
- postrotate … endscript: Run the command after rotating the log
With this configuration, the logs will be kept for long enough to be useful and then removed so that /var/log and the system disk does not fill up with unneeded logs.
So now you have the Eucalyptus console sending logging data to its own file and are having that file rotated every day. This is really just the tip of the iceberg though since the Eucalyptus console uses rsyslog we can also send logs to a remove server or search the logs for errors with a tool such as logstash or greylog.
