Free up disk space in Citrix XenServer

A Citrix XenServer dom0 currently runs with a 4GB root partition which is pretty small by today’s standards. A small amount of usable storage space can be quite easy to quickly fill. It is therefore important that dom0 has free space in order for it to operate correctly. Here we will cover some different methods that can be used to free up disk space within XenServer.

Edit the log rotation scripts

As you probably know, log files are notorious for consuming precious disk space in a system, and XenServer is no different. While you can manually delete old logs from /var/log there are better ways to go about this.

In my adventures through the XenServer cron files, I’ve found something useful that doesn’t appear to be well documented which unfortunately is also the best way for quickly reducing disk space usage by log files in /var/log in my experience. As it turns out there are scripts already in place to perform the majority of required log rotation for us, we can use these to our advantage by modifying them as required.

Our journey begins in the /etc/cron.d/logrotate.cron file which contains the below cron entry.

*  * * * * root /opt/xensource/bin/logrotate-xenserver 

Essentially this is configured to run the /opt/xensource/bin/logrotate-xenserver script every single minute. Within this script two additional scripts are called as below.

# First delete as many old compressed logs as necessary.
/opt/xensource/bin/delete_old_logs_by_space

# Then rotate the live logs if needed, i.e. the uncompressed files to which
# processes are logging data.
/opt/xensource/bin/rotate_logs_by_size

First lets take a look into /opt/xensource/bin/delete_old_logs_by_space, right at the top we notice the below information.

# Starting with the oldest, delete enough compressed log files (if they exist)
# until /var/log occupies less than
BUDGET_MB=100 # MiBytes

Great, so we can just modify BUDGET_MB to something lower if needed, for instance 50MB. After editing the file and saving it, you can either run the bash script manually or just wait for the cron to take care of it.

It’s a similar story within the /opt/xensource/bin/rotate_logs_by_size file, as shown below.

# Allow live logs to grow this big in total...
LIVE_TOTAL_MB_TRIGGER=36 # MiBytes

# ...then use logrotate on the biggest ones to end up under this size:
LIVE_TOTAL_MB_TARGET=12 # MiBytes

These values can be modified in order to maintain a smaller log footprint on the disk and can be adjusted as you see fit.

Enable log compression

This one is quick and easy to implement and also has some official documentation from Citrix. Essentially you can enable log compression to save disk space by uncommenting ‘compress’ from the /etc/logrotate.conf file.

Modify Sysstat

Sysstat logs are found in /var/log/sa and do not typically take up very much disk space by default. In my experience this directory tends to range from 20MB-50MB in total size. While not a large amount, when your root file system is only 4GB in size there are times when every additional bit adds up considerably.

I don’t recommend flat out deleting the files in here as they provide useful performance information which can be useful during troubleshooting. Instead you should consider how many days worth of information you want to keep. By default sysstat will store 7 days worth of data down to 10 minute increments.

You can change how many days worth of history are stored in /var/log/sa by modifying the /etc/sysconfig/sysstat file which is shown below.

# How long to keep log files (days), maximum is a month
HISTORY=7

You could also change how frequently information is logged, by default /etc/cron.d/sysstat is set to perform this every 10 minutes so you could change it to 30 minutes instead for example.

# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib/sa/sa1 1 1

Personally I’ve never needed to refer to the information stored within these files as an external monitoring solution is preferred, so disabling them entirely may also be an option if you’re desperate for the extra space. However in most cases the few MB of size required for the files is probably worth giving up for additional performance information, how much you want to modify the disk space used to useful information logged trade off is up to you and your particular environment.

Delete temporary files

As the name implies, files stored within /tmp are intended to be temporary so this is a directory worth looking at when on the hunt for disk space. In XenServer 6.5 and newer this does not seem to be as much of a problem, however in XenServer 6.2 I’ve found plenty of small xenguest log files being written to /tmp and although they are small they can quickly build up and before long hundreds are present. With that said I’ve found single files in excess of 100MB that have been fine to delete, so it’s worth a check during the clean up process.

You can run the below command to display the total size of /tmp

[root@XenServer ~]# du -h --max-depth=1 /tmp
142M    /tmp

To view the largest files you can run ‘ls -lrSh’ as shown below, where -S sorts by file size, -r reverses the ordering showing the largest files at the bottom of the output, and -h will display the size in a human readable format.

[root@XenServer ~]# ls -lrSh /tmp/

Delete old patch files

Over time various hotfix and update patch files will be uploaded during maintenance activities, these older files can be deleted. Such files are stored in the /var/patch directory, however don’t go through deleting anything just yet! Firstly some important information regarding /var/patch, do not remove anything within /var/patch/applied. Contents in /var/patch (excluding the applied directory) may be safely deleted after XenServer has been rebooted and any pending updates have been fully applied.

You should not delete contents from /var/patch if a XenServer host in the pool is pending a reboot for a hotfix for example. Once all XenServer hosts have been rebooted for updates that have been applied you can then safely remove from /var/patch. Unfortunately this is currently a manual process so you will have to perform it on each XenServer host in your pool.

It is not uncommon to find files in excess of 100MB in here, so purging old files from /var/patch is a good way to reclaim disk space in XenServer.

Conclusion

The low amount of disk space that the XenServer dom0 has to work with (only 4GB) needs to be monitored and managed in order to prevent problems caused by it filling up. How you monitor the disk space is up to you, external monitoring tools such as Nagios for example can fulfil this task. Disk space can be freed up by purging old log files, enabling log compression, modifying sysstat, deleting temporary files, and deleting old patch files. Are there any other methods you use to free up disk space in XenServer? Let me know in the comments below and I can get the post updated.

  1. Oh.. and one thing I started doing as well- I put a job into the logrotate-xenserver job to purge log files that haven’t been modified in 7 days:
    find /tmp/*.log -mtime +7 -exec rm {} \;

    Hope that’s helpful as well- average of 200 MB difference on the systems I’ve been seeing thus far.

  2. Thanks for the information it has been helpfully, by the way, we’re getting a Control Domain Full message and we’re sitll without get a solution to release the /dev/sda partition or locate the file that is using the available space.

    Exist a way to reach up to you or be contact?

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>