StopDiskWrites

From wikipost
Revision as of 21:51, 30 January 2013 by Admin (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

The linux operating system by default writes quite a bit of information to log files during a normal system boot and after it's booted even when the machine is just sitting there idling away the O/S will still add entries to log files.

Specifically for systems that run off a filesystem with limited write-cycles, such as flash memory (SDCard, USB drive, etc..) it is important to minimise, if not stop all together, the writing of information to the filesystem that lives on the flash memory.

  • Solution #1 -- write elsewhere

The 'directory' /dev/shm is a 'ramdisk' and ideal for storing information that is okay to delete in between reboots. The size of /dev/shm is roughly half that of the available RAM memory, but in such a way that if traditional applications need more memory it simply is taken away from /dev/shm. This process is dynamically, meaning that when memory is available again, /dev/shm will grow back to its original size if possible.

Services that are ideal to move to /dev/shm are everything in /var/run and /tmp. Depending on what applications you're running there's other candidates too such as apache and samba. More on that later.


  • Solution #2 -- stop logging

An obvious candidate here is cron as it writes to /var/log/syslog every time it runs a cron job. The logging process can be controlled with /etc/rsyslog.conf and we'll tackle the specifics below.


Below is a list of processes that normally write to disk and how to stop that from writing to flash.



Logon/Logoff

writes to: /var/log/wtmp, /var/log/lastlog



Swap file

a swap file is not required and may drastically reduce flash memory when in use.



Cron Jobs

writes to: /var/log/syslog

solution: configure /etc/rsyslog to stop logging cron info


Open /etc/rsyslog.conf, and change:

*.*;auth,authpriv.none                  -/var/log/syslog

To:

*.*;auth,authpriv,cron.none             -/var/log/syslog

and also add:

:msg, contains, "pam_unix(cron:session): session closed for user root"  ~
:msg, contains, "pam_unix(cron:session): session opened for user root"  ~


Restart the syslog daemon to affect the changes:

# /etc/init.d/rsyslog restart
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Starting enhanced syslogd: rsyslogd.


man-db (cron)

The man-db program updates/cleans? the man pages in /var/cache/man. By default it does this on a daily basis and although nothing is written to these directories, there is a slight bug in the man-db package that always writes/touches? the directories. Bug reports Bug#619726 and Bug#620947 at http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg890841.html (/etc/cron.daily/man-db every day touches localization subdirectories in /var/cache/man/) shows more details.

Since keeping the manpages in good order probably isn't that important we offer a simple solution to the above disk writes, namely to insert a little 'exit 0' at the top of the man-db script in /etc/cron.daily/man-db.

#!/bin/sh
#
# man-db cron daily

exit 0     # added to prevent man-db from writing to the filesystem every day.

set -e

iosched_idle=



MARK entries

writes to: /var/log/messages

solution: configure /etc/default/syslogd to stop logging the MARK entries

Check if this is still happening on your system as although I noticed it on earlier versions of Debian, the latest 'Wheezy' does not leave MARK statements every 20 minutes anymore. In any case, if you do have these MARK statements in /var/log/messages coming up you can disable them by editing /etc/default/syslogd and change

SYSLOGD=""

to

SYSLOGD="-m 0"

Restart the syslog daemon to affect the changes:

# /etc/init.d/rsyslog restart
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Starting enhanced syslogd: rsyslogd.



Most applications write to /var/run and /tmp

writes to: /tmp, /var/run and /var/tmp

solution: move all to tmpfs

Create the following entries in /etc/fstab

tmpfs /tmp     tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/log tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/tmp tmpfs defaults,noatime,mode=1777 0 0

Rebooting the sytem is the preferred method to affect these changes.


ntp

Ntp daemon writes offset values to /var/lib/ntp/ntp.drift and generic logging to /var/log/daemon.log

For the drift file we make a symlink to the whole /var/lib/ntp directory and point it to /dev/shm, so that ntp.drift is stored as /dev/shm/ntp.drift

mv /var/lib/ntp/ntp.drift /dev/shm
rm -rf /var/lib/ntp
ln -s /dev/shm /var/lib/ntp

Ntp will also log entries to /var/log/daemon.log. We edit the file /etc/default/ntp and add an option to log to a different file. You guessed it, to /dev/shm!

open /etc/default/ntp and update the line to:

NTPD_OPTS='-g -l /dev/shm/ntpd.log'

We intercept any remaining logging of messages by updating /etc/rsyslog.conf and adding the following lines:

:msg, contains, "logging to file /dev/shm/ntpd.log" ~
:msg, contains, "ntpd 4.2.6p5@1.2349-o" ~

Restart the syslog daemon to affect the changes:

# /etc/init.d/rsyslog restart
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Starting enhanced syslogd: rsyslogd.



hardware clock

Not sure if this is related to ntp, but let's write move this file to /dev/shm too

mv /etc/fake-hwclock.data /dev/shm && ln -s /dev/shm/fake-hwclock.data /etc/fake-hwclock.data



Apache

Apache logs to /var/log/apache2/.

comment out the default logging to access.log by editing /etc/apache2/sites-available/default

I've noticed that /var/log/apache2 can be missing it's best to add the following lines to the /etc/init.d/apache2 startup script so that the files are always moved to /dev/shm without affecting the operation of apache after reboots.

Open /etc/init.d/apache2 and at the end of the script find where the 'CASE' statements begin. Just above that, create two new procedures:


setup_symlinks() {

                # --- begin: custom code to minimise disk writes ---
                # Force creation of /var/log/apache2 in /dev/shm

                # check if symlink already exists
                if ! test -L /var/log/apache2
                then
                  # symlink is not there yet

                  # check if directory exists
                  if test -d /var/log/apache2
                  then

                    # /var/log/apache2 is a normal directory
                    # action: move to /dev/shm and create new symlinks
                    rm -rf /dev/shm/apache2
                    mv /var/log/apache2 /dev/shm
                    ln -s /dev/shm/apache2 /var/log/apache2

                  else

                    # /var/log/apache2 has not been created yet
                    # just create empty framework

                    mkdir /dev/shm/apache2
                    ln -s /dev/shm/apache2 /var/log/apache2

                  fi
                fi
                # --- end: custom code to minimise disk writes ---
}

delete_symlinks() {
 rm -rf /dev/shm/apache2
 rm -rf /var/log/apache2

}

Then add 'setup_symlinks' to the start, reload and restart cases, and the delete_symlinks to the stop case.


Example:

case $1 in
        start)

                log_daemon_msg "Starting web server" "apache2"


To:

case $1 in
        start)
                setup_symlinks
                log_daemon_msg "Starting web server" "apache2"

Samba

Even though samba wasn't doing anything I still noticed that /etc/samba/dhcp was created. Since the file was a 0-Byte placeholder file I simply created a symlink with a slight name change to /dev/shm for it.


mv /etc/samba/dhcp.conf /dev/shm/samba-dhcp.conf && ln -s /dev/shm/samba-dhcp.conf /etc/samba/dhcp.conf



dhcp client

The dchp client writes the search path and domain name as well as a list of DNS servers in /etc/resolv.conf. We move this file to /dev/shm too.

edit /etc/init.d/networking and add the following procedure:

setup_symlinks() {

    # --- begin: custom code to minimise disk writes ---
    # Force creation of /etc/resolv.conf in /dev/shm

    # check if symlink already exists
    if ! test -L /etc/resolv.conf
    then
        # symlink is not there yet

        # check if file exists
        if test -f /etc/resolv.conf
        then

            # /etc/resolv.conf is a normal file
            # action: move to /dev/shm and create new symlinks
            rm -rf /dev/shm/resolv.conf
            mv /etc/resolv.conf /dev/shm
            ln -s /dev/shm/resolv.conf /etc

        else

            # /etc/resolv.conf has not been created yet
            # just create an empty file

            touch /dev/shm/resolv.conf
            ln -s /dev/shm/resolv.conf /etc

        fi
    fi

    # --- end: custom code to minimise disk writes ---
}

Then link to it from the start, reload and restart case statements.


The DHCP client also keeps track of it's dchp lease details in a file under /var/lib/dhcp. One for each interface (e.g. eth0, eth1, wlan0, wlan1, etc..). In most cases you will be using only one or two ethernet interfaces to connect the network. Simply create a symlink to /dev/shm for every interface and all is good. The file /var/lib/dhcp/dhclient.leases (size 0) can be left as is.

mv /var/lib/dhcp/dhclient.wlan0.leases /dev/shm && ln -s /dev/shm/dhclient.wlan0.leases /var/lib/dhcp
mv /var/lib/dhcp/dhclient.eth0.leases /dev/shm && ln -s /dev/shm/dhclient.eth0.leases /var/lib/dhcp


Lastly, DHCP will also log its activities in /var/log/syslog. Simply add a filter in /etc/rsyslog.conf to intercept these messages before they get logged. Open /etc/rsyslog.conf and add the following lines:

:msg, contains, "dhclient:" ~
:msg, contains, "ifplugd(wlan0)[" ~
:msg, contains, "wpa_supplicant" ~



Alsa

The sound system Alsa also creates a file in /var/lib. Again, we create a symlink to /dev/shm for it.


mv /var/lib/alsa/asound.state /dev/shm && ln -s /dev/shm/asound.state /var/lib/alsa


WPA rekeying

Part of the specification of WPA/2 is to periodically change the encryption keys, this is known as rekeying and it logs to /var/log/daemon.log every time it changes keys. We can intercept these messages before they get logged by adding a filter in /etc/rsyslog.conf.

Open /etc/rsyslog.conf and add the following line:

:msg, contains, "WPA: Group rekeying completed"    ~


Other resources

http://www.cyrius.com/debian/nslu2/linux-on-flash.html