Backup and recover Data.fs in linux

This how-to complements the how-to on backup at http://plone.org/documentation/how-to/backup-plone/

I use a script to backup, in my case to another mounted by nfs. The script is run nightly by cron. It will make an incremental backup of items in the DATA and CONFIG lists every day, and an incremental backup using repozo.py every day.

This is a traditional backup approach using a daily job. You could of course setup rsync to sync files to another server, and backup from there. You would still need to backup Data.fs using repozo.py as is done here. Please note this is for the simple case where zeo clients and server are on the same server. Please note that repozo.py expects to see all of its files in its own folder.

Note that if you're using BLOB storage, which has become an standard for Image and Field content-types in Plone 4, you'll have to back-up them separately too. They're stored by default in the var/blobstorage folder of your buildout.

Backed up files are retained for as many days as specified in RETENTION. This includes the files backed up by repozo if they are in the same path as the other files (they are in the default script, just in a subdirectory). Other variables below should be changed depending on your setup and ought to be reasonably obvious to an administrator.

Notes about repozo full/incremental:

• By default, repozo.py does an incremental backup. Use the -F option to force a full backup each time. It will do a full backup if the file has been packed since the least backup. I pack weekly, so I'll get a full backup once a week.
• Be careful about archiving the backup files in the repozo folder. The script output will tell you which files are needed to backup within the current incremental set.
  

This how-to contains the following scripts:

• backup.sh
• restorezodb.sh

backup.sh

#!/bin/bash

# Script to backup data and config files seperately. Does a weekly full and daily incremental

SERVERNAME="backupsvr"
DATA="/opt/Plone-2.5"
DATA_EXCLUDE="*.mp9 /opt/Plone-2.5/zeocluster/server/var/Data.fs" # Data.fs is being backed up on its own. *.mp9 is a dummy wildcard so that it is never empty
CONFIG="/etc /var/www /var/log /home /root" #all these directories will get backed up
CONFIG_EXCLUDE="*.mp9" # Any config files that shouldn't be backed up. tar doesn't like a blank exclude
LIST="/tmp/backlist_$$.txt"
LIST2="/tmp/backups_$$.txt" # a precaution - don't want to mistakenly delete all the backed up files later!!!!
SERVICES="" # #/etc/init.d/zeo# You can list multiple services to stop. For some reason the 'service xxx start' syntax won't work in this script
DESTINATION="/mnt/nfs/${SERVERNAME}"
RETENTION=21 #days of backup files including full and incrementals
FULLBACKUPDAY="Thu" # ddd format. Thursday morning seems like the most 'non' day as there could be downtime depending on items in SERVICES list
MOUNTDEV="backupsvr:/backup"
MOUNTPOINT="/mnt/nfs"
ADMINS="operator"
# This DBBACKUP is the command line used to back up the database. I will always use a full backup every day
DBBACKUP="/opt/Plone-2.5/bin/repozo.py -BvzQ -r ${DESTINATION}/repozo -f /opt/Plone-2.5/zeocluster/server/var/Data.fs"
PYTHONPATH=/opt/Plone-2.5/lib/python
export PYTHONPATH # otherwise repozo.py will probably not be able to find its libraries

#############################################################
# should be no configuration items below here               #
# other than commands for stop/start of services            #
# if not standard init.d-style scripts                      #
#############################################################
function haltservices {
# This is necessary to stop services so that the state is consistent
for doservice in $SERVICES
do
# echo "About to stop $doservice"
 echo `$doservice stop` > /dev/null
done
return
}
function startservices {
for doservice in $SERVICES
do
# echo "About to start $doservice"
 echo `$doservice start` > /dev/null
        if [ $RESULT -ne 0 ] ; then
              logger "Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "Service start returned result code of " $RESULT | mail -s "Message from backup script" $ADMINS
      fi
done
return
}

logger "Starting backup for ${SERVERNAME}"
# Do mount. Quit the backup if it fails.
echo "${DESTINATION}/last.txt file exist" > /dev/null
if [ -f "${DESTINATION}/last.txt" ]
then
    echo "Already Mounted ... that is a bit odd"
    logger "backup: ${MOUNTPOINT} was already mounted ... that is a bit odd"
else
    mount $MOUNTDEV $MOUNTPOINT 
    RESULT=$?
       if [ $RESULT -ne 0 ] ; then
              logger "backup: Aborted Backup for ${SERVERNAME}" $RESULT
          echo "Aborted backup: mount returned result code of " $RESULT | mail -s "Failure in backup" $ADMINS
        exit 1
     fi
fi
#
# We need to check how many incrementals we've had since the last full backup, and we
# will tar everything since the last full backup. So the day after a full, last.txt will have
# a value of 1, so we'll back up the last 1 x 24hrs of files. So incrementals get bigger and bigger
# each day. This is a safe approach that doesn't require you to apply all incrementals since the 
# last full backup. So the day before a full backup, there will be 6 days of data in the incremental.
#
if [ -f "${DESTINATION}/last.txt" ]
then
    echo "${DESTINATION}/last.txt file exist" > /dev/null
else   # There is no last.txt, so we'll create one with a value of 7 (just to be safe)
    echo "Sorry, ${DESTINATION}/last.txt file does not exist" > /dev/null
    touch "${DESTINATION}/last.txt"
    echo "7" > "${DESTINATION}/last.txt" # a safe default value of 7
fi
Cnt=`cat ${DESTINATION}/last.txt`
#
set $(date) # date parms $1 ddd, $2 mmm, $3 dd, $4 hh:mm:ss, $5 NZDT, $6 yyyy 
if test "$1" = $FULLBACKUPDAY ; then
#############################################################
#
# weekly a full backup of all data and config. settings:
#
#############################################################
# First stop services and tar the data files
#
     haltservices        
     tar cfz "${DESTINATION}/data_full_$6-$2-$3.tgz" $DATA --exclude $DATA_EXCLUDE
        RESULT=$?
        if [ $RESULT -ne 0 ] ; then
              logger "Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "tar returned result code of " $RESULT | mail -s "Message from backup script" $ADMINS
      fi
     logger "backup: Finished tarring data files..."
# then run the database backup script
        `${DBBACKUP}`
        RESULT=$?
        if [ $RESULT -ne 0 ] ; then
              logger "Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "DB Backup script returned result code of " $RESULT | mail -s "Message from backup script" $ADMINS
      fi
     logger "backup: Finished backing up database..."
        startservices
#
# Then tar the config files
#
        tar cfz "${DESTINATION}/config_full_$6-$2-$3.tgz" $CONFIG --exclude $CONFIG_EXCLUDE
        RESULT=$?
        if [ $RESULT -ne 0 ] ; then
              logger "backup: Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "tar returned result code of " $RESULT | mail -s "Message from backup script" $ADMINS
      fi
    logger "backup: Finished tarring config files..."
#
# Reset the counter of days since last full backup
#
     echo "1" > "${DESTINATION}/last.txt" 

else
#############################################################
#
# incremental backup:
#
#############################################################
# First stop services and tar the data files
#
        haltservices
        find $DATA -depth -type f \( -ctime -${Cnt} -o -mtime -${Cnt} \) -print > $LIST
        tar cfzT "${DESTINATION}/data_diff_$6-$2-$3.tgz" "$LIST" --exclude $DATA_EXCLUDE
        RESULT=$?
        if [ $RESULT -ne 0 ] ; then
              logger "Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "tar produced this " $RESULT #| mail -s "Message from backup script" $ADMINS
        fi
        logger "backup: Finished tarring data files..."
        `${DBBACKUP}`
        if [ $RESULT -ne 0 ] ; then
              logger "Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "Backup script produced this " $RESULT #| mail -s "Message from backup script" $ADMINS
        fi
        logger "backup: Finished backing up database..."
        startservices     
        rm -f "$LIST"
#
# Then tar the config files
#      
        find $CONFIG -depth -type f  \( -ctime -${Cnt} -o -mtime -${Cnt} \) -print > $LIST
        tar cfzT "${DESTINATION}/config_diff_$6-$2-$3.tgz" "$LIST" --exclude $CONFIG_EXCLUDE
        RESULT=$?
        if [ $RESULT -ne 0 ] ; then
              logger "Potential problem in backup of ${SERVERNAME}" $RESULT
          echo "tar produced this " $RESULT #| mail -s "Message from backup script" $ADMINS
        fi
        logger "backup: Finished tarring config files..."
        rm -f "$LIST"
#
# Increment the counter of days since last full backup and write to last.txt
#
     echo `expr $Cnt + 1` > "${DESTINATION}/last.txt" 
fi


#############################################################
#
# Clean up cycle...
# Will find all backup files older than the retention period, then delete them
# 
#############################################################

find $DESTINATION -type f -ctime +${RETENTION} -print  > $LIST2 # create the list
xargs rm -f < $LIST2 # delete the files
echo "Removing: " `cat $LIST2` # prints the list of files deleted to stdout
rm -f "$LIST2" # delete the temp list

umount -l $MOUNTPOINT # unmount backup device

logger "backup: Backup for ${SERVERNAME} is complete"
#############################################################

exit 0

restorezodb.sh

It's all very well having a backup, but you need to be able to restore too. Have this script tested and ready to speed up that day that you have to recover Data.fs. If you also need to restore configs, products, etc, that requires you to use tar or whatever approach you used to backup in the first place.

Please read the documentation on repozo.py so that you understand what files etc are needed.

You need to manually specify and edit the as-of date-time in the script if you want to roll back to a previous date/time:

#!/bin/bash

# Script to restore ZODB
# 
# 
#Options for -R/--recover:
#   -D str
#   --date=str
#       Recover state as of this date.  Specify UTC (not local) time.
#           yyyy-mm-dd[-hh[-mm[-ss]]]
#       By default, current time is used.
#
#   -o filename
#   --output=filename
#       Write recovered ZODB to given file.  By default, the file is
#       written to stdout.
#
SERVERNAME="vmfedora"
SOURCE="/mnt/nfs/${SERVERNAME}/repozo"
ASOFDATETIME="" # "-D 2007-03-25-02" # "-D yyyy-mm-dd[-hh[-mm[-ss]]]" Specify UTC (not local) time.
DESTINATION="/root/Copy.fs" # This is called Copy.fs for safety, to avoid overwriting the original
MOUNTDEV="vmhost:/backup"
MOUNTPOINT="/mnt/nfs"
DBRESTORE="/opt/Plone-2.5/bin/repozo.py -Rv -r ${SOURCE} ${ASOFDATETIME} -o ${DESTINATION}"
PYTHONPATH=/opt/Plone-2.5/lib/python
export PYTHONPATH # otherwise repozo.py will probably not be able to find its libraries

# Do mount. Quit the backup if it fails.
echo "${DESTINATION}/last.txt file exist" > /dev/null
if [ -f "${DESTINATION}/last.txt" ]
then
    echo "Already Mounted ... that is a bit odd"
    logger "backup: ${MOUNTPOINT} was already mounted ... that is a bit odd"
else
    mount $MOUNTDEV $MOUNTPOINT 
    RESULT=$?
       if [ $RESULT -ne 0 ] ; then
              logger "backup: Problem in backup of ${SERVERNAME}" $RESULT
          echo "mount had this to say..." $RESULT | mail -s "Failure in backup" $ADMINS
        exit 1
     fi
fi
# Restore Data.fs to target directory using repozo.py -R

`${DBRESTORE}`

# unmount source
umount -l $MOUNTPOINT

exit 0