Difference between revisions of "BackupPC Notes"

From New IAC Wiki
Jump to navigation Jump to search
 
(29 intermediate revisions by the same user not shown)
Line 4: Line 4:
 
==Web Interface==
 
==Web Interface==
 
The webpage that hold status messages for BackupPC is [http://192.168.40.196/backuppc/ http://192.168.40.196/backuppc/] The address will change as the system goes live.
 
The webpage that hold status messages for BackupPC is [http://192.168.40.196/backuppc/ http://192.168.40.196/backuppc/] The address will change as the system goes live.
 +
 +
==Adding a client for rsync==
 +
===rsync-over-ssh method===
 +
#On the windows box
 +
## Install cygwin
 +
### Make sure to unhide obsolete packages and start from All->uninstall
 +
### admin - cygrunsrv
 +
### base - all
 +
### net - rsync
 +
### net - openssh
 +
## Configure cygwin server: '''ssh-host-config -y'''
 +
### If the config script asks for a password for a cyg_server account, supply one
 +
## Start ssh service: '''cygrunsrv -S sshd'''
 +
## Look for a user to log in as.
 +
### Windows 7 - Admin
 +
### Windows XP - Administrator
 +
### Any version - any other user with admin privileges
 +
## Copy ''id_rsa.pub'' to ''~/.ssh/authorized_keys'' (make sure ~ is the right home dir for the above user)
 +
##Set the firewall
 +
### open port 22 TCP with the scope 134.50.87.0/255.255.255.0,134.50.3.0/255.255.255.0
 +
### In Windows 7, create an exception for the port, then go to properties and change the scope to 134.50.0.0/16
 +
### allow ping
 +
#### (advanced - ICMP in Windows XP)
 +
#### [http://www.sysprobs.com/enable-ping-reply-windows-7 Windows 7] firewall setup for ping
 +
##Disable power saving
 +
# On vienna
 +
## Log in manually once to accept ssh key and test
 +
## Edit /etc/BackupPC/hosts to add new entry
 +
## In /etc/BackupPC/pc copy to a new file matching name in hosts file
 +
## In the new __.pl file, change
 +
### ''$Conf{ClientNameAlias} = __;'' to match the address
 +
### ''$Conf{XferMethod} = 'rsync'; ''
 +
### '' $Conf{RsyncShareName} = ['/cygdrive/c/']; ''
 +
### '' $Conf{RsyncClientCmd} = '$sshPath -q -x -l iacuser $host $rsyncPath $argList+' '' where ''iacuser'' is the user found above
 +
## Change ownership of new file: '''chown backuppc __.pl'''
 +
 +
===rsyncd method===
 +
''(depreciated)''
 +
# On winnebago (or other Linux box):
 +
## make a new password with makepasswd --chars=128
 +
## update rsyncd.secrets file: echo backuppc:DW045wp1vFY4b22ng53dwbi082Ve8CdsM0m5QpDyEuPvAxLtp2Xc3Q16PwSAwc2XWaj2T5eANN1Yfc1uXvMpoAmmWRV6Wo1FXDuwxMjKQhxhBndh96VI0Np9IWioSYJn > rsyncd.secrets
 +
#On vienna
 +
##edit /etc/BackupPC/hosts to add new entry
 +
##in /etc/BackupPC/pc copy to a new file matching name in hosts file
 +
##run ./checkports.sh to find a free rsync redirection port
 +
##change port, ip, and rsyncpassword in new .pl file
 +
##change ownership of new file: chown backuppc __.pl
 +
#On the client computer:
 +
## Install cygwin w/ rsync on the client with the following packages
 +
### admin - cygrunsrv
 +
### base - all
 +
### net - rsync
 +
### net - openssh
 +
### utils - bzip2
 +
## install rsyncd service: cygrunsrv -I rsyncd -e CYGWIN=nontsec -p c:/cygwin/bin/rsync.exe -a "--daemon --no-detach"
 +
## download rsyncd.conf and rsyncd.secrets file
 +
## start rsyncd service: cygrunsrv -S rsyncd
 +
##Set the firewall
 +
### open port 873 TCP with the scope 134.50.87.0/255.255.255.0,134.50.3.0/255.255.255.0
 +
### allow ping (advanced ICMP)
  
 
==Config Files==
 
==Config Files==
Line 22: Line 82:
 
**'''/sys'''
 
**'''/sys'''
 
**'''/mnt'''
 
**'''/mnt'''
 +
**'''/media''' Ubuntu mounts for removable media
 
**Firefox internet cache
 
**Firefox internet cache
 
**Other caches??
 
**Other caches??
Line 27: Line 88:
  
 
==Backup Method==
 
==Backup Method==
Unless testing shows otherwise, the preferred method is rsync-over-ssh. This requires a minimal ssh and rsync install on the clients (probably a stripped-down cygwin). It has the advantage of being compressible and fully encrypted over the wire, and only transferring the changed parts of files. The downside is a higher CPU load on the server and clients for encryption and compression. For Linux hosts we should use a backuppc user and sudo privileges to avoid possible root-level compromises.
+
rsync directly over ssh was rejected because of incompatibilities with rsync daemons launched via the ssh process. Instead the preferred method is rsync connected to a persistent rsyncd through an ssh tunnel. This tunnel can either be set up to the machine directly, or through an intermediary such at the IAC nat firewall box. ssh authentication is taken care of through authorized_keys and rsync authentication uses a password that is randomly generated when the client is first set up for backups.
 +
 
 +
Operating system specific configuration parameters (such as excluded files) are handled in separate files (such as /etc/BackupPC/baseXP.pl) included from host files.
  
 
==Storage Setup on Vienna==
 
==Storage Setup on Vienna==
Vienna has 8x500GB drives set up with an XFS filesystem on LVM over Linux md RAID6. The device is mounted to ''/data'' and has 2.8TB of usable space. The RAID6 configuration allows up to 2 drives to fail without loss of data, and the LVM allows the filesystem to be expanded as new drives are added in the future.
+
Vienna has 4x1TB and 2x2TB drives set up under Solaris10 with a ZFS filesystem. The RAID level for the OS is RAID1 and the RAID level for the datapool is RAID10. Compression is disabled in BackupPC and is handled by ZFS instead.
 +
 
 +
Devices must have at least 1000204885504 bytes. Many 1TB drives do not.
 +
 
 +
==Notes on booting Vienna==
 +
The hostname isn't set correctly on boot so apache won't start. BackupPC also doesn't start by default.
 +
hostname [-s] vienna
 +
svcadm enable '*apache*'
 +
/etc/init.d/backuppc start
 +
And the webserver is restricted to the local interface, so you need to log in and ssh forward the connection:
 +
ssh vienna.iac.isu.edu -L 8080:localhost:80
  
 
==Things we need to think about==
 
==Things we need to think about==
*ssh authentication method (authorized_keys, host-based authentication)
+
*Open files (Outlook) (shadow drives?)
*Open files (Outlook)
 
 
*Thousands of small files can take very long to index (Pulse Recording can accumulate >60k)
 
*Thousands of small files can take very long to index (Pulse Recording can accumulate >60k)
*Finding dhcp hosts reliably
 
 
*checksum seeding
 
*checksum seeding
*[http://parchive.sourceforge.net Parity Archive]
 
*Database dumps from MySQL on webserver
 
 
*User notification policy??
 
*User notification policy??

Latest revision as of 21:27, 12 March 2012

BackupPC Docs

Web Interface

The webpage that hold status messages for BackupPC is http://192.168.40.196/backuppc/ The address will change as the system goes live.

Adding a client for rsync

rsync-over-ssh method

  1. On the windows box
    1. Install cygwin
      1. Make sure to unhide obsolete packages and start from All->uninstall
      2. admin - cygrunsrv
      3. base - all
      4. net - rsync
      5. net - openssh
    2. Configure cygwin server: ssh-host-config -y
      1. If the config script asks for a password for a cyg_server account, supply one
    3. Start ssh service: cygrunsrv -S sshd
    4. Look for a user to log in as.
      1. Windows 7 - Admin
      2. Windows XP - Administrator
      3. Any version - any other user with admin privileges
    5. Copy id_rsa.pub to ~/.ssh/authorized_keys (make sure ~ is the right home dir for the above user)
    6. Set the firewall
      1. open port 22 TCP with the scope 134.50.87.0/255.255.255.0,134.50.3.0/255.255.255.0
      2. In Windows 7, create an exception for the port, then go to properties and change the scope to 134.50.0.0/16
      3. allow ping
        1. (advanced - ICMP in Windows XP)
        2. Windows 7 firewall setup for ping
    7. Disable power saving
  2. On vienna
    1. Log in manually once to accept ssh key and test
    2. Edit /etc/BackupPC/hosts to add new entry
    3. In /etc/BackupPC/pc copy to a new file matching name in hosts file
    4. In the new __.pl file, change
      1. $Conf{ClientNameAlias} = __; to match the address
      2. $Conf{XferMethod} = 'rsync';
      3. $Conf{RsyncShareName} = ['/cygdrive/c/'];
      4. $Conf{RsyncClientCmd} = '$sshPath -q -x -l iacuser $host $rsyncPath $argList+' where iacuser is the user found above
    5. Change ownership of new file: chown backuppc __.pl

rsyncd method

(depreciated)

  1. On winnebago (or other Linux box):
    1. make a new password with makepasswd --chars=128
    2. update rsyncd.secrets file: echo backuppc:DW045wp1vFY4b22ng53dwbi082Ve8CdsM0m5QpDyEuPvAxLtp2Xc3Q16PwSAwc2XWaj2T5eANN1Yfc1uXvMpoAmmWRV6Wo1FXDuwxMjKQhxhBndh96VI0Np9IWioSYJn > rsyncd.secrets
  2. On vienna
    1. edit /etc/BackupPC/hosts to add new entry
    2. in /etc/BackupPC/pc copy to a new file matching name in hosts file
    3. run ./checkports.sh to find a free rsync redirection port
    4. change port, ip, and rsyncpassword in new .pl file
    5. change ownership of new file: chown backuppc __.pl
  3. On the client computer:
    1. Install cygwin w/ rsync on the client with the following packages
      1. admin - cygrunsrv
      2. base - all
      3. net - rsync
      4. net - openssh
      5. utils - bzip2
    2. install rsyncd service: cygrunsrv -I rsyncd -e CYGWIN=nontsec -p c:/cygwin/bin/rsync.exe -a "--daemon --no-detach"
    3. download rsyncd.conf and rsyncd.secrets file
    4. start rsyncd service: cygrunsrv -S rsyncd
    5. Set the firewall
      1. open port 873 TCP with the scope 134.50.87.0/255.255.255.0,134.50.3.0/255.255.255.0
      2. allow ping (advanced ICMP)

Config Files

Most of the configuration for BackupPC is in /etc/backuppc/config.pl Many of the options can be over-ridden with host-specific files, but the config.pl should be edited to give a good default for Windows machines. Most importantly $Conf{BackupFilesExclude} or $Conf{RsyncArgs} need to be configured to exclude the followng items:

  • Windows clients
    • hiberfil.sys holds RAM contents during hibernation
    • Temporary Internet Files IE cache
    • Firefox Internet Cache
    • RECYCLER Recyle Bin
    • pagefile.sys Swap file
    • System Volume Information System Restore files
    • NTUSER.DAT Can cause problems with being in use??
    • TEMP
  • Linux clients
    • /temp
    • /dev
    • /proc
    • /sys
    • /mnt
    • /media Ubuntu mounts for removable media
    • Firefox internet cache
    • Other caches??
    • Parts of /var??

Backup Method

rsync directly over ssh was rejected because of incompatibilities with rsync daemons launched via the ssh process. Instead the preferred method is rsync connected to a persistent rsyncd through an ssh tunnel. This tunnel can either be set up to the machine directly, or through an intermediary such at the IAC nat firewall box. ssh authentication is taken care of through authorized_keys and rsync authentication uses a password that is randomly generated when the client is first set up for backups.

Operating system specific configuration parameters (such as excluded files) are handled in separate files (such as /etc/BackupPC/baseXP.pl) included from host files.

Storage Setup on Vienna

Vienna has 4x1TB and 2x2TB drives set up under Solaris10 with a ZFS filesystem. The RAID level for the OS is RAID1 and the RAID level for the datapool is RAID10. Compression is disabled in BackupPC and is handled by ZFS instead.

Devices must have at least 1000204885504 bytes. Many 1TB drives do not.

Notes on booting Vienna

The hostname isn't set correctly on boot so apache won't start. BackupPC also doesn't start by default.

hostname [-s] vienna
svcadm enable '*apache*'
/etc/init.d/backuppc start

And the webserver is restricted to the local interface, so you need to log in and ssh forward the connection:

ssh vienna.iac.isu.edu -L 8080:localhost:80

Things we need to think about

  • Open files (Outlook) (shadow drives?)
  • Thousands of small files can take very long to index (Pulse Recording can accumulate >60k)
  • checksum seeding
  • User notification policy??