Difference between revisions of "Installing A Cluster"

From New IAC Wiki
Jump to navigation Jump to search
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
==Network configuration==
 
==Network configuration==
−
Head nodes works as NAT for slave nodes
+
Head nodes works as NAT router for slave nodes
  
 
eth0 connects to outside world
 
eth0 connects to outside world
Line 9: Line 9:
 
==OS installation==
 
==OS installation==
 
Normal server Linux install with the following packages:
 
Normal server Linux install with the following packages:
−
* ssh server
+
* openssh-server
 
* tftpd-hpa
 
* tftpd-hpa
 
* dhcp3-server
 
* dhcp3-server
 
* nfs-kernel-server
 
* nfs-kernel-server
 
* debootstrap
 
* debootstrap
−
 
 
* libpmi
 
* libpmi
 
* mpich2
 
* mpich2
 
* slurm-llnl
 
* slurm-llnl
−
* syslinux?
+
* slurm-llnl-slurmdbd
−
 
+
* syslinux
−
Grab pxelinux from the web
 
  
 
===Setting up a chroot for the node root===
 
===Setting up a chroot for the node root===
−
The chroot is a directory on the head nodes that hosts the root filesystem that the slave nodes will use.
+
* Boot to a live CD and mount the root drive as /new
−
A chroot file system is installed with '''debootstrap''' and can be chrooted into so it acts like a standard install.
+
* Copy the root fs to a chroot '''rsync -av /new/ /new/nodes/precise/ --exclude /nodes'''
−
mkdir -p /nodes/lucid
+
* Enter the chroot '''chroot /nodes/precise/'''
−
debootstrap lucid /nodes/lucid/ http://backup.iac.isu.edu:9999/ubuntu/
+
* Change the following lines in ''/etc/initramfs-tools/initramfs.conf''
−
cp /etc/apt/sources.list /nodes/lucid/etc/apt/
+
<pre>
−
cp /etc/resolv.conf /nodes/lucid/resolv.conf
+
  MODULES=netboot
−
chroot /nodes/lucid/
+
  BOOT=nfs
−
  apt-get update
+
</pre>
−
  locale-gen en_US.UTF-8
+
The DEVICE= and NFSROOT= lines may also be of use if the tftpboot configuration isn't working
−
dpkg-reconfigure tzdata
+
* Update the initramfs '''update-initramfs -c -k all'''
−
 
 
−
More information about [https://wiki.ubuntu.com/DebootstrapChroot chroots], especially helpful locale setup.
 
−
 
 
−
In the chroot we want to install the following packages:
 
−
* slurm-llnl
 
−
* munge
 
−
* nfs-common
 
−
* openssh-client
 
−
* openssh-server
 
  
 
===NFS===
 
===NFS===
Line 52: Line 41:
 
  /etc/default/nfs-common  
 
  /etc/default/nfs-common  
 
  /etc/default/nfs-kernel-server
 
  /etc/default/nfs-kernel-server
 +
 +
Reload the NFS server settings
 +
/etc/init.d/nfs-kernel-server force-reload
  
 
===Netbooting===
 
===Netbooting===
Line 58: Line 50:
 
Edit /etc/dhcp3/dhcpd.conf as follows:
 
Edit /etc/dhcp3/dhcpd.conf as follows:
 
<pre>
 
<pre>
−
# The ddns-updates-style parameter controls whether or not the server will
 
−
# attempt to do a DNS update when a lease is confirmed. We default to the
 
−
# behavior of the version 2 packages ('none', since DHCP v2 didn't
 
−
# have support for DDNS.)
 
 
  ddns-update-style none;
 
  ddns-update-style none;
−
 
−
# option definitions common to all supported networks...
 
 
  option domain-name "iac.isu.edu";
 
  option domain-name "iac.isu.edu";
−
#note the comma between dns servers
 
 
  option domain-name-servers 134.50.254.5, 134.50.57.57;
 
  option domain-name-servers 134.50.254.5, 134.50.57.57;
 +
authoritative;
  
−
#these short times are for testing only
+
  default-lease-time 600;
−
  default-lease-time 60;
+
  max-lease-time 1200;
−
  max-lease-time 120;
 
−
 
 
−
# If this DHCP server is the official DHCP server for the local
 
−
# network, the authoritative directive should be uncommented.
 
−
authoritative;
 
  
−
# This is a very basic subnet declaration.
 
 
  subnet 10.0.200.0 netmask 255.255.255.0   
 
  subnet 10.0.200.0 netmask 255.255.255.0   
 
  {
 
  {
 
   range 10.0.34.1 10.0.34.200; #we really don't even need a range
 
   range 10.0.34.1 10.0.34.200; #we really don't even need a range
 
   option routers 10.0.200.1;
 
   option routers 10.0.200.1;
 +
  #address of the TFTP server, optional if the same as the dhcp server
 +
  next-server 10.0.200.1;
 
   filename "pxelinux.0";
 
   filename "pxelinux.0";
 
  }
 
  }
Line 98: Line 80:
 
This will avoid dhcp serving on the outside network!
 
This will avoid dhcp serving on the outside network!
  
−
  service dhcp3-server start
+
  service dhcp3-server restart
  
 
====Setting up tftp====
 
====Setting up tftp====
 
Edit /etc/default/tftpd-hpa
 
Edit /etc/default/tftpd-hpa
−
  RUN_DAEMON="yes" #had problems with inetd in the past
+
  TFTP_USERNAME="tftp"
−
  OPTIONS="-l -a 10.200.0.1 -s /var/lib/tftpboot"
+
TFTP_DIRECTORY="/var/lib/tftpboot"
 +
#Keep TFTP on the inside network
 +
  TFTP_ADDRESS="10.0.200.1:69"
 +
TFTP_OPTIONS="--secure"
  
−
Set up the filesystem to over pxe/tftpd:
+
Set up the filesystem to boot using pxe and tftpd:
 
  mkdir -p /var/lib/tftpboot/pxelinux.cfg
 
  mkdir -p /var/lib/tftpboot/pxelinux.cfg
 
  cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
 
  cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/
Line 116: Line 101:
 
  TIMEOUT 5
 
  TIMEOUT 5
 
  DISPLAY boot.msg
 
  DISPLAY boot.msg
 +
DEFAULT vmlinuz
 +
APPEND root=/dev/nfs initrd=initrd.img netboot=nfs nfsroot=10.0.200.1:/nodes/precise,nolock,ro nomodeset
  
 +
Set up a kernel and initrd in tftpboot
 +
cp /nodes/precise/boot/vmlinuz-3.2.0-24-generic /var/lib/tftpboot/vmlinuz
 +
chmod a+r /var/lib/tftpboot/vmlinuz
 +
cp /nodes/precise/boot/initrd.img-3.2.0-24-generic /var/lib/tftpboot/initrd.img
  
 
====Setting up pxelinux====
 
====Setting up pxelinux====
Line 136: Line 127:
 
  Generating a pseudo-random key using /dev/urandom completed.
 
  Generating a pseudo-random key using /dev/urandom completed.
 
  root@brems:# /etc/init.d/munge start
 
  root@brems:# /etc/init.d/munge start
 +
 +
==Adding a node to the cluster==
 +
#Set node to PXE boot
 +
#Add a new entry to /etc/dhcp3/dhcpd.conf
 +
#Reload dhcpd
 +
#Add new host to /etc/hosts file
 +
#Copy /etc/hosts to /nodes/lucid/etc/hosts
 +
#Create a new var '''cp -r /nodes/lucidvar/template /nodes/lucidvar/newhost'''
 +
#Boot the node
 +
#Add the ssh key to system-wide known-hosts '''ssh-keyscan newhost >> /etc/ssh/ssh_known_hosts'''
 +
#Add the host to cluster ssh
 +
#Add the node to slurm config
  
 
==Customising initrd==
 
==Customising initrd==

Latest revision as of 21:09, 6 August 2012

Network configuration

Head nodes works as NAT router for slave nodes

eth0 connects to outside world eth1 is internal

Internal network is 10.200.0.0/255.255.0.0

OS installation

Normal server Linux install with the following packages:

  • openssh-server
  • tftpd-hpa
  • dhcp3-server
  • nfs-kernel-server
  • debootstrap
  • libpmi
  • mpich2
  • slurm-llnl
  • slurm-llnl-slurmdbd
  • syslinux

Setting up a chroot for the node root

  • Boot to a live CD and mount the root drive as /new
  • Copy the root fs to a chroot rsync -av /new/ /new/nodes/precise/ --exclude /nodes
  • Enter the chroot chroot /nodes/precise/
  • Change the following lines in /etc/initramfs-tools/initramfs.conf
 MODULES=netboot
 BOOT=nfs

The DEVICE= and NFSROOT= lines may also be of use if the tftpboot configuration isn't working

  • Update the initramfs update-initramfs -c -k all

NFS

On the head node, add the following to /etc/exports 

/nodes/lucid    10.200.0.0/24(ro,async,no_root_squash,no_subtree_check)
/home           10.200.0.0/24(rw,async,root_squash,no_subtree_check)

see man 5 exports for more information. There are also more nfs settings in 

/etc/default/nfs-common 
/etc/default/nfs-kernel-server

Reload the NFS server settings 

/etc/init.d/nfs-kernel-server force-reload

Netbooting

Setting up dhcp

Edit /etc/dhcp3/dhcpd.conf as follows: 

 ddns-update-style none;
 option domain-name "iac.isu.edu";
 option domain-name-servers 134.50.254.5, 134.50.57.57;
 authoritative;

 default-lease-time 600;
 max-lease-time 1200;

 subnet 10.0.200.0 netmask 255.255.255.0  
 {
  range 10.0.34.1 10.0.34.200; #we really don't even need a range
  option routers 10.0.200.1;
  #address of the TFTP server, optional if the same as the dhcp server
  next-server 10.0.200.1;
  filename "pxelinux.0";
 }
 #eth0 on brems2
 host brems2_0
 {
  hardware ethernet 00:50:45:5C:10:54;
  fixed-address 10.0.200.2;
  option host-name "brems2";
 }


Edit /etc/default/dhcp3-server 

INTERFACES=eth1

This will avoid dhcp serving on the outside network! 

service dhcp3-server restart

Setting up tftp

Edit /etc/default/tftpd-hpa 

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/var/lib/tftpboot"
#Keep TFTP on the inside network
TFTP_ADDRESS="10.0.200.1:69"
TFTP_OPTIONS="--secure"

Set up the filesystem to boot using pxe and tftpd: 

mkdir -p /var/lib/tftpboot/pxelinux.cfg
cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/

Make a file similiar to the following as /var/lib/tftpboot/boot.msg 

Booting Brems!!

Put the following in /var/lib/tftpboot/pxelinux.cfg/default. The files are referenced relative to /var/lib/tftpboot/ 

TIMEOUT 5
DISPLAY boot.msg
DEFAULT vmlinuz
APPEND root=/dev/nfs initrd=initrd.img netboot=nfs nfsroot=10.0.200.1:/nodes/precise,nolock,ro nomodeset

Set up a kernel and initrd in tftpboot 

cp /nodes/precise/boot/vmlinuz-3.2.0-24-generic /var/lib/tftpboot/vmlinuz
chmod a+r /var/lib/tftpboot/vmlinuz
cp /nodes/precise/boot/initrd.img-3.2.0-24-generic /var/lib/tftpboot/initrd.img

Setting up pxelinux

Testing

Scheduler installation

The Quick Start Administrator Guide is very helpful.

  1. Install
  • slurm-llnl
  • slurm-llnl-slurmdbd
  • slurm-llnl-doc
    • mkdir /var/run/slurm-llnl

Munge

Munge is an authentication framework recommended by slurm. All the configuration it needs is: 

root@brems:# /usr/sbin/create-munge-key
Generating a pseudo-random key using /dev/urandom completed.
root@brems:# /etc/init.d/munge start

Adding a node to the cluster

  1. Set node to PXE boot
  2. Add a new entry to /etc/dhcp3/dhcpd.conf
  3. Reload dhcpd
  4. Add new host to /etc/hosts file
  5. Copy /etc/hosts to /nodes/lucid/etc/hosts
  6. Create a new var cp -r /nodes/lucidvar/template /nodes/lucidvar/newhost
  7. Boot the node
  8. Add the ssh key to system-wide known-hosts ssh-keyscan newhost >> /etc/ssh/ssh_known_hosts
  9. Add the host to cluster ssh
  10. Add the node to slurm config

Customising initrd

Sometimes a customised initrd is necessary.

Extracting

Create a directory to extract into: 

mkdir init_test; cd init_test

Extract an existing initrd 

gzip -d < /var/lib/tftpboot/initrd.img | cpio -iv

Edit the file tree as needed. You can add files (modules for instance), or editing boot script (/init is run by default).

Then package up the directory into a new initrd: 

find ./ | cpio -ov -H newc | gzip > ../initrd.new
Retrieved from "https://wiki.iac.isu.edu/index.php?title=Installing_A_Cluster&oldid=76442"