Scripted Installation Guide

Scripted Installation

In the virtual world your host system is a big of an enigma. While the VMs need to be cared for and backed up on a regular basis, the host system itself doesn't really hold any important data outside of the VMFS volume. In the case of a disaster ( or mass deployment ) it's important to have a way to easily, repeatably and efficiently stamp out ESX systems. While ESX 3i brings the promise of no longer needing to work with scripted installations, that promise is a bit on the horizon. In the present it doesn't currently address all the needs that were filled by agent installs into the console OS such as hardware management agents, backup clients outside of VCB and some custom scripted functionality. In short I don't see ESX 3i sun-setting the non-integrated version for a few years to come. In light of this I've decided to share my scripted ESX process with the public in a 3 part series. The first part of the series is building an ESX 3.x kickstart that will fully configure an ESX host regardless of what type of custom configuration you do. The second part of the series will be getting the kickstart to your hosts. The third installment will deal with some additional advanced configuration.

The ks.cfg Basics

# Install or Upgrade
install
# Text Mode
text
# Installation Method
url --url http://www.yourdomain.com/esx/3.5.0
# Network Settings
network --bootproto static --ip [ip_address] --netmask [netmask] --gateway [gateway] \
--nameserver [dns server] --hostname [hostname] --device=eth0 

# root Password
rootpw changeme
# Authconfig
authconfig --enableshadow --enablemd5

# BootLoader
bootloader --location=mbr --driveorder=cciss/c0d0

# Mouse
mouse none
# Regional Settings
keyboard us
langsupport --default en_US
lang en_US
timezone America/Los_Angeles
# X windowing System
skipx
# Firewall settings
firewall --disabled
# Reboot after install ?
reboot

# Disk Partitioning
clearpart --all --initlabel --drives=cciss/c0d0
part /boot      --fstype ext3    --size 250   --ondisk cciss/c0d0  --asprimary
part /          --fstype ext3    --size 5192  --ondisk cciss/c0d0  --asprimary
part swap       --fstype swap    --size 1600  --ondisk cciss/c0d0  --asprimary
part /var/log   --fstype ext3    --size 4096  --ondisk cciss/c0d0
part /tmp       --fstype ext3    --size 4096  --ondisk cciss/c0d0
part /home      --fstype ext3    --size 2048  --ondisk cciss/c0d0
part None       --fstype vmfs3   --size 8192  --ondisk cciss/c0d0  --grow
part None       --fstype vmkcore --size 100   --ondisk cciss/c0d0


# Accept the EULA
vmaccepteula

vmlicense --mode=server --server=27000@[License Server] --edition=esxFull \
--features=backup

%packages
@base

Some notes about the basics:

In the example, I use cciss/c0d0 as my target disk to install onto. This should work for any type of Compaq/HP hardware, but if you use one of the other vendors you should change all instances of cciss/c0d0 to sda. But be warned! SAN luns can be inadvertently formatted during this process, you should protect yourself by either disconnecting the fibre to the host system prior to installation, or remove the SAN drivers from the install media.

Partitioning is mostly a personal preference and everyone does it slightly differently. Here is my reasoning for setting up the partition table as it is displayed above:

boot partition:
ESX 3.0 requires a 100M /boot partition ( only 50M was required in 2.5.x ). The /boot partition is where different kernel versions are kept, and each patch of ESX stores an additional kernel here. Theoretically if you're going to upgrade a system a number of times you may want to allocate more than 100M to the system.

swap partition:
The rule of thumb for ESX in the past has been swap = (2 * the amount of memory allocated to the service console). In ESX v3.0 the service console is allocated 272M of memory by default. However, it is possible to adjust the service console memory size to a maximum of 800M. By setting the service console swap partition to 1600M we can be assured that we will be able to meet the rule of thumb should we decide to adjust the amount of memory allocated to the service console at a later date.

/ partition:
The default partitioning scheme will leave the system with a 5G root partition, and 2G in /var/log. In past versions VMware has been pretty skimpy on allocating space to the service console and it has manifested itself as serious problems which can be shown by taking a stroll through the VMTN forums. For most uses 5G will be sufficient, I like to have a little extra room to work with and add an additional 3G to the service console for a total of 8G.

/var/log partition:
By having a separate partition for /var/log we can be assured that when log files grow without being purged or rotated appropriately, that the system will continue to function properly.

vmfs3 partition:
I generally use whatever space is left over on the local disks as a single large VMFS3 partition. Note that since VMKswap has been broken up and moved to into VMs directories on the VMFS3 volume that it may no longer be feasible to have VMKswap on local disks, and VM .vmdk files on the SAN.

vmkcore paritition:
The vmkcore partition is used for kernel dumps should the ESX host PSOD and requires 100M of disk.

The %post section

The %post section is where the real meat of the scripted installation comes from. The %post section is called once the system has been installed which give us access to important tools like the bash shell and perl. You'll find that most of my scripts are written in perl, and my installation scripts are no exception. The first thing I do in the %post section is enable kerberos authentication and create users. I enable kerberos first because in the next section I replace the 'system-auth' PAM file which gets re-written each time esxcfg-auth is called with the parameters below.:

%post

# Enable Kerberos Authentication
/usr/sbin/esxcfg-auth --enablead --addomain=domain.com --addc=domain.com

#############################
# Add Groups and Users
#############################
/usr/sbin/groupadd -g 2000 esxadmin
/usr/sbin/groupadd -g 1001 user1
/usr/sbin/groupadd -g 1002 user2
/usr/sbin/groupadd -g 1003 user3
/usr/sbin/useradd -u 1001 -g user1 -G esxadmin user1
/usr/sbin/useradd -u 1002 -g user2 -G esxadmin user2
/usr/sbin/useradd -u 1003 -g user3 -G esxadmin user3

Enable the NTP Service

# Changing the Run levels and firewall setting for the ntpd service
/usr/sbin/esxcfg-firewall -e ntpClient
/sbin/chkconfig --level 345 ntpd on

Next, I create the directory /root/install, and then create a script to download all the other tidbits that I need to install to customize my installation. The script I use here draws from an http source, and can download and replace files on the local host. As you can see from the example I have a number of utilities that I install on each host such as the HP SIM agent, Emulex configuration utilities, HP/Compaq ACU, esx-autopatch.pl. I also set up local system services such as NTP, DNS, SSH, Sudo and PAM:

# Make a directory for all of the install tools/configs
/bin/mkdir -p /root/install

# Perl script to download all necessary software and configuration files
cat << \EOF > /root/install/download.pl
#!/usr/bin/perl -w
use LWP::Simple;
system("/usr/sbin/esxcfg-firewall --allowOutgoing 2&>/dev/null");
my $download_source = "http://ESX_INSTALL_SERVER/esx/software";
&download("$download_source/hpmgmt.conf", '/root/install/hpmgmt.conf');
&download("$download_source/hpmgmt-7.8.0-vmware3x.tgz",\
 "/root/install/hpmgmt-7.8.0-vmware3x.tgz");
&download("$download_source/elxvmwarecorekit-2.1a29-7.3.2-3.i386.rpm" ,\
 "/root/install/elxvmwarecorekit-2.1a29-7.3.2-3.i386.rpm");
&download("$download_source/hponcfg-1.6.0-1.linux.rpm",\
 "/root/install/hponcfg-1.6.0-1.linux.rpm");
&download("$download_source/cpqacuxe-7.85-18.linux.rpm", \
"/root/install/cpqacuxe-7.85-18.linux.rpm");
&download("$download_source/snmpd.conf", "/root/install/snmpd.conf");
&download("$download_source/ntp.conf", "/etc/ntp.conf");
&download("$download_source/resolv.conf","/etc/resolv.conf");
&download("$download_source/step-tickers","/etc/ntp/step-tickers");
&download("$download_source/sshd_config","/etc/ssh/sshd_config");
&download("$download_source/banner","/etc/banner");
&download("$download_source/system-auth","/etc/pam.d/system-auth");
&download("$download_source/sudoers","/etc/sudoers");
&download("$download_source/esx-autopatch.pl","/usr/sbin/esx-autopatch.pl");
system("/usr/sbin/esxcfg-firewall --blockOutgoing 2&>/dev/null");

sub download(){
 my $file_source = shift;
 my $file_target = shift;
 print "Downloading $file_source to $file_target: ";
 my $status = getstore($file_source,$file_target);
 if ( $status eq '200' ){
   print "ok\n";
 }else{
   print "failed [$status]\n";
 }
}
EOF

# Call the script
/usr/bin/perl /root/install/download.pl

Since the installation packages have been downloaded in the step above, we can actually install the packages and optionally apply the updates before the system even boots up for the first time. I won't go into detail about host esx-autopatch.pl works here, but you can find it in the 'utilities' section on this site.

##############################
# Install Software Agents
##############################

echo "Software installation" > /dev/tty2
# ===========================
# Installing Emulex Config Util
# ===========================
if [ ! -f /root/install/elxvmwarecorekit.log ] ; then
        echo "  Installing Emulex Configuration Utility" >/dev/tty2
        /bin/rpm -ivh /root/install/elxvmwarecorekit-2.1a29-7.3.2-3.i386.rpm \
>/root/install/elxvmwarecorekit.log
        # Services are unneeded, just need to install the utility
        /sbin/chkconfig ElxRMSrv off
        /sbin/chkconfig elxlpfc off
fi

# ===========================
# Installing HP SIM agent
# ===========================
if [ ! -f /root/install/HPSIMInstalled.log ] ; then
        echo "  Installing HP SIM Agent 7.8.0" >/dev/tty2
        cd /root/install
        tar -xvzf hpmgmt-7.8.0-vmware3x.tgz
        rm -f /root/install/hpmgmt-7.8.0-vmware3x.tgz
        cd /root/install/hpmgmt/780
        ./installvm780.sh --silent --inputfile /root/install/hpmgmt.conf
        service snmpd stop
        cp /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.old
        cp /root/install/snmpd.conf /etc/snmp/snmpd.conf --reply=yes
        service snmpd start
        sleep 3
        echo "HP SIM Agent 7.8.0 Installed" >/root/install/HPSIMInstalled.log 
fi

# ===========================
# Installing HP iLO Config Util
# ===========================
if [ ! -f /root/install/hponcfg.log ] ; then
    echo "  Installing HP Online Configuration Utility" >/dev/tty2
     /bin/rpm -ivh /root/install/hponcfg-1.6.0-1.linux.rpm  >/root/install/hponcfg.log
fi

# ===========================
# Installing HP Array Config Utility
# ===========================
if [ ! -f /root/install/hpacu.log ] ; then
    echo "  Installing HP Array Configuration Utility" >/dev/tty2
     /bin/rpm -ivh /root/install/cpqacuxe-7.85-18.linux.rpm  >/root/install/hpacu.log
fi



# ===========================
# Installing VMware ESX Patches
# ===========================
if [ ! -f /root/install/esx-autopatch.log ] ; then
    echo "  Installing VMware ESX Patches" >/dev/tty2
        /usr/sbin/esx-autopatch.pl --no-mm >/root/install/esx-autopatch.log
fi

There are some settings that cannot be made during the installation process because they are controlled by the VMkernel, and the VMkernel isn't loaded when the installation takes place. These settings are are applied with commands that start with 'esxcfg'. To process these commands, a postconfig script it setup to be processed the next time the system boots up, and then the post-configuration script deletes itself.

cat << \EOF > /etc/rc3.d/S99postconfig.sh
#!/bin/sh

# Change the amount of RAM allocated to the Service Console
cp /etc/vmware/esx.conf /etc/vmware/esx.conf.ORIG
perl -spi -e 's|/boot/memSize = \"272\"|/boot/memSize = \"800\"|' /etc/vmware/esx.conf
cp /etc/grub.conf /etc/grub.conf.ORIG
esxcfg-boot -g
esxcfg-boot -b

# Allow the esxadmin group to log into the sim agent
perl -spi -e 's|admin-group><\/admin-group|admin-group>esxadmin<\/admin-group|' \
/opt/hp/hpsmh/conf/smhpd.xml 

# Unlink the default Virtual Switches created during install
#  since it was created with < 64 virtual ports:
/usr/sbin/esxcfg-vswitch -U vmnic0 vSwitch0
/usr/sbin/esxcfg-vswif -d vswif0
/usr/sbin/esxcfg-vswitch -d vSwitch0

# Create the vSwitch0 vSwitch and the Service Console port group
/usr/sbin/esxcfg-vswitch -a vSwitch0:64
/usr/sbin/esxcfg-vswitch -A 'Service Console' vSwitch0
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p "Service Console"
/usr/sbin/esxcfg-vswitch -M vmnic3 vSwitch0 -p "Service Console"
/usr/sbin/esxcfg-vswitch -L vmnic0 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic3 vSwitch0

# Create the vSwitch1 vSwitch and the VMotion port group
/usr/sbin/esxcfg-vswitch -a vSwitch1:64
/usr/sbin/esxcfg-vswitch -A "VMotion" vSwitch1
/usr/sbin/esxcfg-vswitch -M vmnic1 vSwitch1 -p "VMotion"
/usr/sbin/esxcfg-vswitch -M vmnic2 vSwitch1 -p "VMotion"
/usr/sbin/esxcfg-vswitch -L vmnic1 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic2 vSwitch1

# Create the vSwitch2 vSwitch for VM traffic
/usr/sbin/esxcfg-vswitch -a vSwitch2:64
/usr/sbin/esxcfg-vswitch -A "VM Network" vSwitch2
/usr/sbin/esxcfg-vswitch -M vmnic4 vSwitch2 -p "VM Network"
/usr/sbin/esxcfg-vswitch -M vmnic5 vSwitch2 -p "VM Network""
/usr/sbin/esxcfg-vswitch -L vmnic4 vSwitch2
/usr/sbin/esxcfg-vswitch -L vmnic5 vSwitch2 

# Assign IP addresses to the Service Console interface and save it permanently:
/usr/sbin/esxcfg-vswif -a -i ESX_IP -n ESX_NETMASK -p 'Service Console' vswif0
/sbin/route add default gw ESX_GATEWAY

# Assign an IP addresses to the VMotion interface and add a VMkernel default route:
/usr/sbin/esxcfg-vmknic -a -i VMOTION_IP -n VMOTION_NETMASK VMotion
/usr/sbin/esxcfg-route -a default VMOTION_GATEWAY

# Enable VMotion on the VMKernel Interface ( this only works on ESX 3.5 )
vimsh -n -e "/hostsvc/vmotion/vnic_set vmk0"


# Reset system to normal boot mode
echo "Seting System to normal boot up mode"
rm /etc/rc3.d/S99postconfig.sh

EOF

Remember to make the script executable after you create it

chmod +x /etc/rc3.d/S99postconfig.sh