Stateless Hypervisor
PXE Boot ESXI and have configuration saved upon reboot (the missing link is configed host (esxi) saved for a reboot).
Ok first...
Follow this doc.. From http://communities.vmware.com/docs/DOC-6824 (I did mine all in linux but I know some people are not used to linux so this is a good howto for them)
Image Extraction and Booting VMware ESXi Installable
VMware ESXi Server is currently distributed in an ISO format designed to install into flash memory or a local hard drive. These instructions help extract the image files to enable boot-from-flash or network PXE booting. The process takes ~ 5 minutes.
These instructions assume basic familiarity with images and formats. Although other tools are available, we used a free lite version of Access Data’s Forensics Tool Kit and a trial version of RARLAB’s WinRAR:
http://downloads.accessdata.com/current_releases/imager/imager-ftk_imager-2.5.1.lite.exe
http://www.rarlab.com/rar/wrar371.exe.
Download the ESXi ISO image from VMware
VMware ESXi is currently available for free from VMware’s web site at http://www.vmware.com/download/esxi/ (it may move in the future). The download requires registration and includes a license (requires activation). The downloaded image will be named something like:
VMware-VMvisor-InstallerCD-3.5.0_Update_2-103909.i386.iso
Extract install.tgz
Using a tool like the Forensic Tool Kit (FTK) open the ISO as an Image File, browse to \CDROM\Session 1\Track 01\CDROM and extract install.tgz (if using FTK just open install.tgz). Do not use the first image you find in CDROM top level– that is the image used to install the real image:
Extract VMware-VMvisor…dd.bz2
Unzip install.tgz (or open it in FTK – this takes a while, then use “View File List”) and extract the ESXi FAT image from usr\lib\vmware\installer (right click on file and “Export Files …”:
Extract dd image from .bz2
Using something like WinRAR uncompress the bz2 image, which will put the .dd into its own directory:
(Optional) Extract files from dd Image
There are multiple sets of files in the dd image - the booter files are in Partition 4 and the ESXi files are in Partition 5. Partition 8 contains tools useful tools not needed for booting such as the VMtools and the Virtual Infrastructure Client. FTK Imager or similar tools can be used to extract the files in this section.
ESXi uses H. Peter Anvin’s excellent SYSLINUX (http://syslinux.zytor.com/) package for booting (this does not make ESXi Linux in any way). SYSLINUX is split into multiple pieces for booting from local storage, ISO images (CD-ROMs), PXE, etc.
(Optional) Extract SYSLINUX Booter and Configuration from dd Image
Depending on your final installation/configuration strategy, you may need the configuration and boot files from Partition 4.
Extract ESXi from dd Image
The ESXi image files are in Partition 5 (Hypervisor1):
Voila!
You now have ESXi suitable for PXE booting, ISO or flash installation, or booting from local storage:
Flash Drive Booting
ESXi can be run from a USB flash drive (aka thumb drive) that’s at least 1GB, is USB 2.0 certified, and does not have “U3” or other extra features. “U3” or any other Windows security standards can cause problems. The USB flash disk should be wear-leveled or use other technologies that enable the drive to extend flash media lifespan using cells from the unpartitioned portion of the key.
To create the flash image from Windows you need a program such as WinImage (shareware) or the free dd from http://www.chrysocome.net/dd. dd is a low level tool that is very dangerous. Choosing the wrong partition will overwrite system data and is unrecoverable. So be careful.
Copying to Flash
Run “dd --list” to see the disk partitions available on your system. The USB drive will be labeled as “Removable Media”. In the above example it’s Harddisk1, mounted on \\d:. Be careful if you have multiple hard disks or other media on your system – getting the harddisk argument wrong can result in data loss and/or destroy your system.
Copy the dd image extracted in Section 4 to the flash drive with:
dd if=VMware-VMvisor-big-3.5.0-XXXXX.i386.dd of=\\?\Device\Harddisk1\Partition0 bs=1M --size --progress
Remember to eject the drive cleanly to ensure that all data has been written. The drive can now be tested on systems that support boot-from-USB.
PXE Booting
ESXi can be booted directly onto bare metal using any system that supports PXE using the PXELINUX boot tools. PXELINUX is compatible with other PXE-boot systems including open source Linux tools such as xCAT, Windows programs such as tftbd32, and proprietary systems like Altiris. The PXELINUX web site has comprehensive instructions for enabling and configuring the required services on Linux. Consult with your network administrator before enabling any PXE-related services as they may affect other services and systems on your network.
PXELINUX information: http://syslinux.zytor.com/wiki/index.php/PXELINUX
SYSLINUX distribution: http://www.kernel.org/pub/linux/utils/boot/syslinux/
xCAT: http://xcat.sourceforge.net/
tftpd32: http://tftpd32.jounin.net/
Get PXELINUX Booter files
PXE requires a few files from the PXELINUX distribution that are not in the VMware ESXi ISO. Copy the ESXi image and these files from the SYSLINUX distribution and to your TFTP repository:
com32\modules\mboot.c32
com32\menu\menu.c32
core\pxelinux.0
You should have something like this:
Create pxelinux.cfg
In your TFTP repository create the directory pxelinux.cfg and a file called “default” with this configuration text:
default menu.c32
menu title PXE Boot VMware ESXi
timeout 100
label ESXi
menu label Boot VMware ESXi
kernel mboot.c32
append vmkernel.gz --- binmod.tgz --- environ.tgz --- cim.tgz
ipappend 2
label Hard
menu label Boot from local drive
localboot 0
This will boot VMware ESXi by default onto any system requesting PXE service. The PXELinux system is very flexible and can be configured to map MAC or IP addresses to images, give users menus of which image to boot (with timeouts to a default), etc.
Start PXE Server and Boot a Machine
If your DHCP and TFTP services are correctly configured you should now be able to boot a system – remember to enable PXE booting in its boot options, possibly changing the boot order to place PXE before local media.
Figure 1 - PXELINUX boot menu
Figure 2 - Downloading ESXi over Network
Figure 3 - ESXi Initializing
Enjoy!
Thank you Lance M. Berc
Make sure this all works before next steps...
Now to have some real fun.
You need to have Remote CLI install somewhere ..
http://www.vmware.com/download/download.do?downloadGroup=VI-RCLI and accept the EULA to reach the download links page.
http://www.vmware.com/download/eula.do
Now boot your pxe esxi.
FYI I had my pxe boot 2 different host (with different configs), and had the dhcp server assign a IP and hostname to esxi via the MAC address (this is all controlled via pxe) of the pc/server (I used a dell 745).
After it is booted use the VI client to make the changes you want. Like nfs maps, iscsi, install serial number and assign a pasword. If your controller is supported you could use local disk (mine was not). But you want all guest in a single place anyway, no harddrive in a pc, nothing to break. Build your nfs server with raid 5 and have fun.
After you are happy with the config go to your command prompt (where you install remote CLI). Do something like this.
vicfg-cfgbackup --server esxihost2.local.com -s /tmp/host2 --username root
You should see this
"Saving firmware configuration to /tmp/host2 ..."
If so great!!!
Next ...
Take the file called (hear comes some pxe dust magic) /tmp/host2 and copy it to the location you put these files at binmod.tgz boot.cfg cim.tgz environ.tgz license.tgz mboot.c32 oem.tgz vmkernel.gz
Rename the file host2 to configBundle.tgz
Next go into your pxelinux.cfg dir and edit the file default (more pxe dust magic)
to look like this
default 3i
prompt 0
label 3i
kernel mboot.c32
append vmkernel.gz --- binmod.tgz --- environ.tgz --- cim.tgz --- configBundle.tgz
Now reboot your esxi host (I just turned it off). Upon reboot, it will read the configBundle.tgz file which you created and be at the same state.
Now here is a "issue" you make any changes to the host. IE add a guest, change the root password you need to run the vicfg-cfgbackup again and replace the configBundle.tgz file (before a reboot). Best thing to do is script it to make a backup once a hour and replace the configBundle.tgz file. I have looked into how vmware does it. They have a auto-backup.sh script that run once a hour to the local file system, be a usb or disk install. Vmware also has a script to run the backup.sh at a reboot/shutdown request. I'm working on how to do that as well.
But you should be Statelss ESXI!!!! No local hard drive, no usb key, nothing. PXE boot f'in ye ha.
Now I did some pxe fun and have different configs of each of my esxi host (based on mac address), so different machines have different ip, and different configs.
Hope it makes other people happy as me. It was a pain that esxi did not like the local disk controller on a dell 745 pc and I did not want to mess with usb. Thank god for pxe.
I now have 1 pc I can drop in, get some mac address of other pc's, have them pxe boot, and start messing with esxi and vm's. The one pc needs syslog,pxe, dhcp,dns,tftp,nfs, or iscsi setup on it. With wine/vmware workstation running windows to use the VI Client.
Ok, a little update (9/6/08). Some people have emailed me asking how I did 2 different pxe esix host boots with different configs. It is all about pxe, and the tftp dir.
In my pxelinux.cfg dir.
I have the config file for each machine based on it's IP (again this is all pxe/dhcp stuff boat loads of howto's)
So I have a a file called 0a4d3c40 and 0a4d3c41
0a4d3c40 in this file
label 3i
kernel /esxihost1/mboot.c32
append /esxihost1/vmkernel.gz --- /esxihost1/binmod.tgz --- /esxihost1/environ.tgz --- /esxihost1/cim.tgz --- /esxihost1/configBundle.tgz
0a4d3c41 in this file
label 3i
kernel /esxihost2/mboot.c32
append /esxihost2/vmkernel.gz --- /esxihost2/binmod.tgz --- /esxihost2/environ.tgz --- /esxihost2/cim.tgz --- /esxihost2/configBundle.tgz
I also have 2 dir called esxihost1 and esxihost2 in the root of my tftp dir. in those I have.
binmod.tgz boot.cfg cim.tgz configBundle.tgz default environ.tgz license.tgz mboot.c32 oem.tgz pxelinux.0 vmkernel.gz
Don't forget the configBundle.tgz is the config for what ever host you are working with, so replace that file as needed.
I hope this helps.
Ok, another update (9/8/08). I'm in the esx beta. They have changed how the usb/embedded boot works. When vmware says I can let other people know I will let you know. But for now. I can not. But I can tell you I got esxi 4 beta to pxe boot but the config is different then above.
UPDATE 4/14/09
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008971
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009034
OK as requested by many many users. PXE Boot ESX4i.
This is a quick run down.
pxelinux.cfg/default file
default menu.c32
menu title PXE Boot VMware ESXi
timeout 100
label ESXi
menu label Boot VMware ESXi
kernel mboot.c32
append vmkboot.gz --- vmk.gz --- pkgdb.tgz --- oem.tgz --- license.tgz --- cim.vgz --- cimstg.tgz --- sys.vgz --- mod.tgz
ipappend 2
label Hard
menu label Boot from local drive
localboot 0
Files needed in tftpboot
boot.cfg default menu.c32 pkgdb.tgz safeboot.c32 vmk.gz
cimstg.tgz license.tgz mod.tgz pxelinux.0 sys.vgz
cim.vgz mboot.c32 oem.tgz pxelinux.cfg vmkboot.gz
These files are from the .dd extract.
http://www.vmware.com/support/developer/vcli/
(ubuntu 9.04)
apt-get install libssl-dev
apt-get install libxml2-dev
cpan force install XML::LibXML::Common
cpan force install XML::LibXMLcpan Class::MethodMaker
cpan force install Crypt::SSLeay
At the bootloader type "shift-O" this will bring up an advanced option
and type "shell", continue booting and then you can click "alt+f1" to
be able to login. You'll get some of the standard tools (esxcfg-*,etc)
you see in 3.0.x / 3.5.x but this option is unsupported by VMware, use
at your own risk. I believe this is for further support troubleshoot
and is not documented.
To get access to the console you can type "unsupported" as described in this KB article
http://kb.vmware.com/kb/1003677
but this will only give you console access and not SSH. You'll need to
look at the RCLI (vifs) or other tools to copy a VM to and from your 3i
host.
edit the sshd_config file by entering the command:
nano /etc/ssh/sshd_config
Look for the line called "PermitRootLogin no" and change it to "PermitRootLogin yes"
After changing the line you have to restart the sshd server by running the command:
./etc/init.d/sshd restart
Boot your ESXi server, wait for it to finish loading, and then do the following:
1. ALT-F1 to change to the main console.
2. Type 'unsupported' (you will not be able to see what you're typing).
3. When prompted, enter the root user's password.
4. Type: vi /etc/inetd.conf
5. Find the line that begins with #ssh
6. Cursor over the first 's' and press the 'i' key (for insert mode)
7. Press backspace
8. Press ESC
9. Type SHIFT+colon (:) and then 'wq!' to write and exit.
10. Type 'ps | grep inetd' to find the inetd process
11. Send the hangup signal to the process ID output from step 10 with: kill -s HUP <PID>
12. You can now SSH into your ESXi server.
This port dcron from slackware.
You can create
/var/spool/cron/crontabs/user files,
for example:
# cat /var/spool/cron/crontabs/root
# Run daily cron jobs at 4:40 every day:
40 4 * * * /etc/cron/daily > /dev/null 2>&1
host esxhost1 {
hardware ethernet 00:00:00:00:00:00;
fixed-address 192.168.0.100;
}
./connect.pl --server 10.77.10.77 --username root --password pass
mccann.jim@gmail.com