Daniel Jesse Chu: Case 10029 Details

Add New VMware vSphere Server to Existing 4 Server Cluster

Closed	Case #	10029	Affiliated Job:	New Trier Township District 203¹
	Opened:	Tuesday, April 6, 2010	Closed:	Wednesday, April 7, 2010
	Total Hit Count:	29870	Last Hit:	Thursday, April 18, 2024 2:01:44 PM
	Unique Hit Count:	8851	Last Unique Hit:	Thursday, April 18, 2024 2:01:44 PM
Case Type(s):	Server, Network, Vendor Support

Case Notes(s):

All cases are posted for review purposes only. Any implementations should be performed at your own risk.

Project:

An identical Poweredge R710 server was acquired to add into our existing VMware vCenter cluster of already 4 servers supporting 50 guest sessions. The process, best practices, useful commands and settings used on our existing 4 systems are below in the action item. These were mostly acquired through the consulting agency used over the summer to build our current 4 server VMware vCenter cluster of servers.

As part of this process I pulled down the more recent firmware and drivers to use with the new build and also upgrade our existing systems. The firmware from Dell is bin files that automate the install; however, the driver updates to VMware are a little more complex. Both types have some examples below.

One of the hiccups I had previously seen when I rebuilt our 3rd server was with the QLogic controller. It suddenly would not load the qla2xxx driver. Through some research online, it seems if at first you do not succeed, perform the command again. In this case I am referring to the "esxcfg-module -s ql2xmaxqdepth=64 qla2xxx" - also make very sure to type this command correctly - as any mistakes can cause the driver to stop loading successfully at boot. some troubleshooting commands are included in case the driver stops loading off of a sudden like it did for me twice now.

We received the vSphere v4 U2 beta today (4/7) as part of a solution to Case #10002. This is supposed to address our problem with the Broadcom NIC ports failing after excessive network utilization. Since this issue was acknowledged as a known issue, Dell loaned us 4 Intel add-in 4 port cards to use until this release was available; however, at this point we plan to wait until the official release of the U2 scheduled for June 2010.

Action(s) Performed:

Total Action(s): 1

Action #

Recorded Date

Type

Hit(s)

User

Expand Details

10112

4/7/2010 8:10:27 PM

Server

3816

contact@danieljchu.com

The below are procedures followed to add a recent server into an already ex Collapse ...

Last Hit: Tuesday, April 16, 2024 8:27:43 PM

The below are procedures followed to add a recent server into an already existing 4 server VMware vCenter cluster.

Dell R710 Firmware Revisions (As of 04-06-2010)

o	QLogic: 2.02 [New 2.10]
	- To see the version information:
	+ cat /proc/scsi/qla2xxx/4
	+ cat /proc/scsi/qla2xxx/5
o	BIOS: 1.3.6 [New 2.0.11 03/23/2010]
	- Using winscp, create /tmp/install folder
	- Copy bin file to server /tmp/install folder
	- chmod 777 *
	- Execute ./PER710_BIOS_LX_2.0.11.BIN
	- Follow steps, a reboot will be required.
o	iDRAC 1.3
o	PERC: 6.2.0-0013
o	NIC: 5.0.11 NCSI 2.0.5 [New 5.0.13, A13 03/31/2010]
	- Using winscp, create /tmp/install folder
	- Copy bin file to server /tmp/install folder
	- chmod 777 *
	- Execute ./NETW_FRMW_LX_R259547.BIN
	- Follow steps, a reboot will be required.

BIOS Settings

o	Enable: Execute Disable
o	Enable: Virtualization Technology

QLogic Controller Best Practice Settings

o	Adapter Settings (Changed)
	- Hard Loop 0 (125)
	- Connection Options 2 (1)
	- Fiber Channel Tape Support Enabled (Disabled)
o	Advanced Adapter (Changed)
	- 45 (30)
	- 45 (30)

Install vSphere

o	Named: LOCAL_VMFS_NEWSERVER05 (Block Size 1MB [Max 256 GB])
o	Linux native: 250 MB
o	Linux native: *8 GB [/]
o	Linux swap: *1.6 GB [Swap]
o	Extended
	- Linux native: *4 GB [/var]
	- Linux native: *4 GB [/home]
	- Linux native: *4 GB [/tmp]
	- Linux native: *4 GB [/opt]
	- VMFS: 41.85 GB [System will automatically create this using remaining space)
	- VMware Diagnostic: 101.98 MB

VMware Advanced Setting Change

o	Mem.AllocGuestLargePage (0)

Install PowerPath (Navisphere)

o	Perform the below one at a time (Client vs Agent)
o	unzip <filename>.zip (one at a time, in a folder like /tmp/install)
o	rename
	- mv navicli-6.28.20.1.40-1.noarch.rpm navicli.noarch.rpm
	- mv naviagent-6.28.20.1.40-1.noarch.rpm naviagentcli.noarch.rpm
o	chmod 777 *
o	install
	- ./ESX_install.sh navicli
	- ./ESX_install.sh naviagentcli
o	cd /etc/Navisphere
o	nano agent.config
o	ctrl-w & search for "root"
o	add two lines:
	- user system@<IP of SPA>
	- user system@<IP of SPB>
o	ctrl-x save

QLogic Settings

o	vmkload_mod -l \| grep qla
	- List name of HBA (i.e. qla2xxx)
o	esxcfg-module -s ql2xmaxqdepth=64 qla2xxx
o	esxcfg-boot -b
o	Troubleshooting:
	- esxcfg-module -g qla2xxx (Fetches information)
	- esxcfg-module -d qla2xxx (Disables driver)
	- esxcfg-module -e qla2xxx (Enables driver)
	- vmkload_mod qla2xxx (Manually start driver)
	- cat /proc/vmware/version (See loaded drivers)

Enable root SSH remote access

o	cd /etc/ssh
o	nano sshd_config
o	ctrl-w & search for "root"
o	change "no" to "yes" for "PermitRootLogin"
o	ctrl-x save
o	service sshd restart

Install OpenManage 6.2.0

o	Using winscp, create /tmp/install folder
o	Copy tar.gz file to server /tmp/install folder
o	Expand the file: "tar -zxvf OM_6.2.0_ManNode_A00.tar.gz"
o	"cd linux/supportscripts"
o	"./srvadmin-install.sh -x"
o	"srvadmin-services.sh restart" will restart services
o	Open port 1311 for O.M. "esxcfg-firewall -o 1311,tcp,in,OpenManage"

Updated Broadcom 5709 VMware Drivers

o	Available through vmware.com - downloads - vsphere 4 - drivers & tools - driver CDs
o	Put server into maintenance mode
o	mkdir /tmp/install
o	mkdir /mnt/disk (if it doesn't already exist)
o	Copy downloaded iso to /tmp/install as "disk.iso"
o	mount -o loop /tmp/install/disk.iso /mnt/disk
o	cd /mnt/disk
o	cd offline-bundle
o	esxupdate --bundle=.zip update
o	Restart the server
o	cat /proc/vmware/version
	- Look for "bnx2" to see the driver revision

NIC Structure

o	Broadcom [Or temporary Intel add-in card]
	- vSwitch0
	+ Security Tab
	o Reject/Accept/Accept
	+ NIC Teaming
	o Route based on the originating virtual port ID
	o Link Status only
	o Yes/Yes
	- VM Administrative Network [Assigned to vSwitch0]
	+ Virtual Machine Port Group
	+ VLAN ID: # (i.e. 100)
	- VM Internal Network [Assigned to vSwitch0]
	+ Virtual Machine Port Group
	+ VLAN ID: # (i.e. 200)
	- Service Console [Assigned to vSwitch0]
	+ Service Console Port
	+ VLAN ID: # (i.e. 100)
	+ Static set IP (192.168.100.111-115)
o	Perm. Intel add-in card
	- vSwitch1
	+ Security Tab
	o Reject/Accept/Accept
	+ NIC Teaming
	o Route based on the originating virtual port ID
	o Link Status only
	o Yes/Yes
	- VM DMZ Network [Assigned to vSwitch1]
	+ Virtual Machine Port Group
	+ VLAN ID: # (i.e. 999)
	- vSwitch2
	+ Security Tab
	o Reject/Accept/Accept
	+ NIC Teaming
	o Route based on the originating virtual port ID
	o Link Status only
	o Yes/Yes
	- VMotion [Assigned to vSwitch2]
	+ VMotion and IP Storage Port
	+ VLAN ID: # (i.e. 175)
	+ VMotion Enabled!
	+ Static set IP (192.168.175.111-115)

Connect host to SAN

o	Connect to Fiber Switch and Configure the VSAN/Zoning
	- Example for an existing VSAN (i.e. 1) accross two paths (2 fiber switches) into 2 SPs (SPA & SPB)
	- Note, I did not test these commands for the fiber switches, they are what I reviewed online and believe to be correct
	- Fiber Switch 1 (The WWN address of port 0 of the HBA will be identical below on Switch 1)
	+ zone name CX4-120_SPA0-NEWSERVER05_HBA0 vsan 1
	+ fcalias name CX4-120_SPA0 vsan 1
	+ member pwwn 00:A0:00:00:00:00:00:00 (WWN of SPA 0 - BTW, do not type this part...)
	+ exit
	+ fcalias name NEWSERVER05_HBA0 vsan 1
	+ member pwwn 00:00:00:00:00:00:00:00 (WWN of Port 0)
	+ exit
	+ zone name CX4-120_SPB0-NEWSERVER05_HBA0 vsan 1
	+ fcalias name CX4-120_SPB0 vsan 1
	+ member pwwn 00:B0:00:00:00:00:00:00 (WWN of SPB 0)
	+ exit
	+ fcalias name NEWSERVER05_HBA0 vsan 1
	+ member pwwn 00:00:00:00:00:00:00:00 (WWN of Port 0)
	+ exit
	- Fiber Switch 2 (The WWN address of port 1 of the HBA will be identical below on Switch 2)
	+ zone name CX4-120_SPA1-NEWSERVER05_HBA1 vsan 1
	+ fcalias name CX4-120_SPA1 vsan 1
	+ member pwwn 01:A1:00:00:00:00:00:00 (WWN of SPA 1)
	+ exit
	+ fcalias name NEWSERVER05_HBA1 vsan 1
	+ member pwwn 01:00:00:00:00:00:00:00 (WWN of Port 1)
	+ exit
	+ zone name CX4-120_SPB1-NEWSERVER05_HBA1 vsan 1
	+ fcalias name CX4-120_SPB1 vsan 1
	+ member pwwn 01:B1:00:00:00:00:00:00 (WWN of SPB 1)
	+ exit
	+ tfcalias name NEWSERVER05_HBA1 vsan 1
	+ member pwwn 01:00:00:00:00:00:00:00 (WWN of Port 1)
	+ exit
o	Connect host to Storage Group in Navisphere
	- In Navisphere, first check to see if it reconizes the new host by clicking the "Hosts" tab
	- If not there, try clicking "Tools" - "Connect Host" and following the prompts...
	- Back on the "Storage" tab, Expand Storage Domains - Local domain - EMC SAN Unit - Storage Groups
	- Right click the storage group to add the new VM server to & select Connect Hosts
	- Select the new host from the list and add it to the array
	- Confirm this on the other Storage Processor
	- Using the vSphere Client directly to the server, select the "Configuration" tab - "Storage Adapter" and select to "Rescan"
	- Than select "Storage" - click "Refresh"
	- You should see all the VMFS volumes in the "Datastores" list

Add Host to vCenter

o	Add DNS entry for new server pointed to service console IP (i.e. newserver05.domain.com -> 192.168.100.115)
o	Add new license to vCenter
o	Put server into Maintenance Mode (Prevent DRS)
o	Right click the site (i.e. Site Name) -> "Add Host"
o	Host IP or DNS name (newserver05.domain.com)
o	Username/Password (root)
o	Select license to apply to new server
o	Follow through remaining prompts to integrate server into vCenter

Configure Time Services in vSphere client

o	0.vmware.pool.ntp.org
o	1.vmware.pool.ntp.org
o	2.vmware.pool.ntp.org

Outcome:

After completing these procedures, the new server will become available to the cluster with the existing 4 servers. These procedures also were followed in rebuilding server 3 after Dell delivered a replacement system to us in trying to troubleshoot the Broadcom network port problems. The only difference is removing the host from vCenter and then adding the host back.

Assign VLAN IDs if on a trunked switch port. Otherwise you can leave this field empty.