Reconnecting ESX(i) hosts with 1000V installed

I follow a certain policy: “You break it; you fix it”.  Why?  When you break something, you learn a lot about why it broke and what it takes to fix it.  Hopefully, you also learn how to prevent it from happening again.  Two, I’m not stuck dealing with the problem someone else caused.

Tonight was one where I broke something, and now I needed to fix it.

I had to reconfigure a UCS system running 3 vSphere 4.1 ESXi hosts connected to a Cisco 1000V.  Before powering down the hosts, I made some changes to the 1000v and the UCS.  So far, so good.  All the hosts were powered down along with vCenter and the 1000V VSMs, which were running on the UCS chassis.  I made my configuration changes on the UCS and powered up the first host.  No connectivity.  Something was not jiving between the 1000V config on the host itself and the vNIC configuration from the UCS.  It turned out to be a misconfiguration of the native vlan on the vNIC presented to the host and the native vlan configuration configured on the host by the 1000V.  I returned some values to their previous settings, and the host came back up.  I updated the 1000V config, which took the host back down, but I adjusted the UCS to the new configuration I wanted to bring the host back up.  Great, except I still had to bring two more hosts back online.

This is where I dove into vemcmd on the ESXi host.  When I brought up the next host, I opened a local console to the system.  “vemcmd show port” showed me the native vlan configuration error that was causing my issue.  So how do you fix this?  It’s actually quite simple.  When looking at the output of the previous command, you’ll notice three Trunk ports on the system.  Two are the actual physical uplinks connected from the  system.  The third is the port channel that is formed between the two nics.  To bring the system back online, you need to issue the following command: vemcmd set port-mode trunk native-vlan <native vlan> ltl <ltl of port-channel>.  After issuing this command, the system was back online.  You’ll notice that after issuing the command, the native vlan of the physical nics remains the same.  After the VEM gets the updated configuration, the native vlan is now the correct on the physical nics.

Advertisement

Working with UCS and vCenter: IPMI Settings

In my last blog post, Working with Cisco UCS XML API, I went through the steps needed to login and logout of UCS Manager using PHP and PowerShell.  Today, I decided to write the first PowerShell script I needed which would automatically update the IPMI settings on the ESX hosts with the settings from UCS Manager.  In writing this script, I did come across this post from Mike Laverick on using PowerShell to enable DPM settings

Now, this script is important for two reasons.  The first reason is that to configure IPMI on ESX hosts, you need both IP address and the MAC address of the CIMC controller from the UCS blades.  When having to do this for multiple ESX hosts, a script is very handy.  Now the second reason is more important.  Currently in UCS, the CIMC IP is tied to the actual blade and not the service profile.  This means if a service profile is associated with a new blade, the ESX host IPMI information is now pointing to the old blade.  Not good!

With that said, here is the first version of this script as a PowerShell script.  (This will also come as a PHP script and possibly other languages in the near future.)  In order to run this script, you will need PowerCLI installed.  You will also need IP and credentials for vCenter and UCS, as well as the IPMI user and password.  This script has only been tested against UCS 1.3.1 and vSphere 4.1.  I will be testing this shortly against vSphere 4.0.  This is also a v0.1 script, so there is no error checking (who needs it right?).  That will be coming in a future version.

###DISCLAIMER###

I provide this script for academic purposes only.  Use at your own risk.  I am not responsible for any damages it may cause.

###DISCLAIMER###

So what does the script do?  The script first logs into vCenter and UCS Manager.  It then grabs the UUID from each ESX host.  Once done, it will pass this UUID to UCS to find the corresponding blade and return the CIMC IP and MAC for that blade.  The script will then write these settings to the IPMI of the ESX host.  Finally, it will log out of the UCS and vCenter.

#####################################################################
# UCS_VMware_IMPI.ps1 v0.01  By: Justin Guidroz
#
# This script will connect to UCS Manager, collect IPMI information
# for the provisioned blades, and update the appropriate IPMI
# information on the ESX hosts.  The script can also be run to
# update the IPMI information if a service profile is moved to a
# different blade.
#
#####################################################################
Add-PSSnapin VMware.VimAutomation.Core

### Needed variables
$ucsUrl = “”
$ucsUser = “”
$ucsPassword = “”
$vCenterUrl = “”
$vCenterUser = “”
$vCenterPassword = “”
$ipmiUser = “”
$ipmiPassword = “”

### Function ucs_post
### Required variables: $url = UCS Url $data = XML to send to UCS
### Returns: XML response from UCS
function ucs_post($url,$data) {
$request = [System.Net.HttpWebRequest] [System.Net.HttpWebRequest]::Create(“http://&#8221; + $url +”/nuova”)
$request.Method = “POST”
$request.ContentType = “text/xml”
$sendData = new-object System.IO.StreamWriter($request.GetRequestStream())
$sendData.Write($data)
$sendData.Close()
$response = $request.GetResponse()
$sr = new-object System.IO.StreamReader($response.GetResponseStream())
$xml = [xml] $sr.ReadToEnd()
return $xml
}

### Function ucs_login
### Required variables: $inName = UCS username $inPassword = UCS password $url = UCS url
### Returns: Cookie after login
### Todo: Error Checking
function ucs_login($inName, $inPassword, $url) {
$aaaLogin = “<aaaLogin inName='” + $inName + “‘ inPassword='” + $inPassword + “‘ />”
$xml = ucs_post $url $aaaLogin
$outCookie = $xml.aaaLogin.outCookie
return $outCookie
}

### Function ucs_logout
### Required variables: $url = UCS url $inCookie = Cookie for session to logout
### Returns: Status of logout
### Todo: Error Checking
function ucs_logout($url, $inCookie) {
$aaaLogout = “<aaaLogout inCookie='” + $inCookie + “‘ />”
$xml = ucs_post $url $aaaLogout
$outStatus = $xml.aaaLogout.outStatus
return $outStatus
}

### Function get_esx_hosts
### Required variables: $vCenter = $vCenter server object
### Returns: ESX hosts in vCenter
### ToDo: Error checking. More logic
function get_esx_hosts($vCenter) {
$esxhosts = @()
$VMHosts = Get-VMhost -server $vCenter
foreach ($h in $VMHosts) {
$esxhost = “” | Select-Object Name, Uuid, IpmiIp, IpmiMac
$esxhost.Name = $h.name
$v = Get-VMHost -Name $h | Get-View
$esxhost.Uuid = $v.Summary.Hardware.Uuid
$esxhosts += $esxhost
}
return $esxhosts
}

### Function get_blade_dn
### Required variables: $uuid = ESX UUID $url = UCS url $cookie = UCS cookie for session
### Returns: DN of physical blade
### Todo: Error Checking
function get_blade_dn($uuid, $url, $cookie) {
$computeBlade = “<configResolveClass cookie='” + $cookie + “‘ inHierarchical=’false’ classId=’computeBlade’><inFilter><eq class=’computeBlade’ property=’uuid’ value='” + $uuid + “‘ /></inFilter> </configResolveClass>”
$bladeXml = ucs_post $url $computeBlade
return $bladeXml.configResolveClass.outConfigs.computeBlade.dn
}

### Function get_blade_ipmi
### Required variables: $dn = DN of physical blade $url = UCS url $cookie = UCS cookie for session
### Returns: Management Interface XML response from UCS
### Todo: Error Checking
function get_blade_ipmi($dn, $url, $cookie) {
$mgmtIf = “<configResolveClass cookie='” + $cookie + “‘ inHierarchical=’false’ classId=’mgmtIf’><inFilter><eq class=’mgmtIf’ property=’dn’ value='” + $dn + “/mgmt/if-1′ /></inFilter> </configResolveClass>”
$mgmtIfXml = ucs_post $url $mgmtIf
return $mgmtIfXml
}

### Function get_host_ipmi
### Required variables: $esxhost = ESX Host object $url = UCS url $cookie = UCS cookie for session
### Returns: Updated ESX host object
### Todo: Error checking
function get_host_ipmi($esxhost, $url, $cookie) {
$bladeDn = get_blade_dn $esxhost.Uuid $url $cookie
$mgmtIfXml = get_blade_ipmi $bladeDn $url $cookie
$esxhost.IpmiIp = $mgmtIfXml.configResolveClass.outConfigs.mgmtIf.extIp
$esxhost.IpmiMac = $mgmtIfXml.configResolveClass.outConfigs.mgmtIf.mac
return $esxhost
}

### Function set_host_ipmi
### Required variables: $esxhost = ESX host object $vCenter = vCenter Server Object
### Returns: nothing (should be changed)
### Todo: Error checking
function set_host_ipmi($esxhost, $vCenter) {
$v = Get-VMHost -server $vCenter -Name $esxhost.Name | % {Get-View $_.Id}
$ipmi = New-Object Vmware.Vim.HostIpmiInfo
$ipmi.BmcIpAddress = $esxhost.IpmiIp
$ipmi.BmcMacAddress = $esxhost.IpmiMac
$ipmi.Login = $ipmiUser
$ipmi.Password = $ipmiPassword
$v.UpdateIpmi($ipmi)
}

### Where the fun begins
### Lets log in to vCenter and UCS
$vCenter = Connect-VIServer -server $vCenterUrl -user $vCenterUser -password $vCenterPassword
$cookie = ucs_login $ucsUser $ucsPassword $ucsUrl

### Grabbing ESX hosts from vCenter
Write-Host “Getting ESX Hosts from vCenter”
$esxhosts = get_esx_hosts $vCenter

### Get the IPMI settings from UCS and update ESX hosts with information
Write-Host “Getting IPMI Settings from UCS and configuring ESX”
foreach ($h in $esxhosts) {
$h = get_host_ipmi $h $ucsUrl $cookie
set_host_ipmi $h $vCenter
}

### Fun as ended, time to log out.
Write-Host “Logging out of UCS”
$outStatus = ucs_logout $ucsUrl $cookie
Write-Host $outStatus
Write-Host “Logging out of vCenter”
Disconnect-VIServer -server $server -confirm:$false

And here is the PowerShell file.  UCS_VMware_IPMI.ps1.doc The .doc will need to be deleted from the end of the file (only way I could get the file uploaded).

Looks like I need to figure out a way to make code display on my blog. Another day.

Working with the Cisco UCS XML API

Working with Cisco UCS systems on a daily basis, I decided to take some time this weekend to learn more about the XML API.  Honestly, it is a thing of beauty and simplicity.  Working with the Cisco UCS XML API is as simple as sending an HTTP POST with an XML document to the UCS Manager and getting the response.  That’s it.  Isn’t that great?

Seeing as I have a couple of projects that will benefit from interacting with UCS systems, I decided to write a script in PHP to login and logout of a UCS for starters.  Then Rob Markovic asked, “Why not PowerShell?”.  So I wrote the same script in PowerShell as well.  I do need to give credit to Steve Chambers for this post: Access UCS API with Ruby.  This was beneficial as I was reading the XML API documentation and starting to work on my scripts.

***Disclaimer***

I am not an expert in PHP or PowerShell.  I am providing this information for academic purposes only to show how easy it is to work with the UCS XML API.  I am also not responsible for any damage this may cause your systems.

***Disclaimer***

One thing you’ll notice with the Cisco UCS XML API is that the only methods that require a username and password to be passed is the aaaLogin and aaaRefresh methods.  All other methods require a cookie to be passed, which is only obtained by logging in or refreshing your session to the UCS Manager.

Let’s start with what’s needed for PHP to work with the Cisco XML API.  I downloaded and installed HTTP_Request2 from PEAR to make the HTTP call easier.

require_once HTTP/Request2.php;

Now we need to set the url of our server and the XML parameters for logging into the UCS system.

$url = “http://server ip or dns/nuova”;

$aaaLogin = “<aaaLogin inName=’username’ inPassword=’password’ />”;

With the url and XML set, we can now create our HTTP request for logging in.

$request = new HTTP_REQUEST2($url, HTTP_REQUEST2::METHOD_POST);

$request->setHeader(“Content-type: text/xml”);

$request->setBody($aaaLogin);

After the request is created, we can send the data to the UCS Manager and get our response.

$response = $request->send();

To make it easier to work with the response, we are going to create SimpleXMLElement from the response.

$xml = new SimpleXMLElement($response->getBody());

Now to get our cookie from the XML response, we just need to do the following:

$outCookie = $xml->attributes()->outCookie

With the cookie now obtained, we are free to call other UCS XML API methods for interacting with the system.  To logout, we need to create the logout xml and issue an HTTP POST with this data.  (I will not repeat the code for issuing the HTTP POST).  Here is the XML for logging out of the system.

$aaaLogout = “<aaaLogout inCookie=’cookie for session’ />”;

To verify status of logout, you look at the outStatus attribute from the XML response.

$outStatus = $xml->attributes()->outStatus;

Here is how to do the same with PowerShell.

$url = “http://server ip or dns/nuova”

$aaaLogin = “<aaaLogin inName=’username’ inPassword=’password’ />

$request = [System.Net.HttpWebRequest] [System.Net.HttpWebRequest]::Create($url)

$request.Method = “POST”

$request.ContentType = “text/xml”

$sendData = new-object System.IO.StreamWriter($request.GetRequestStream())

$sendData.Write($aaaLogin)

$sendData.Close()

$response = $request.GetResponse()

$sr = new-object System.IO.StreamReader($response.GetReponseStream())

$xml = [xml] $sr.ReadToEnd()

$outCookie = $xml.aaaLogin.outCookie

To logout with PowerShell, we create the aaaLogout XML with the cookie we just obtained and issue another HTTP Request (code not repeated).

$aaaLogout =”<aaaLogout inCookie=’cookie for session’ />”

And to get the status of the logout.

$outstatus = $xml->aaaLogout->outStatus

There you have it.  PHP and PowerShell example code for logging into and out of Cisco UCS Manager.  You can find the programmer’s guide here:  Cisco UCS Manager XML API Programmer’s Guide.  I would also suggest checking out these Cisco blog posts on the Cisco UCS Manager XML API by John McDonough: UCS XML API “Hello World”, UCS XML API “curl and xml”, and UCS XML API Query Methods.

Automating ESXi 4.1 continuation

William Lam has posted a great article on tips and tricks of automating ESXi 4.1 here.  Go read his post first.

Seriously, go read his post first.

Ok, you’re back?

I wanted to add to his post with some issues and fixes I’ve run across with moving from ESX to ESXi automation.

With ESX 4.0, at the beginning of the ks.cfg file, you would specify an install line.  Nothing else was needed as the url location of the install bits was passed in the PXE config line.  No more.  For ESX(i) 4.1, you must specify the url for the install location in the ks.cfg file.

Beginning with ESX(i) 4.1, ip append 2 in the PXE config boot line no longer works.  What does this do exactly?  This told the ESX(i) to use the same nic it booted from for downloading the install bits.  If this was not specified, ESX(i) would try to use the first nic it detected, vmnic0.  With ESX(i) 4.1, you must specify ksdevice=vmnic(x) if you are booting and installing from a different nic than vmnic0.  I’m not sure why this behavior changed between 4.0 and 4.1.

William did mention vim-cmd.  In ESX, the command is actually vmware-vim-cmd.  For ESXi, be sure to update these commands to vim-cmd.

For disk setup, I have two lines in my ks.cfg file: clearpart –onfirstdisk –overwritevmfs and autopart –onfirstdisk –overwritevmfs.  I did notice this will create the datastore name on each ESXi host as datastore.  I need to investigate if this can be changed at this part, or add a line in the script to update that name.

In my working with scripting ESXi installations, I’ve found it much easier to have everything run after firstboot then trying to run the configurations before firstboot.

Plenty of advanced configurations (SNMP, Active Directory, adding to vCenter) can be handled through the scripted install, but involve writing a python script to handle.  For some, it will be easier using the supplied commands in the vMA or writing PowerShell scripts.

Outside of these differences, scripted ESXi installs are awesome.