Reconnecting ESX(i) hosts with 1000V installed

I follow a certain policy: “You break it; you fix it”.  Why?  When you break something, you learn a lot about why it broke and what it takes to fix it.  Hopefully, you also learn how to prevent it from happening again.  Two, I’m not stuck dealing with the problem someone else caused.

Tonight was one where I broke something, and now I needed to fix it.

I had to reconfigure a UCS system running 3 vSphere 4.1 ESXi hosts connected to a Cisco 1000V.  Before powering down the hosts, I made some changes to the 1000v and the UCS.  So far, so good.  All the hosts were powered down along with vCenter and the 1000V VSMs, which were running on the UCS chassis.  I made my configuration changes on the UCS and powered up the first host.  No connectivity.  Something was not jiving between the 1000V config on the host itself and the vNIC configuration from the UCS.  It turned out to be a misconfiguration of the native vlan on the vNIC presented to the host and the native vlan configuration configured on the host by the 1000V.  I returned some values to their previous settings, and the host came back up.  I updated the 1000V config, which took the host back down, but I adjusted the UCS to the new configuration I wanted to bring the host back up.  Great, except I still had to bring two more hosts back online.

This is where I dove into vemcmd on the ESXi host.  When I brought up the next host, I opened a local console to the system.  “vemcmd show port” showed me the native vlan configuration error that was causing my issue.  So how do you fix this?  It’s actually quite simple.  When looking at the output of the previous command, you’ll notice three Trunk ports on the system.  Two are the actual physical uplinks connected from the  system.  The third is the port channel that is formed between the two nics.  To bring the system back online, you need to issue the following command: vemcmd set port-mode trunk native-vlan <native vlan> ltl <ltl of port-channel>.  After issuing this command, the system was back online.  You’ll notice that after issuing the command, the native vlan of the physical nics remains the same.  After the VEM gets the updated configuration, the native vlan is now the correct on the physical nics.

Advertisement

Working with UCS and vCenter: IPMI Settings

In my last blog post, Working with Cisco UCS XML API, I went through the steps needed to login and logout of UCS Manager using PHP and PowerShell.  Today, I decided to write the first PowerShell script I needed which would automatically update the IPMI settings on the ESX hosts with the settings from UCS Manager.  In writing this script, I did come across this post from Mike Laverick on using PowerShell to enable DPM settings

Now, this script is important for two reasons.  The first reason is that to configure IPMI on ESX hosts, you need both IP address and the MAC address of the CIMC controller from the UCS blades.  When having to do this for multiple ESX hosts, a script is very handy.  Now the second reason is more important.  Currently in UCS, the CIMC IP is tied to the actual blade and not the service profile.  This means if a service profile is associated with a new blade, the ESX host IPMI information is now pointing to the old blade.  Not good!

With that said, here is the first version of this script as a PowerShell script.  (This will also come as a PHP script and possibly other languages in the near future.)  In order to run this script, you will need PowerCLI installed.  You will also need IP and credentials for vCenter and UCS, as well as the IPMI user and password.  This script has only been tested against UCS 1.3.1 and vSphere 4.1.  I will be testing this shortly against vSphere 4.0.  This is also a v0.1 script, so there is no error checking (who needs it right?).  That will be coming in a future version.

###DISCLAIMER###

I provide this script for academic purposes only.  Use at your own risk.  I am not responsible for any damages it may cause.

###DISCLAIMER###

So what does the script do?  The script first logs into vCenter and UCS Manager.  It then grabs the UUID from each ESX host.  Once done, it will pass this UUID to UCS to find the corresponding blade and return the CIMC IP and MAC for that blade.  The script will then write these settings to the IPMI of the ESX host.  Finally, it will log out of the UCS and vCenter.

#####################################################################
# UCS_VMware_IMPI.ps1 v0.01  By: Justin Guidroz
#
# This script will connect to UCS Manager, collect IPMI information
# for the provisioned blades, and update the appropriate IPMI
# information on the ESX hosts.  The script can also be run to
# update the IPMI information if a service profile is moved to a
# different blade.
#
#####################################################################
Add-PSSnapin VMware.VimAutomation.Core

### Needed variables
$ucsUrl = “”
$ucsUser = “”
$ucsPassword = “”
$vCenterUrl = “”
$vCenterUser = “”
$vCenterPassword = “”
$ipmiUser = “”
$ipmiPassword = “”

### Function ucs_post
### Required variables: $url = UCS Url $data = XML to send to UCS
### Returns: XML response from UCS
function ucs_post($url,$data) {
$request = [System.Net.HttpWebRequest] [System.Net.HttpWebRequest]::Create(“http://&#8221; + $url +”/nuova”)
$request.Method = “POST”
$request.ContentType = “text/xml”
$sendData = new-object System.IO.StreamWriter($request.GetRequestStream())
$sendData.Write($data)
$sendData.Close()
$response = $request.GetResponse()
$sr = new-object System.IO.StreamReader($response.GetResponseStream())
$xml = [xml] $sr.ReadToEnd()
return $xml
}

### Function ucs_login
### Required variables: $inName = UCS username $inPassword = UCS password $url = UCS url
### Returns: Cookie after login
### Todo: Error Checking
function ucs_login($inName, $inPassword, $url) {
$aaaLogin = “<aaaLogin inName='” + $inName + “‘ inPassword='” + $inPassword + “‘ />”
$xml = ucs_post $url $aaaLogin
$outCookie = $xml.aaaLogin.outCookie
return $outCookie
}

### Function ucs_logout
### Required variables: $url = UCS url $inCookie = Cookie for session to logout
### Returns: Status of logout
### Todo: Error Checking
function ucs_logout($url, $inCookie) {
$aaaLogout = “<aaaLogout inCookie='” + $inCookie + “‘ />”
$xml = ucs_post $url $aaaLogout
$outStatus = $xml.aaaLogout.outStatus
return $outStatus
}

### Function get_esx_hosts
### Required variables: $vCenter = $vCenter server object
### Returns: ESX hosts in vCenter
### ToDo: Error checking. More logic
function get_esx_hosts($vCenter) {
$esxhosts = @()
$VMHosts = Get-VMhost -server $vCenter
foreach ($h in $VMHosts) {
$esxhost = “” | Select-Object Name, Uuid, IpmiIp, IpmiMac
$esxhost.Name = $h.name
$v = Get-VMHost -Name $h | Get-View
$esxhost.Uuid = $v.Summary.Hardware.Uuid
$esxhosts += $esxhost
}
return $esxhosts
}

### Function get_blade_dn
### Required variables: $uuid = ESX UUID $url = UCS url $cookie = UCS cookie for session
### Returns: DN of physical blade
### Todo: Error Checking
function get_blade_dn($uuid, $url, $cookie) {
$computeBlade = “<configResolveClass cookie='” + $cookie + “‘ inHierarchical=’false’ classId=’computeBlade’><inFilter><eq class=’computeBlade’ property=’uuid’ value='” + $uuid + “‘ /></inFilter> </configResolveClass>”
$bladeXml = ucs_post $url $computeBlade
return $bladeXml.configResolveClass.outConfigs.computeBlade.dn
}

### Function get_blade_ipmi
### Required variables: $dn = DN of physical blade $url = UCS url $cookie = UCS cookie for session
### Returns: Management Interface XML response from UCS
### Todo: Error Checking
function get_blade_ipmi($dn, $url, $cookie) {
$mgmtIf = “<configResolveClass cookie='” + $cookie + “‘ inHierarchical=’false’ classId=’mgmtIf’><inFilter><eq class=’mgmtIf’ property=’dn’ value='” + $dn + “/mgmt/if-1′ /></inFilter> </configResolveClass>”
$mgmtIfXml = ucs_post $url $mgmtIf
return $mgmtIfXml
}

### Function get_host_ipmi
### Required variables: $esxhost = ESX Host object $url = UCS url $cookie = UCS cookie for session
### Returns: Updated ESX host object
### Todo: Error checking
function get_host_ipmi($esxhost, $url, $cookie) {
$bladeDn = get_blade_dn $esxhost.Uuid $url $cookie
$mgmtIfXml = get_blade_ipmi $bladeDn $url $cookie
$esxhost.IpmiIp = $mgmtIfXml.configResolveClass.outConfigs.mgmtIf.extIp
$esxhost.IpmiMac = $mgmtIfXml.configResolveClass.outConfigs.mgmtIf.mac
return $esxhost
}

### Function set_host_ipmi
### Required variables: $esxhost = ESX host object $vCenter = vCenter Server Object
### Returns: nothing (should be changed)
### Todo: Error checking
function set_host_ipmi($esxhost, $vCenter) {
$v = Get-VMHost -server $vCenter -Name $esxhost.Name | % {Get-View $_.Id}
$ipmi = New-Object Vmware.Vim.HostIpmiInfo
$ipmi.BmcIpAddress = $esxhost.IpmiIp
$ipmi.BmcMacAddress = $esxhost.IpmiMac
$ipmi.Login = $ipmiUser
$ipmi.Password = $ipmiPassword
$v.UpdateIpmi($ipmi)
}

### Where the fun begins
### Lets log in to vCenter and UCS
$vCenter = Connect-VIServer -server $vCenterUrl -user $vCenterUser -password $vCenterPassword
$cookie = ucs_login $ucsUser $ucsPassword $ucsUrl

### Grabbing ESX hosts from vCenter
Write-Host “Getting ESX Hosts from vCenter”
$esxhosts = get_esx_hosts $vCenter

### Get the IPMI settings from UCS and update ESX hosts with information
Write-Host “Getting IPMI Settings from UCS and configuring ESX”
foreach ($h in $esxhosts) {
$h = get_host_ipmi $h $ucsUrl $cookie
set_host_ipmi $h $vCenter
}

### Fun as ended, time to log out.
Write-Host “Logging out of UCS”
$outStatus = ucs_logout $ucsUrl $cookie
Write-Host $outStatus
Write-Host “Logging out of vCenter”
Disconnect-VIServer -server $server -confirm:$false

And here is the PowerShell file.  UCS_VMware_IPMI.ps1.doc The .doc will need to be deleted from the end of the file (only way I could get the file uploaded).

Looks like I need to figure out a way to make code display on my blog. Another day.

Working with the Cisco UCS XML API

Working with Cisco UCS systems on a daily basis, I decided to take some time this weekend to learn more about the XML API.  Honestly, it is a thing of beauty and simplicity.  Working with the Cisco UCS XML API is as simple as sending an HTTP POST with an XML document to the UCS Manager and getting the response.  That’s it.  Isn’t that great?

Seeing as I have a couple of projects that will benefit from interacting with UCS systems, I decided to write a script in PHP to login and logout of a UCS for starters.  Then Rob Markovic asked, “Why not PowerShell?”.  So I wrote the same script in PowerShell as well.  I do need to give credit to Steve Chambers for this post: Access UCS API with Ruby.  This was beneficial as I was reading the XML API documentation and starting to work on my scripts.

***Disclaimer***

I am not an expert in PHP or PowerShell.  I am providing this information for academic purposes only to show how easy it is to work with the UCS XML API.  I am also not responsible for any damage this may cause your systems.

***Disclaimer***

One thing you’ll notice with the Cisco UCS XML API is that the only methods that require a username and password to be passed is the aaaLogin and aaaRefresh methods.  All other methods require a cookie to be passed, which is only obtained by logging in or refreshing your session to the UCS Manager.

Let’s start with what’s needed for PHP to work with the Cisco XML API.  I downloaded and installed HTTP_Request2 from PEAR to make the HTTP call easier.

require_once HTTP/Request2.php;

Now we need to set the url of our server and the XML parameters for logging into the UCS system.

$url = “http://server ip or dns/nuova”;

$aaaLogin = “<aaaLogin inName=’username’ inPassword=’password’ />”;

With the url and XML set, we can now create our HTTP request for logging in.

$request = new HTTP_REQUEST2($url, HTTP_REQUEST2::METHOD_POST);

$request->setHeader(“Content-type: text/xml”);

$request->setBody($aaaLogin);

After the request is created, we can send the data to the UCS Manager and get our response.

$response = $request->send();

To make it easier to work with the response, we are going to create SimpleXMLElement from the response.

$xml = new SimpleXMLElement($response->getBody());

Now to get our cookie from the XML response, we just need to do the following:

$outCookie = $xml->attributes()->outCookie

With the cookie now obtained, we are free to call other UCS XML API methods for interacting with the system.  To logout, we need to create the logout xml and issue an HTTP POST with this data.  (I will not repeat the code for issuing the HTTP POST).  Here is the XML for logging out of the system.

$aaaLogout = “<aaaLogout inCookie=’cookie for session’ />”;

To verify status of logout, you look at the outStatus attribute from the XML response.

$outStatus = $xml->attributes()->outStatus;

Here is how to do the same with PowerShell.

$url = “http://server ip or dns/nuova”

$aaaLogin = “<aaaLogin inName=’username’ inPassword=’password’ />

$request = [System.Net.HttpWebRequest] [System.Net.HttpWebRequest]::Create($url)

$request.Method = “POST”

$request.ContentType = “text/xml”

$sendData = new-object System.IO.StreamWriter($request.GetRequestStream())

$sendData.Write($aaaLogin)

$sendData.Close()

$response = $request.GetResponse()

$sr = new-object System.IO.StreamReader($response.GetReponseStream())

$xml = [xml] $sr.ReadToEnd()

$outCookie = $xml.aaaLogin.outCookie

To logout with PowerShell, we create the aaaLogout XML with the cookie we just obtained and issue another HTTP Request (code not repeated).

$aaaLogout =”<aaaLogout inCookie=’cookie for session’ />”

And to get the status of the logout.

$outstatus = $xml->aaaLogout->outStatus

There you have it.  PHP and PowerShell example code for logging into and out of Cisco UCS Manager.  You can find the programmer’s guide here:  Cisco UCS Manager XML API Programmer’s Guide.  I would also suggest checking out these Cisco blog posts on the Cisco UCS Manager XML API by John McDonough: UCS XML API “Hello World”, UCS XML API “curl and xml”, and UCS XML API Query Methods.

Upgrading a UCS

Today I had the opportunity to upgrade a UCS from 1.2.1b to 1.3.1c.  I had watched it done before, and it is straight forward process.  However, there is one thing to pay careful attention to.

When reading the upgrade documentation, you will notice there is one piece of hardware that can be upgraded either at the beginning of the process or at the end.  This piece of hardware is the adapters in the blades.  If a firmware policy is not applied to the service profile, then the adapter can be updated at the beginning of the process.  If a firmware policy is applied, then the adapters will be upgraded as part of applying the new firmware policy.

So what is the difference??

When you upgrade the adapters at the beginning, you are given the option to activate the firmware on next boot of the blade.  This is helpful as there is no disruption of network or SAN traffic at this time.  However, updating the adapter through firmware policy is a different story.  When you apply the firmware policy to the blade, the new adapter firmware is written to the blade, and THEN activated, causing a disruption of network and SAN traffic.  When you are booting from SAN, this can cause an interesting situation.

So what to do?

There are two ways to handle adapter upgrades when using firmware policies.  Either remove the firmware policy from the blades before starting so the adapters can be upgraded first, or shutdown your servers before applying the firmware policy.  Both methods work; it’s just really a matter of preference.