Geaux Virtual

Helping virtualize the datacenter…

Archive for the ‘VMware’ Category

Reconnecting ESX(i) hosts with 1000V installed

leave a comment »

I follow a certain policy: “You break it; you fix it”.  Why?  When you break something, you learn a lot about why it broke and what it takes to fix it.  Hopefully, you also learn how to prevent it from happening again.  Two, I’m not stuck dealing with the problem someone else caused.

Tonight was one where I broke something, and now I needed to fix it.

I had to reconfigure a UCS system running 3 vSphere 4.1 ESXi hosts connected to a Cisco 1000V.  Before powering down the hosts, I made some changes to the 1000v and the UCS.  So far, so good.  All the hosts were powered down along with vCenter and the 1000V VSMs, which were running on the UCS chassis.  I made my configuration changes on the UCS and powered up the first host.  No connectivity.  Something was not jiving between the 1000V config on the host itself and the vNIC configuration from the UCS.  It turned out to be a misconfiguration of the native vlan on the vNIC presented to the host and the native vlan configuration configured on the host by the 1000V.  I returned some values to their previous settings, and the host came back up.  I updated the 1000V config, which took the host back down, but I adjusted the UCS to the new configuration I wanted to bring the host back up.  Great, except I still had to bring two more hosts back online.

This is where I dove into vemcmd on the ESXi host.  When I brought up the next host, I opened a local console to the system.  “vemcmd show port” showed me the native vlan configuration error that was causing my issue.  So how do you fix this?  It’s actually quite simple.  When looking at the output of the previous command, you’ll notice three Trunk ports on the system.  Two are the actual physical uplinks connected from the  system.  The third is the port channel that is formed between the two nics.  To bring the system back online, you need to issue the following command: vemcmd set port-mode trunk native-vlan <native vlan> ltl <ltl of port-channel>.  After issuing this command, the system was back online.  You’ll notice that after issuing the command, the native vlan of the physical nics remains the same.  After the VEM gets the updated configuration, the native vlan is now the correct on the physical nics.

Written by jguidroz

September 28, 2010 at 7:15 pm

Posted in Networking, UCS, VMware

Working with UCS and vCenter: IPMI Settings

with 5 comments

In my last blog post, Working with Cisco UCS XML API, I went through the steps needed to login and logout of UCS Manager using PHP and PowerShell.  Today, I decided to write the first PowerShell script I needed which would automatically update the IPMI settings on the ESX hosts with the settings from UCS Manager.  In writing this script, I did come across this post from Mike Laverick on using PowerShell to enable DPM settings

Now, this script is important for two reasons.  The first reason is that to configure IPMI on ESX hosts, you need both IP address and the MAC address of the CIMC controller from the UCS blades.  When having to do this for multiple ESX hosts, a script is very handy.  Now the second reason is more important.  Currently in UCS, the CIMC IP is tied to the actual blade and not the service profile.  This means if a service profile is associated with a new blade, the ESX host IPMI information is now pointing to the old blade.  Not good!

With that said, here is the first version of this script as a PowerShell script.  (This will also come as a PHP script and possibly other languages in the near future.)  In order to run this script, you will need PowerCLI installed.  You will also need IP and credentials for vCenter and UCS, as well as the IPMI user and password.  This script has only been tested against UCS 1.3.1 and vSphere 4.1.  I will be testing this shortly against vSphere 4.0.  This is also a v0.1 script, so there is no error checking (who needs it right?).  That will be coming in a future version.

###DISCLAIMER###

I provide this script for academic purposes only.  Use at your own risk.  I am not responsible for any damages it may cause.

###DISCLAIMER###

So what does the script do?  The script first logs into vCenter and UCS Manager.  It then grabs the UUID from each ESX host.  Once done, it will pass this UUID to UCS to find the corresponding blade and return the CIMC IP and MAC for that blade.  The script will then write these settings to the IPMI of the ESX host.  Finally, it will log out of the UCS and vCenter.

#####################################################################
# UCS_VMware_IMPI.ps1 v0.01  By: Justin Guidroz
#
# This script will connect to UCS Manager, collect IPMI information
# for the provisioned blades, and update the appropriate IPMI
# information on the ESX hosts.  The script can also be run to
# update the IPMI information if a service profile is moved to a
# different blade.
#
#####################################################################
Add-PSSnapin VMware.VimAutomation.Core

### Needed variables
$ucsUrl = “”
$ucsUser = “”
$ucsPassword = “”
$vCenterUrl = “”
$vCenterUser = “”
$vCenterPassword = “”
$ipmiUser = “”
$ipmiPassword = “”

### Function ucs_post
### Required variables: $url = UCS Url $data = XML to send to UCS
### Returns: XML response from UCS
function ucs_post($url,$data) {
$request = [System.Net.HttpWebRequest] [System.Net.HttpWebRequest]::Create(“http://&#8221; + $url +”/nuova”)
$request.Method = “POST”
$request.ContentType = “text/xml”
$sendData = new-object System.IO.StreamWriter($request.GetRequestStream())
$sendData.Write($data)
$sendData.Close()
$response = $request.GetResponse()
$sr = new-object System.IO.StreamReader($response.GetResponseStream())
$xml = [xml] $sr.ReadToEnd()
return $xml
}

### Function ucs_login
### Required variables: $inName = UCS username $inPassword = UCS password $url = UCS url
### Returns: Cookie after login
### Todo: Error Checking
function ucs_login($inName, $inPassword, $url) {
$aaaLogin = “<aaaLogin inName='” + $inName + “‘ inPassword='” + $inPassword + “‘ />”
$xml = ucs_post $url $aaaLogin
$outCookie = $xml.aaaLogin.outCookie
return $outCookie
}

### Function ucs_logout
### Required variables: $url = UCS url $inCookie = Cookie for session to logout
### Returns: Status of logout
### Todo: Error Checking
function ucs_logout($url, $inCookie) {
$aaaLogout = “<aaaLogout inCookie='” + $inCookie + “‘ />”
$xml = ucs_post $url $aaaLogout
$outStatus = $xml.aaaLogout.outStatus
return $outStatus
}

### Function get_esx_hosts
### Required variables: $vCenter = $vCenter server object
### Returns: ESX hosts in vCenter
### ToDo: Error checking. More logic
function get_esx_hosts($vCenter) {
$esxhosts = @()
$VMHosts = Get-VMhost -server $vCenter
foreach ($h in $VMHosts) {
$esxhost = “” | Select-Object Name, Uuid, IpmiIp, IpmiMac
$esxhost.Name = $h.name
$v = Get-VMHost -Name $h | Get-View
$esxhost.Uuid = $v.Summary.Hardware.Uuid
$esxhosts += $esxhost
}
return $esxhosts
}

### Function get_blade_dn
### Required variables: $uuid = ESX UUID $url = UCS url $cookie = UCS cookie for session
### Returns: DN of physical blade
### Todo: Error Checking
function get_blade_dn($uuid, $url, $cookie) {
$computeBlade = “<configResolveClass cookie='” + $cookie + “‘ inHierarchical=’false’ classId=’computeBlade’><inFilter><eq class=’computeBlade’ property=’uuid’ value='” + $uuid + “‘ /></inFilter> </configResolveClass>”
$bladeXml = ucs_post $url $computeBlade
return $bladeXml.configResolveClass.outConfigs.computeBlade.dn
}

### Function get_blade_ipmi
### Required variables: $dn = DN of physical blade $url = UCS url $cookie = UCS cookie for session
### Returns: Management Interface XML response from UCS
### Todo: Error Checking
function get_blade_ipmi($dn, $url, $cookie) {
$mgmtIf = “<configResolveClass cookie='” + $cookie + “‘ inHierarchical=’false’ classId=’mgmtIf’><inFilter><eq class=’mgmtIf’ property=’dn’ value='” + $dn + “/mgmt/if-1′ /></inFilter> </configResolveClass>”
$mgmtIfXml = ucs_post $url $mgmtIf
return $mgmtIfXml
}

### Function get_host_ipmi
### Required variables: $esxhost = ESX Host object $url = UCS url $cookie = UCS cookie for session
### Returns: Updated ESX host object
### Todo: Error checking
function get_host_ipmi($esxhost, $url, $cookie) {
$bladeDn = get_blade_dn $esxhost.Uuid $url $cookie
$mgmtIfXml = get_blade_ipmi $bladeDn $url $cookie
$esxhost.IpmiIp = $mgmtIfXml.configResolveClass.outConfigs.mgmtIf.extIp
$esxhost.IpmiMac = $mgmtIfXml.configResolveClass.outConfigs.mgmtIf.mac
return $esxhost
}

### Function set_host_ipmi
### Required variables: $esxhost = ESX host object $vCenter = vCenter Server Object
### Returns: nothing (should be changed)
### Todo: Error checking
function set_host_ipmi($esxhost, $vCenter) {
$v = Get-VMHost -server $vCenter -Name $esxhost.Name | % {Get-View $_.Id}
$ipmi = New-Object Vmware.Vim.HostIpmiInfo
$ipmi.BmcIpAddress = $esxhost.IpmiIp
$ipmi.BmcMacAddress = $esxhost.IpmiMac
$ipmi.Login = $ipmiUser
$ipmi.Password = $ipmiPassword
$v.UpdateIpmi($ipmi)
}

### Where the fun begins
### Lets log in to vCenter and UCS
$vCenter = Connect-VIServer -server $vCenterUrl -user $vCenterUser -password $vCenterPassword
$cookie = ucs_login $ucsUser $ucsPassword $ucsUrl

### Grabbing ESX hosts from vCenter
Write-Host “Getting ESX Hosts from vCenter”
$esxhosts = get_esx_hosts $vCenter

### Get the IPMI settings from UCS and update ESX hosts with information
Write-Host “Getting IPMI Settings from UCS and configuring ESX”
foreach ($h in $esxhosts) {
$h = get_host_ipmi $h $ucsUrl $cookie
set_host_ipmi $h $vCenter
}

### Fun as ended, time to log out.
Write-Host “Logging out of UCS”
$outStatus = ucs_logout $ucsUrl $cookie
Write-Host $outStatus
Write-Host “Logging out of vCenter”
Disconnect-VIServer -server $server -confirm:$false

And here is the PowerShell file.  UCS_VMware_IPMI.ps1.doc The .doc will need to be deleted from the end of the file (only way I could get the file uploaded).

Looks like I need to figure out a way to make code display on my blog. Another day.

Written by jguidroz

September 20, 2010 at 6:37 pm

Posted in Scripting, UCS, VMware

Automating ESXi 4.1 continuation

with 4 comments

William Lam has posted a great article on tips and tricks of automating ESXi 4.1 here.  Go read his post first.

Seriously, go read his post first.

Ok, you’re back?

I wanted to add to his post with some issues and fixes I’ve run across with moving from ESX to ESXi automation.

With ESX 4.0, at the beginning of the ks.cfg file, you would specify an install line.  Nothing else was needed as the url location of the install bits was passed in the PXE config line.  No more.  For ESX(i) 4.1, you must specify the url for the install location in the ks.cfg file.

Beginning with ESX(i) 4.1, ip append 2 in the PXE config boot line no longer works.  What does this do exactly?  This told the ESX(i) to use the same nic it booted from for downloading the install bits.  If this was not specified, ESX(i) would try to use the first nic it detected, vmnic0.  With ESX(i) 4.1, you must specify ksdevice=vmnic(x) if you are booting and installing from a different nic than vmnic0.  I’m not sure why this behavior changed between 4.0 and 4.1.

William did mention vim-cmd.  In ESX, the command is actually vmware-vim-cmd.  For ESXi, be sure to update these commands to vim-cmd.

For disk setup, I have two lines in my ks.cfg file: clearpart –onfirstdisk –overwritevmfs and autopart –onfirstdisk –overwritevmfs.  I did notice this will create the datastore name on each ESXi host as datastore.  I need to investigate if this can be changed at this part, or add a line in the script to update that name.

In my working with scripting ESXi installations, I’ve found it much easier to have everything run after firstboot then trying to run the configurations before firstboot.

Plenty of advanced configurations (SNMP, Active Directory, adding to vCenter) can be handled through the scripted install, but involve writing a python script to handle.  For some, it will be easier using the supplied commands in the vMA or writing PowerShell scripts.

Outside of these differences, scripted ESXi installs are awesome.

Written by jguidroz

September 11, 2010 at 12:07 pm

Posted in Scripting, VMware

Automatically add ESX(i) to vCenter from ESX(i)

with 3 comments

I have to say, I love my job.  Since starting at the end of May, I have had the opportunity to work with some great people and tackle tasks that I have been wanting to complete yet never had the time to do at previous employers.  Specifically, I am talking about automating ESX(i) deployments.  I have performed many scripted installs before (in fact I had not done a full load of ESX for over 3 years and only recently did so because CNA drivers were not in the ISO), but now I am in the situation where build after build must be exactly the same no matter who is performing the builds.  With that, our team started looking at the ESX Deployment Appliance (great appliance BTW), but I quickly noticed some lacking features which I quickly added to fit our current needs.  (Keep your eyes out for announcement in the near future on a collaboration myself and @lynxbat will put together regarding a deployment appliance).

With the work I’ve been doing lately, I asked myself the question: why couldn’t an ESX(i) host add itself to vCenter?  The easiest answer to this question is that VMware has not written a program or script to perform this task.  But is their a technical reason why this would not be possible?

In order to perform any type of action on vCenter, API access is required.  There are two ways to access the vCenter APIs: MOB and Web Services API.  The MOB, or Managed Object Browser, is a web site that allows retrieving and setting values for vCenter.  Traversing the MOB is not easy and requires frequent trips to the vSphere API documentation for assistance.  The Web Services API is a web service that can be used to retrieve and set values via a SOAP client.  VMware even provides SDKs for Java and Perl.

Do we have an SDK available on ESX(i)? No.  This means we must see if we can access the MOB or Web Services API from ESX(i) by writing a script that does not rely on an SDK.  Looking through an ESXi host, I noticed python was on the host.  Why is python on the host?  The MOB is written in python.  There is no SOAP client libs loaded for python on ESX(i)(if there are, please post a comment), and this solution should not require loading of additional libraries to ESX(i).  With that I set out and wrote a script that will connect an ESX(i) host to vCenter.

***DISCLAIMER***

Use at your own risk.  I provide this script as an academic example of how to do this.  I am not responsible if it does havoc on your environment.  This script has only been tested with vSphere 4.1 ESXi and vCenter.  ESX 4.1, ESX 4.0, and vCenter 4.0  have not been tested.

***DISCLAIMER***

First we must import the libraries we need.

import re,os,urllib,urllib2

Next, let’s set some variables that will be needed.  This is where it starts to get interesting.  <CLUSTER> below needs to be replaced with the cluster the host will be added to. For this exercise, this is a static assignment. This name will be in the form of domain-c21 or something similar and can be found in the MOB.

url = "https://vcenteraddress/mob/?moid=<CLUSTER>&method=addHost"
username = "vcenterusername"
password = "vcenterpassword"

This section configures the authentication for when we connect to the MOB.

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)

This next section computes the SHA1 hash for the host and strips out all unnecessary characters.  As you can tell, my regex skills are a bit rusty, and this probably could be cleaned up.

cmd = "openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout -fingerprint"
tmp = os.popen(cmd)
tmp_sha1 = tmp.readline()
tmp.close()
s1 = re.split('=',tmp_sha1)
s2 = s1[1]
s3 = re.split('\n', s2)
sha1 = s3[0]

This next section creates the ConnectHostSpec information that needs to be passed to the addHost method.

xml = """<spec xsi:type="HostConnectSpec">
<hostName>192.168.188.130</hostName>
<sslThumbprint>%s</sslThumbprint>
<userName>root</userName>
<password>rootpassword</password>
<force>1</force>
</spec>"""
xml = xml.replace("%s",sha1)

Once the ConnectHostSpec is created, we can now encode the parameters that must be sent with the POST to the addHost method.  As you can see, besides the ConnectHostSpec, we must also pass values for asConnected, resoursePool, and license.  Only spec and asConnected require a value.  After the values are encoded, a request can be created and the data sent to the URL.

params = {'spec':xml,'asConnected':'1','resourcePool':'','license':''}
e_params = urllib.urlencode(params)
req = urllib2.Request(url,e_params)
page = urllib2.urlopen(req).read()

And that’s it. This small script will take the parameters needed to add an ESX(i) host to vCenter through the MOB and complete this task from the host itself.  I am currently testing integrating this script with scripted installs.  Stay tuned as I am already getting more ideas for this script.

Written by jguidroz

July 25, 2010 at 3:57 pm

Posted in Scripting, VMware

Rapid ESX Deployments

with one comment

At the end of this past week, we were actually working on installing ESX to our UCS blades for a stateless environment.  When you have 16 blades to install, the last thing you want to do is build each one through the ESX gui.

Enter the ESX Deployment Appliance, or EDA for short.  (You can read up about EDA here http://www.vmware.com/appliances/directory/va/89313 )

As great as version 0.90 is, I quickly ran into two three issues.

Issue 1: EDA does not allow assigning VLAN to Service Console

For our deployments, the Service Console port of all our ESX hosts reside on a separate VLAN from the rest of the network.  When installing through the gui, the ESX installer provides a way to assign the Service Console port to a VLAN.  In the current version of EDA, the only way to accomplish this is by adding the VLAN to the Service Console portgroup through the POST section of the KS.  However, the ESX host will not have network connectivity until you first go ping the gateway from the console.  It works, but not convenient to rapid deployments.

Issue 2: Naming of local ESX VMFS volume

Typically when I build an ESX host, I name the local VMFS volume with the host name of the ESX host.  This allows for the local VMFS volumes of all ESX hosts to be easily recognized in vCenter.  This is currently not an option in 0.90 of EDA.

Issue 3: Removing default VM Portgroup during ESX installation

This is a very minor issue.  If we were using standard vSwitches, then I would probably be fine with this as I’m starting to move to the one vSwitch rules them all camp.  However for these deployments, we are utilizing the Cisco Nexus 1000v, so a local VM portgroup is not needed.

Since these options were crucial for our rapid deployments, I decided to start hacking away on EDA on my flight back from Boston Friday.  Fixing these issues (or adding these features) didn’t take long at all, and I’m currently testing out the modified appliance.  For those wondering, I will not be distributing the modified appliance.  If there is enough interest, I’ll write up a blog post on the exact changes that need to be made to these fixes (or features).

Written by jguidroz

June 27, 2010 at 8:52 pm

Posted in VMware

The next chapter in my career

with one comment

I do have to say, my voluntary vacation the past 6 weeks has been great.  I did not travel anywhere exotic, or actually do much of anything.  It was just great having downtime away from work.

Now it’s time to focus on the next chapter of my professional career, which starts on May 24th.  I will be joining ACADIA as a Network Architect.  I am looking forward to the opportunities and challenges that will be presented in this position and company.

I will keep this update short and head off to today’s RAoN: Running.

Written by jguidroz

May 13, 2010 at 2:28 pm

Posted in Networking, Random, VMware

And a new era begins

leave a comment »

Six months ago I decided to make career and life change and move to Fort Worth for a new job.  It was a fantastic opportunity at a small company.  I never envisioned being at the company for longer than two years, but I figured the opportunities for career growth were better.  Surprisingly, I also did not imagine my time at the company to be so short lived.  It is actually quite unfortunate, since I worked with a lot of a great of people.

There always comes a time where you have to ask yourself if you can accept and deal with the changes, or is it better to find another opportunity?

I had been fighting this for the past couple of months, and today I made the decision, for both personal and professional reasons, to move on.  I had made similar decisions before, but always because I was moving to a new employer for a better opportunity.  This time is different.  I am just moving on.  And surprisingly, I am very relaxed and happy with the decision.

So what is next for me?

The last two projects I really enjoyed were designing and architecting the server refresh at my previous company, and then the implementation of the 100% virtualized data center at my now former company.  Both revolved around VMware vSphere, Cisco Nexus 5000 switches, and EMC storage.  I would like to pursue a position around data center design and architecture using the same or similar technologies.  Along a similar path, I also have aspirations for attaining CCIE and VCDX certifications.

And that’s a wrap.  I am off to enjoy the rest of my day.

Written by jguidroz

March 29, 2010 at 2:06 pm

Posted in Networking, Random, VMware