Automatically add ESX(i) to vCenter from ESX(i)

I have to say, I love my job.  Since starting at the end of May, I have had the opportunity to work with some great people and tackle tasks that I have been wanting to complete yet never had the time to do at previous employers.  Specifically, I am talking about automating ESX(i) deployments.  I have performed many scripted installs before (in fact I had not done a full load of ESX for over 3 years and only recently did so because CNA drivers were not in the ISO), but now I am in the situation where build after build must be exactly the same no matter who is performing the builds.  With that, our team started looking at the ESX Deployment Appliance (great appliance BTW), but I quickly noticed some lacking features which I quickly added to fit our current needs.  (Keep your eyes out for announcement in the near future on a collaboration myself and @lynxbat will put together regarding a deployment appliance).

With the work I’ve been doing lately, I asked myself the question: why couldn’t an ESX(i) host add itself to vCenter?  The easiest answer to this question is that VMware has not written a program or script to perform this task.  But is their a technical reason why this would not be possible?

In order to perform any type of action on vCenter, API access is required.  There are two ways to access the vCenter APIs: MOB and Web Services API.  The MOB, or Managed Object Browser, is a web site that allows retrieving and setting values for vCenter.  Traversing the MOB is not easy and requires frequent trips to the vSphere API documentation for assistance.  The Web Services API is a web service that can be used to retrieve and set values via a SOAP client.  VMware even provides SDKs for Java and Perl.

Do we have an SDK available on ESX(i)? No.  This means we must see if we can access the MOB or Web Services API from ESX(i) by writing a script that does not rely on an SDK.  Looking through an ESXi host, I noticed python was on the host.  Why is python on the host?  The MOB is written in python.  There is no SOAP client libs loaded for python on ESX(i)(if there are, please post a comment), and this solution should not require loading of additional libraries to ESX(i).  With that I set out and wrote a script that will connect an ESX(i) host to vCenter.

***DISCLAIMER***

Use at your own risk.  I provide this script as an academic example of how to do this.  I am not responsible if it does havoc on your environment.  This script has only been tested with vSphere 4.1 ESXi and vCenter.  ESX 4.1, ESX 4.0, and vCenter 4.0  have not been tested.

***DISCLAIMER***

First we must import the libraries we need.

import re,os,urllib,urllib2

Next, let’s set some variables that will be needed.  This is where it starts to get interesting.  <CLUSTER> below needs to be replaced with the cluster the host will be added to. For this exercise, this is a static assignment. This name will be in the form of domain-c21 or something similar and can be found in the MOB.

url = "https://vcenteraddress/mob/?moid=<CLUSTER>&method=addHost"
username = "vcenterusername"
password = "vcenterpassword"

This section configures the authentication for when we connect to the MOB.

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)

This next section computes the SHA1 hash for the host and strips out all unnecessary characters.  As you can tell, my regex skills are a bit rusty, and this probably could be cleaned up.

cmd = "openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout -fingerprint"
tmp = os.popen(cmd)
tmp_sha1 = tmp.readline()
tmp.close()
s1 = re.split('=',tmp_sha1)
s2 = s1[1]
s3 = re.split('\n', s2)
sha1 = s3[0]

This next section creates the ConnectHostSpec information that needs to be passed to the addHost method.

xml = """<spec xsi:type="HostConnectSpec">
<hostName>192.168.188.130</hostName>
<sslThumbprint>%s</sslThumbprint>
<userName>root</userName>
<password>rootpassword</password>
<force>1</force>
</spec>"""
xml = xml.replace("%s",sha1)

Once the ConnectHostSpec is created, we can now encode the parameters that must be sent with the POST to the addHost method.  As you can see, besides the ConnectHostSpec, we must also pass values for asConnected, resoursePool, and license.  Only spec and asConnected require a value.  After the values are encoded, a request can be created and the data sent to the URL.

params = {'spec':xml,'asConnected':'1','resourcePool':'','license':''}
e_params = urllib.urlencode(params)
req = urllib2.Request(url,e_params)
page = urllib2.urlopen(req).read()

And that’s it. This small script will take the parameters needed to add an ESX(i) host to vCenter through the MOB and complete this task from the host itself.  I am currently testing integrating this script with scripted installs.  Stay tuned as I am already getting more ideas for this script.

Advertisement

Rapid ESX Deployments

At the end of this past week, we were actually working on installing ESX to our UCS blades for a stateless environment.  When you have 16 blades to install, the last thing you want to do is build each one through the ESX gui.

Enter the ESX Deployment Appliance, or EDA for short.  (You can read up about EDA here http://www.vmware.com/appliances/directory/va/89313 )

As great as version 0.90 is, I quickly ran into two three issues.

Issue 1: EDA does not allow assigning VLAN to Service Console

For our deployments, the Service Console port of all our ESX hosts reside on a separate VLAN from the rest of the network.  When installing through the gui, the ESX installer provides a way to assign the Service Console port to a VLAN.  In the current version of EDA, the only way to accomplish this is by adding the VLAN to the Service Console portgroup through the POST section of the KS.  However, the ESX host will not have network connectivity until you first go ping the gateway from the console.  It works, but not convenient to rapid deployments.

Issue 2: Naming of local ESX VMFS volume

Typically when I build an ESX host, I name the local VMFS volume with the host name of the ESX host.  This allows for the local VMFS volumes of all ESX hosts to be easily recognized in vCenter.  This is currently not an option in 0.90 of EDA.

Issue 3: Removing default VM Portgroup during ESX installation

This is a very minor issue.  If we were using standard vSwitches, then I would probably be fine with this as I’m starting to move to the one vSwitch rules them all camp.  However for these deployments, we are utilizing the Cisco Nexus 1000v, so a local VM portgroup is not needed.

Since these options were crucial for our rapid deployments, I decided to start hacking away on EDA on my flight back from Boston Friday.  Fixing these issues (or adding these features) didn’t take long at all, and I’m currently testing out the modified appliance.  For those wondering, I will not be distributing the modified appliance.  If there is enough interest, I’ll write up a blog post on the exact changes that need to be made to these fixes (or features).

The next chapter in my career

I do have to say, my voluntary vacation the past 6 weeks has been great.  I did not travel anywhere exotic, or actually do much of anything.  It was just great having downtime away from work.

Now it’s time to focus on the next chapter of my professional career, which starts on May 24th.  I will be joining ACADIA as a Network Architect.  I am looking forward to the opportunities and challenges that will be presented in this position and company.

I will keep this update short and head off to today’s RAoN: Running.

And a new era begins

Six months ago I decided to make career and life change and move to Fort Worth for a new job.  It was a fantastic opportunity at a small company.  I never envisioned being at the company for longer than two years, but I figured the opportunities for career growth were better.  Surprisingly, I also did not imagine my time at the company to be so short lived.  It is actually quite unfortunate, since I worked with a lot of a great of people.

There always comes a time where you have to ask yourself if you can accept and deal with the changes, or is it better to find another opportunity?

I had been fighting this for the past couple of months, and today I made the decision, for both personal and professional reasons, to move on.  I had made similar decisions before, but always because I was moving to a new employer for a better opportunity.  This time is different.  I am just moving on.  And surprisingly, I am very relaxed and happy with the decision.

So what is next for me?

The last two projects I really enjoyed were designing and architecting the server refresh at my previous company, and then the implementation of the 100% virtualized data center at my now former company.  Both revolved around VMware vSphere, Cisco Nexus 5000 switches, and EMC storage.  I would like to pursue a position around data center design and architecture using the same or similar technologies.  Along a similar path, I also have aspirations for attaining CCIE and VCDX certifications.

And that’s a wrap.  I am off to enjoy the rest of my day.

Lab Manager Tip

When deploying a large configuration to a new workspace that requires changing the network and possibly resetting the MAC of the assigned NIC, it is beneficial to utilize the Configuration Diagram afterwards to verify all VMs in the configuration are connected to the correct network.

The Configuration Diagram tab is at the top of the current open configuration.

Where is it? It’s got to be here, it just has to be…

I’ve been thinking about this for a while, and since this is a new year, here is a list of 9 features or changes I would like see.

In no particular order:

9.  When doing a manual VMotion, maintain the current resource pool of the VM.  Only ask for a resource pool if I move the VM to a new cluster. (VMware)

8. Ability to configure one default alarm action to apply to some or all default alarms. (VMware)

7.  Utilize Cisco Nexus 5000 switches as VSMs for Cisco Nexus 1000v deployments. (Cisco)

6.  PowerPath support for QLogic QLE8100 series cards. (EMC; Yes, I know it’s coming, I’m just impatient)

5.  Native Mac VMware vSphere client. (VMware; I’m really reaching with this one)

4.  Removal of PowerPath License Server requirement (EMC)

3.  Port Profiles added to the Cisco Nexus 5000 switches (Cisco)

2.  vCenter Virtual Appliance (VMware)

And last, but probably the most important…..

1.  Ability to migrate templates (VMware)

Drops mic and walks off stage….

Explanation makes no sense

Today I’m updating my vSphere environment in our new datacenter with the latest VMware updates that were released yesterday.  This is my first real test of VUM 4.0 U1 as well.  I had 5 patches to install, so I figured I’d stage the patches first on the hosts, then remediate.  I started out one host at a time, but after two hosts, I decided to stage the 5 patches on all the hosts first, then go to each host and remediate.  This would also allow me to verify DRS is working properly in my cluster.

I selected the 8 remaining hosts in my cluster, and started a stage job to stage the 5 patches on each host.  The job completed, but the patches were not staged on one host.  No error was given either.  So, I tried staging the patches to the last host, and I received this error:

“The host has a VM with VMware vCenter Update Manager or VMware vCenter installed.  The VM needs to be moved to another host for the remediation to proceed.”

What is wrong with this error?

First, I was able to successfully stage the patches to the host where my VMware vCenter VM is running.

Second, I am staging patches, not remediating the host.  If I was remediating the host, the host would be in maintenance mode, and the VM that caused this issue would be running on another host.

I think I will open up an SR with VMware and see what they have to say about this issue.

UPDATE

So it seems my vCenter server was migrated to a different host (host that errored).

And then you notice this…

Our recent Senior Infrastructure Engineer hire was remarking today how one of the LUNs on our EMC storage array was hitting 100% utilization in the middle of the night.  At first glance, everyone thought this was the LUN for our data warehouse.  After digging further, it turned out the LUN housed the development environments for our new platform.

So, what was occurring in the middle of the night for the LUN to see 100% utilization?

Disk defrag.

To get a better grasp of our development environment, we have one VM in Lab Manager that gets assigned to different users.  Each copy of this VM was trying to defrag it’s disk at the same exact time.  If you understand how Lab Manager and link clones work, you can understand why this was quickly disabled.

Ah, the joys of work.  Yes this blog post was short, and hopefully I’ll post more in the future about my recent tasks of setting up Nexus 5020 and Nexus 1000v, both awesome products.

Looking beyond Long Distance VMotion…

First post in a while, but this should be changing in the near future.

Back from VMworld 2009, there was a topic that interested me: Long Distance VMotion.  However, when looking at the technology behind it, it left me wanting more.

What more could I be looking for?  Datacenter VMotion.

How does this differ from Long Distance VMotion?  Datacenter VMotion would be a combination of Long Distance VMotion and migrating the storage live as well (Long Distance Storage VMotion?).  If storage is already replicated to the second location, it would just be a matter of syncing the changed blocks of the VM to the second location.  One click, all VMs (or at least critical VMs), migrate to the second location to complete the Datacenter VMotion.

Where’s my CPU Affinity??

We were having a discussion today over a vendor that said they currently do not support virtualizing their server application because of the “real-time clock” issues with virtualization, but they were working on this.  This led me to go find the VMworld 2008 presentation on Real-Time Applications.  The solution in the presentation was to set CPU Affinity for the VM.  Well, for the next 15 minutes or so, we went looking for CPU affinity with no luck in finding it.  Finally, we stumbled across it….

CPU affinity is hidden in the preferences for the VM when the VM is actively part of a DRS cluster.  For VMs in the DRS cluster with DRS disabled, CPU affinity can be found under the Advanced CPU selection in the resources tab.