Déjà vu

Flashback 3 years ago.

The phone rings with Nick Weaver on the line. He pitches an opportunity to me, and next thing I know, I’m on a plane to Dallas to interview for an opportunity that took me to the next step in my career.

A few weeks ago, I received a phone call from a person pitching not just the next step in my career but a career changing opportunity. Next thing I know, I’m on a plane to Dallas for an interview.

Déjà vu.

I have learned a good deal (and still learning) over the past 8 years when it comes to my career and potential opportunities. When these opportunities present themselves, I sit down and evaluate what I am currently doing, what I want to be doing, and where I want to be in the next year, five years, ten years down the road, and whether my current job or the new opportunity will be the better path for achieving my career goals. I look at the risks involved, and what affects changing jobs will have on my life (I will say this, I do not do this for every opportunity that comes to me, only for the opportunities that I feel present a big enough change from what I’m currently doing)

In 8 years, I have worked for 4 different companies. Each opportunity presented itself as the next step in my career, but also came with varying risks. Whether it was accepting a contract-to-hire position with a large company that was only created as part of a multi billion dollar expansion that had not been approved yet, or moving 500 miles to work for a small, private company, or moving to the vendor side of the industry with a “startup”, the risks were weighed against the reward, and as you can see, the reward has, so far, paid off (though I did learn a thing or two about relocating for a position).

As I thought about what was presented to me with this latest opportunity, I looked back through my career and realized this is something I have done on a small scale in each position I have worked. However, it was never anything I was able to work on for a full-time basis. I began to question if I was up for this next challenge. Could I handle this opportunity on a full-time basis? Did the reward outweigh the risks? I also realized this would not only be the next step in a career, but also a career changing opportunity. Success with this opportunity would open many more doors in future years.

I will be honest, I had my doubts. Am I really ready for this challenge?

In recent years, there is a personal motto I fall back to on tough decisions. It is “Do not be afraid to fail.” I believe if you are afraid to fail, you will never let yourself take that next step to greatness.

And with that, I have decided after two and half years to leave VCE and accept a roll at VMware as a Sr. Engineer – DevOps Automation in Cloud Infrastructure and Services. It will be a great challenge and opportunity to work alongside Nick Weaver and the other rockstars that are a part of this team. I leave a great group of people at VCE. This decision was about not only progressing my career, but also challenging myself.

“…life is about opportunities, challenges, and empty-time between. And when presented with the right challenge you have to make the choice to abandon the comfortable and take on the unknown.”

Well said Nick, well said.

Follow up on ESXCLI with PowerCLI

In a previous post here, Applying Patches to ESXi 5 with the help of PowerCLI, I showed how simple it was to call ESXCLI that is part of vCLI through a PowerCLI script, but did you know that ESXCLI is also part of PowerCLI?

ESXCLI is actually part of the VMware API for an ESXi host. Note: While the API for vSphere 4 also had ESXCLI included, it is much more robust with vSphere 5. Most if not all of the commands I show will only work against vSphere 5. PowerCLI has a cmdlet called Get-EsxCli that can be used when connected directly to an ESXi host.

First, connect to an ESXi host through PowerCLI.

Connect-VIServer

Next, lets run the Get-EsxCli cmdlet.

$esxcli = Get-EsxCli

Now we have our ESXCLI object. Now what can we do with it? Well basically anything that you would do with ESXCLI through vCLI or local to the ESXi host. So let’s say our host is running on a Cisco UCS system, and we wanted to verify the enic driver version loaded on the host. We could issue the following command to view the driver version.

$esxcli.system.module.get("enic").version

Now, to follow-up on the previous post, we could use the $esxcli object obtained through PowerCLI to issue the commands to view the vibs on the host and apply vibs to the host in the same manor that we were calling ESXCLI before. If we wanted to see all the VIBs installed on a host, we can issue the following command to give us the name, vendor, and version of all VIBs installed on the host in formatted table output.

$esxcli.software.vib.list() | ft Name,Vendor,Version

Let’s look at the example from the previous blog post. Here we set the patch name and location of the patch on the datastore that the host could access.

$patch = "PatchFile.zip" #patch file we want to apply
$datastore = "datastore1" #datastore where we want to store the patch
$remoteLocation = "/vmfs/volumes/" + $datastore + "/" + $patch #remote location of patch

Now once the patch is on the datastore, which I covered in the previous post, you could install it with the command

$esxcli.software.vib.install($remoteLocation)

Here is another tip as well for PowerShell and PowerCLI. If you are unsure of the arguments that a function takes, such as the command above, you can run

$esxcli.software.vib.install

and the command will output some useful data about the method. One particular useful output is Value. This tells you what the expected value type will be given the proper inputs. For the above command, you can see the output as follows:

Hopefully this proves useful to someone.

Interesting troubleshooting with Cisco Nexus 1000V

Today, I assisted a couple of coworkers with troubleshooting a VLAN issue. The VLAN had been added to all the appropriate devices, but the VM was still unable to ping the gateway. As our troubleshooting progressed from device to device, we noticed that although the uplink port profile on the Cisco Nexus 1000V was configured correctly, the VEM on the host was not updating with the changes. This could be seen by running

vemcmd show port vlans

on the host itself. As we dug through the running config of the Cisco Nexus 1000V, we noticed that besides the port channels inheriting the uplink port profile, a VLAN was manually added to all port channels. It turns out this is a simple fix (after many hours of troubleshooting). By issuing the command

default switchport trunk allowed vlan

to all the port channels, the port channels started inheriting the VLANs from the port profile again. We verified this by adding/removing a VLAN and the port profile updated as expected. We are not sure when this VLAN was added this way, but it did give us a bit of a headache.

So remember when working on the Cisco Nexus 1000V, add/remove VLANs via the port profiles, not from the port channels or ports themselves.

Working with Sinatra and Cloud Foundry

The other night I had idea pop in my head, that honestly, I was quite surprised hadn’t thought about before. I then set about working through the design of what would be needed to pull this idea off (sorry, the idea is under strict NDA that requires many pints of Guinness to even be able to sign).

Recently, I have started to move most of my work to Ruby, even working on another side project in Ruby that this project will be able to utilize. This project is a web based application, so with that, I needed a web/application server plus a platform to run it on. Now I have done my share of setting up Linux boxes with the required software before, but this time I wanted to see if the open source Cloud Foundry bits (VCAP) would prove useful in what I was looking to do. For the web/application server, I decided to go with Sinatra and Thin.

VCAP

This was not my first time setting up a VCAP server or deploying apps to one. For some reason, I decided to stand up yet another VCAP VM on my laptop instead of using an existing instance :/. Of course in this process, I ran into issues, such as corrupted vmc profiles and a rvm installation issue in the soon to be deprecated default install process. Using the experimental Chef based installation, I had no issues getting a VCAP VM up and running. One interesting point to note when using the default or Chef based installation off of github, the default domain is vcap.me. When you ping vcap.me, this resolves to 127.0.0.1 :). So how do you access the VM via DNS? Well you set up an SSH tunnel from your workstation to the VCAP VM (also explained in the installation instructions).

In order to deploy applications to your VCAP VM, you need to install VMC to your workstation. This is as simple as running gem install vmc. If you want the latest beta, add –pre behind that command like so gem install vmc --pre. With your SSH tunnel setup and VMC installed, now you can follow the “Trying your setup” section on the VCAP github page (VCAP).

Sinatra/Thin

Now with VCAP up and running, I focused on getting Sinatra and Thin working. Sinatra is actually quite simple, actually. First install the Sinatra gem:

gem install sinatra

and here is a simple web application

require 'sinatra'
get '/' do
  "Hello World"
end

And that’s it. When you run ruby app.rb (or whatever you named the file), Sinatra fires up webrick and starts serving your page. See? Simple.

So lets move on. Now there are two ways to run Sinatra apps. The first is classic mode, as shown above. The second is in modular mode in a file called server.rb, shown below:

require 'sinatra/base'

class Server < Sinatra::Base
  get '/' do
    "Hello World"
  end
end

Each mode of Sinatra application is also started differently. With the classic mode, you can start it with Sinatra::Application.run! However, in modular mode, Rack is used to start the server, and here in comes the fun of getting Sinatra/Thin working on Cloud Foundry VCAP VM. Classic mode works quite well on the VCAP VM, but modular mode requires a little bit more configuration.

First, you want to create a config.ru file in the root of your application. The location is key. By default, Rack looks for config.ru in the root of your application directory. There is a :config setting that can be changed, but for some reason when I set this key with a different config.ru location, the default port Thin starts on changed from 9292 to 8080. I have not figured out why this is occurring as nothing in my config.ru changed the Thin port; however, this will cause it to start on port 8080 in VCAP and not be accessible. The config.ru file needs to look like the following:

require 'sinatra'

set :run, false #This disables Sinatra trying to start Thin and let's Rack start Thin

require File.dirname(__FILE__) + "/path/to/server/file"

#The following is key. Sinatra Modular apps could
#be started with just run Server, but when deployed to
#VCAP, you need to map the application #to a location.
#In this instance, I want my server application
#to be mapped to the default path.
map '/' do
  run Server
end

Ok, now that our config.ru is configured (and of course more configurations could be added to it), we need to configure our main application file. For this, we need to use Rack to start our server. Lets say this file is called app.rb

require File.dirname(__FILE__) + "/path/to/server/file"
require 'rack'
require 'sinatra' #required for VCAP to detect this is a Sinatra Application
require 'thin'

server = ::Rack::Server.new()
server.start

Now before we push our application to our VCAP VM, we need to create a Gemfile that looks like the following:

source "http://rubygems.org"

gem 'sinatra'
gem 'thin'

You may wonder why Rack is not listed. Sinatra is dependent on Rack and will install the Rack gems when its installed. With the Gemfile created, we can run bundle package && bundle install to package up all the gems required by our application. If you require any more gems for your application, you will want to run bundle package && bundle install before you update your application.

With done, we need to push our application. We start this process with the command:

vmc push --runtime ruby19

If you are still using Ruby 1.8.7, do not add the –runtime ruby19. There is one more key step when pushing the application. VMC will ask for the “Application Deployment URL”. The default url it will show will be [APPLICATION NAME].vcap.me:[SSH TUNNEL PORT]. You will receive an error if you keep the default. You will want to remove the [SSH TUNNEL PORT] from the url, so it looks like [APPLICATION NAME].vcap.me.

With that, you should see VMC staging and starting your application on your VCAP VM. When started, you can access via web browser or curl at [APPLICATION NAME].vcap.me:[SSH TUNNEL PORT].

It took me a while to get this working, but now that is, it is working great. The next step will be to see if Sinatra-Synchrony will work as well. I do not believe my application will require this, but it is more for an academic purpose at this point.

ESXi 5, Nexus 1000V, Custom ISO…oh my…

So a quick post today on an issue I ran into (and was also told about by @kendrickcoleman). After building a custom VMware ESXi 5 ISO with the Cisco Nexus 1000V vib, EMC PowerPath 5.7 vibs, and the vSphere HA vib, I deployed out 3 hosts. After deployment, I noticed all three hosts had one CPU core spiked while the rest remained idle. It turns out, that once the host is added to the Cisco Nexus 1000V dVS, CPU usage returns to normal.

Just an FYI on this topic in case other people run into it.

Blog::Post.new

Over the past year, I dropped the ball on this blog. It’s been a busy time, but I’ll try over the next year to make a few more posts. So what has been going on? Here are a few of the items keeping me busy:

• Received the results on the VCAP-DCA; did not pass. The equipment I was using was also moved, so I did not get the opportunity to prepare and take it again.
• Made a couple of trips to Ireland and spent about a month in England.
• Moved back to Louisiana.
• Dabbled in PowerShell, PHP, Python, Scala, Ruby
• Changed managers about two or three times.
• Passed the VCP 5

So what’s in store in the coming year? Well hopefully a lot of cool projects. There are a few topics I want to blog about and hopefully post some useful tools. I’m looking to move everything I do to Ruby, though there may still be some internal PowerShell scripts for some items.

This is my short update. Hopefully everyone had a good year.

Applying patches to ESXi 5 with the help of PowerCLI

So today, Chris Colitti posted a blog article about patching ESXi 5 without VUM. If you haven’t read it, I suggest you click this link and read his post.

After reading this post, I was surprised to find out that esxcli did not support pushing patch or extension files remotely. Prior to ESXi 5, you could use vihostupdate via the vCLI to install the patch or extension zip files to a host. With vSphere 5, vihostupdate has been deprecated. The esxcli command supports installing a patch or extension from a remote depot url (VUM for instance) or from a local file path on the server, as Chris pointed out. Another option would be to host the actual vib files on an http or ftp server and use the -v option with esxcli to point to the remote location of these files. However, this is not the point of this blog post. This blog post will talk about using PowerShell and PowerCLI to assist esxcli in installing patches remotely. I will not provide a whole script, but just some useful commands.

First, to make things easier, we’ll set a couple of variables.
$esxcli = "C:\Program Files\VMware\VMware vSphere CLI\bin\esxcli.exe" #location of esxcli
$server = "192.168.1.10" #server we want to patch
$patch = "PatchFile.zip" #patch file we want to apply
$patchLocation = "C:\" #local path to patch"
$datastore = "datastore1" #datastore where we want to store the patch
$remoteLocation = "/vmfs/volumes/" + $datastore + "/" + $patch #remote location of patch

Now, we want to mount a datastore to our workstation. First we need to connect to the ESXi host before doing anything.

Connect-VIServer $server
New-PSDrive -name "mounteddatastore" -Root \ -PSProvider VimDatastore -Datastore (Get-Datastore $datastore)

After we mount our datastore, we can then copy the patch to the datastore.

Copy-Datastoreitem $patchLocation + $patch -Destination mounteddatastore:

Once our file is copied, we can then execute esxcli to install our patch

& $esxcli --server $server software vib install -d $remoteLocation

This will execute esxcli on your workstation to install the patch we just uploaded to the datastore. Once the patch is installed, we can remove the patch and disconnect the datastore from the workstation.

del mounteddatastore:$patch
Remove-PSDrive -name "mounteddatastore" -PSProvider VimDatastore

So with this little bit of knowledge, you could write a PS script to copy the patch to a datastore (preferably shared across hosts), and then call esxcli to install the patch on the required hosts.

Update: A new blog post is up detailing using the Get-EsxCli cmdlet in PowerCLI. This can be located here.

Setting time on ESXi hosts through PowerCLI

So you go and build your new ESXi hosts, set NTP, but notice your time is still off. Did you know you can set your ESXi host time through PowerCLI?

After connecting to a host, you can set the time with the following commands. Always make sure you use UTC time when setting the time on your host as ESXi does not use timezones.

$t = Get-Date
$dst = Get-VMHost | %{ Get-View $_.ExtensionData.ConfigManager.DateTimeSystem }
$dst.UpdateDateTime((Get-Date($t.ToUniversalTime()) -format u))


And there you go. Now your ESXi hosts will have the same time as the machine you ran your script on, so just make sure that machine has the correct time.

And on why I have not blogged in a while, well that is another blog post.

VCAP-DCA….Finally!

VMware has finally released the VCAP-DCA exam, and I was able to schedule it pretty fast (still not sure if that was good or bad with my work schedule). Unfortunately, you agree to an NDA for the exam, so I can not post detailed information from the exam, nor would I. Seriously, do not ask, and if you do, prepare to pay a $10 non-refundable fee for any answer you may get back in return.

Here are my thoughts after taking the exam. First, I felt a lot better about this exam than I did the VCP 4. Now if I passed, well, I won’t know that for another 10 days. Make that another stressful 10 days. It was a very fair exam. I think VMware did an excellent job with a 100% lab based exam. I will say that I did not finish the exam. Well I was close. I did skip a couple of questions because time was running short, mostly because I knew what the answer was, but did not know the exact parameter to configure. So I moved on.

So what can you do to prepare?
1. Build a lab.
2. Practice
3. Practice
4. Practice

Honestly, you will not pass this exam if you just study documentation. You need hands on work with vSphere. Why? It’s lab based and there is a time limit. There is no google to help you out. Either you know it, or you don’t. There is documentation available, but you will waste precious seconds if you have to continually look up commands.

Now if anyone knows people involved with the exam, please find out if they would like feedback on it and get their information to me. I was disappointed there wasn’t anything available after the exam was completed. I would have liked to provide feedback because there were some errors in questions, and some items just did not work. I hope this gets taken into account when the exam is scored.

Now to wait….

Using PHP5 SOAP with vSphere API

I will more blog later detailing better usage of PHP with vSphere API.  This post is focused on how to use PHP5 Soap Client methods to interact with the vSphere API.

The vSphere API is SOAP based.  First step is to download the WSDL from vCenter or the ESX(i) host.  So how do we do this with PHP5?

First, we need a SOAP Client.

$connection = new SoapClient(“https://<hostname or ip/sdk/vimService.wsdl”, array(“trace” => 1, “location”=>”https://<hostname or ip>/sdk/”);

From this code snippet, we created a new SoapClient, $connection, set the location of the wsdl, set trace to 1 for debugging purposes, and set the location that we will send requests to.  Next we need to create a SOAP message to send to the server.

$soapmsg[“_this”] = new SoapVar(“ServiceInstance”, XSD_STRING, “ServiceInstance”);

Basically every call to vSphere API will require a “_this” key.  Most calls will require more information.  From this code snippet, we are creating an array with a “_this” key and setting a SoapVar with data of “ServiceInstance” and type of “ServiceInstance”.  Now, we are going to retrieve the Service Instance.

$result = $connection->RetrieveServiceContent($soapmsg);

And what do we get?  Definitely not the result we were expecting.  With trace enabled, we can actually view the SOAP request that was made to the server.

var_dump($connection->__getLastRequest());

What is the problem with the SOAP request?  Well, PHP5 SOAP function sets the type of the SOAP message as “xsi:type”.  The vSphere API expects a type of just “type”.  How do we fix this?  We must create our own class that extends SoapClient and overrides the __doRequest method.  Here, we can strip “xsi:” out of the request and send the vSphere API the request it is expecting.

class mySoapClient extends SoapClient {

     function __doRequest($request, $location, $action, $version) {

          $request = str_replace(“xsi:”, “”, $request);

          return parent::__doRequest($request, $location, $action, $version);

     }

}

After altering the request, we can now successfully retrieve the Service Instance from the vSphere API.