Reconnecting ESX(i) hosts with 1000V installed

I follow a certain policy: “You break it; you fix it”.  Why?  When you break something, you learn a lot about why it broke and what it takes to fix it.  Hopefully, you also learn how to prevent it from happening again.  Two, I’m not stuck dealing with the problem someone else caused.

Tonight was one where I broke something, and now I needed to fix it.

I had to reconfigure a UCS system running 3 vSphere 4.1 ESXi hosts connected to a Cisco 1000V.  Before powering down the hosts, I made some changes to the 1000v and the UCS.  So far, so good.  All the hosts were powered down along with vCenter and the 1000V VSMs, which were running on the UCS chassis.  I made my configuration changes on the UCS and powered up the first host.  No connectivity.  Something was not jiving between the 1000V config on the host itself and the vNIC configuration from the UCS.  It turned out to be a misconfiguration of the native vlan on the vNIC presented to the host and the native vlan configuration configured on the host by the 1000V.  I returned some values to their previous settings, and the host came back up.  I updated the 1000V config, which took the host back down, but I adjusted the UCS to the new configuration I wanted to bring the host back up.  Great, except I still had to bring two more hosts back online.

This is where I dove into vemcmd on the ESXi host.  When I brought up the next host, I opened a local console to the system.  “vemcmd show port” showed me the native vlan configuration error that was causing my issue.  So how do you fix this?  It’s actually quite simple.  When looking at the output of the previous command, you’ll notice three Trunk ports on the system.  Two are the actual physical uplinks connected from the  system.  The third is the port channel that is formed between the two nics.  To bring the system back online, you need to issue the following command: vemcmd set port-mode trunk native-vlan <native vlan> ltl <ltl of port-channel>.  After issuing this command, the system was back online.  You’ll notice that after issuing the command, the native vlan of the physical nics remains the same.  After the VEM gets the updated configuration, the native vlan is now the correct on the physical nics.

Advertisement