Fixing my Update Manager issue

I recently upgraded vCenter to 2.5 Update 3, and this was my first time using Update Manager to update the hosts at one of my sites to ESX 3.5 Update 3 since the vCenter upgrade.  I migrated all the VMs off my “test” system, and ran an Update Manager scan to see which updates needed to be applied.  And this is when I saw this error message:

VMware Update Manager had a failure.

So I preceded to try again.  Same error.  Restart the service on the vCenter server.  Same error.  I stopped the service to look at the logs, and I see a SQL error regarding a foreign key constraint.  Searching for the error on Google, VMware, etc., I came across KB1007512.  This is the exact issue I had seen.  So following the KB, I had to let all the ESX/ESXi updates re-download.

Fast forward a couple of hours to the completion of the re-download of 5.70GB of updates.  I restart the Update Manager service and rescan my host.  I receive the same error:

VMware Update Manager had a failure.

Stop the Update Manager service once again.  Upon inspection of the log files, I see a different SQL error, this time about duplicate primary keys.  Search Google, VMware, etc.  Nothing.

I really wanted to push this off till the morning, but with the hectic schedule I had the next day, I decided to use my trusty VMware Support.  Gave them a call, and after uploading various log files to the support team, they had a fix for my issue.  In the Update Manager database, there are two tables: VCI_SCANHISTORY and VCI_SCANRESULTS.  The identity value for VCI_SCANHISTORY must be larger than the identity value for VCI_SCANRESULTS.  Running SELECT MAX(ID) FROM for VCI_SCANHISTORY and VCI_SCANRESULTS showed the identity values to be 20 and 134, respectively.  The fix was to issue the following command:

DBCC CHECKIDENT (“VCI_SCANHISTORY”, RESEED, 135)

This command reset the identity value of VCI_SCANHISTORY to 135.  With the identity value reset, I restarted the Update Manager server and successfully scanned my host to begin updating.  Thanks to VMware Support, I was able to go to bed at a reasonable hour.

Advertisement

My beef with Update Manager

First, I’ll start off with what caused me to make this post.

KB Article 1007512 – Scan of host fails after upgrade to Virtual Center 2.5 Update 3.

The resolution to this KB article is to rename the hostupdate folder and let Update Manager download all the ESX patches again.  5.60 GB worth!!!  So, without further ado, here are my beefs with Update Manager:

1. Update Manager is not smart enough to know I do not have any ESX 3.0.3 hosts in my cluster.  Furthermore, there is no option to disable downloading ESX 3.0.3 updates.
2. Unlike with Windows hosts where Update Manager only downloads the updates that are needed as per the baselines, Update Manager downloads every ESX update available, even if it’s not needed.
3. Resolution to KB 1007512
4. No ability to schedule reboots when patching a Windows host.  You can schedule when a patch will be pushed to a Windows host, but you can not push patches and then schedule a reboot for a maintenance window at a later time.
5. Unable to have more than one patch repository (would be very useful in networks with one VC server, but multiple clusters located across a WAN).

Now, Update Manager is a really good product for a 1.0 version.  I hope to see some of these changes in future releases of this software.

Campus Network Design

My role in modernizing our work network started a little over a year ago when I joined the network team.  Now to better understand our network, it’s best to think of it as a campus network.  We have a lot of buildings, with only a few having a large amount of users, connected all together in a switched network.  In total, I believe I have over 100 switches right now currently making up our network.  The list of issues on our network are many:

• All users, switch management, servers, printers, etc. all on VLAN 1
• Access switches daisy chained together
• Switches handling routing that are not the core switches of the network

Now outside of everything on VLAN 1, some may ask what exactly is wrong with this layout.  Well for one, it’s very difficult to locate devices on the network, since all devices are on the same subnet.  Thankfully, we have a Fluke Optiview that can locate devices in a large layer 2 network.  The network is also prone to flooding, from broadcasts, unicast, and multicast flooding.  In fact for a couple of weeks, we experienced unicast flooding due to a misconfiguration between our core switches and the newly-installed-but-completely-out-of-my-control layer 3 switches. (Maybe one day I’ll do a post about outsourcing and it’s “benefits”).

So I’ve set out to improve the network.  I am working to model the network after the Cisco Campus Network Design.  The basis of this design is that access switches, routers, VPN routers, datacenters, etc. connect to distribution switches, and the distribution switches connect to two core switches.  The two core switches are the control center for the network.  Now, if you follow the Cisco recommendations, the Core switches only handle layer 3, and the distribution switches handle layer 2 and layer 3.  Recently, there was a design discussing bringing routing all the way down the access level.

I see two issues with bringing routing all the way to the access layer in our network.  First, we have a guest VLAN for our contractors that is spread through one vlan in our network, and we have a security VLAN for our security network that is spread through one vlan in our network.  VRF-lite is basically VLANs for layer 3, but I believe each network in the vrf would be in it’s own subnet.  However,  I haven’t verified this yet.  If this is the case, then that will increase management.  Second, with over 100 switches, we are talking about 200 separate networks (data and voice) on each switch.  That is 200 subnets to manage dhcp address for compared to the 2 we have now.  This would be a huge increase in management.  Granted, we could easily locate devices on the network by just performing a traceroute.  Broadcast flooding or unicast flooding would be limited to only one switch.  With the access switches doing routing, you could standardize on a similar configuration for all switches.

But do the increased efficiencies in the network offset the increased management of the network?

Lets forget about routing at the access layer, and go back to the design with the distribution switches being layer 2/3 devices.  I would still have to look at using VRF-lite for the security and guest networks.  With about 8 distribution spots in the network, I could have a minimum 16 vlans (1 voice and 1 data vlan for each distribution point) and 16 subnets, but you would have more than one access switch per vlan.  A limit could be set and have say 5 access switches per vlan.  This would increase the number of vlans and subnets to manage, but would reduce the number of devices in each vlan.  But in this setup, there is still an issue of locating devices on the network.  If one went with the minimum setup of 16 vlans and 16 subnets, private vlans could be used to isolate traffic between buildings, but even private vlans are not without their security issues, so this would need to be taken into account.

All in all, it appears I have a daunting task ahead of me.  At some point, I will have to make concessions to both manageability and efficiency in the network.  Though this exercise has made me think about subnets and manageability vs efficiency.  Say I wanted a different subnet for each access switch.  I install a 24 port switch.  Is it really efficient to assign a /24 subnet to this switch just for management sake since this is what most people are used to?

I’ll update this with links once I figure out what’s going on with not being able to add links.  I believe it may something to do with using Safari 4 Beta.

Another blog….so what…

Another blog is created.  Will anyone notice?

So what is this all about?  I’m looking to share my ideas and thoughts on virtual datacenter design and networking layouts.  Every now and then, other non-related information may be posted.  If anyone finds a use for my ideas, I’ll be more than happy to accept a beer as payment.  I’m a VMware VCP living in South Louisiana working towards the VMware VCDX.  I’m currently waiting for my Enterprise Exam to be scheduled.

And it begins…

The beginning of something.  We’ll just see where this goes.