by ·Comments Off on VCSA6 Utility: port-accessible

There have always been a severe lack of port querying tools with ESXi until nc was added to the builds and now it looks like the VCSA finally gets something along the same lines. This utility is somewhat limited to what you can and cannot do but it helps when you are in a pinch.

This script is called port-accessible.py and is located in

So if we run the python script we will observe the following:

So if I want to issue a test to say one of my hosts to make sure that we can communicate over 902 I would issue the following :

If we are using tcpdump at the same time we can see the rrequest and then we can see the response

If we try it on a port that we know isn’t open on the host we get the following

The command will sit there and hang and while tcpdumping the interface we observe constant retries but the script will not terminate until there is a total of 6 failures

Here is what is returned after the failure

I do know for a fact that the ESXi host has port 80 open so I want to try to see what a HTTP return is when I set the flag for it.

Verdict: It’s not perfect but I think that it is a great start. I would love if they could include nc on the VCSA but I am sure that they have their reasons. My main complaint is that you can’t specify the protocol that you can use when sending the traffic.

 

by ·Comments Off on E1000E and Me: A late night date that pisses me off

This week it has been my goal to learn vCAC and during this time I have been running into old remnants of past labs that have been causing me problems. I was finally getting around to installing my IaaS VM and I kept having the VM BSOD during the install and instantly my host would PSOD. Since I don’t have a KVM I had to just hard power it and wait for it to come back up.

Checking in /var/core I find the following

a

Well that isn’t great but atleast we have a core and with a core we can find out what happened (usually). So after talking to my cousin Dillon I found out there was a tool that we can use

esxcfg-dumppart -L dumpname

This will output the results in the current working directory where you run the command. In my case it was in /var/core

b

If we look at the log we can see the following

c

After doing a bit of  talking to Dillon and Googling we can find this http://kb.vmware.com/kb/2059053 which turns out is a known issue and the only fix is to upgrade.

Turns out I haven’t updated this host in a while.

2014-11-06_19-21-29

Since I am currently only running 1 host in this environment I am using the script from http://www.v-front.de/2014/03/how-to-update-your-standalone-host-to.html with the new package number for 5.5 U2

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.5.0-20140902001-standard

Lesson learned….keep my lab hosts up to date before I start new side project goodness.

 

by ·Comments Off on Using vSphere 5.5’s Enhanced Host-Level Packet Capture

In a previous post I gave an introduction to a new tool that was added in ESXi 5.5 that allows for additional granularity when troubleshooting Networking issues. I’ve finally found some time to play around and understand how to use the command and wanted to put those notes down so that I could come back to them when needed.

When running pktcap-uw the structure of the command should look something similar to (NOTE** The blog page is formatting the commands with only a single – . All parameters should include two – )

pktcap-uw –capturepoint <capture point> –interface <interface> –dir <0/1>  –stage <0/1> –dstport <port> –proto <0xproto>

The name capturepoint is a bit deceiving. I originally assumed this meant the item that we would be dumping (vmkernel, physical nic, switchport etc); however, it is actually an additional parameter to go along with the interface. A great example is the capture point PortOutput which shows traffic being delivered from the vSwitch to the Guest OS. To get a list of available options of capturepoints run pktcap-uw -A. By default the direction of all captures are set to receive (–dir 0) but can be changed to see outbound traffic as well (–dir 1). At this time I have not been able to identify a way to capture both ingress/egress traffic. The stage parameter identifies whether the traffic is captured before, or post, capture point. Ultimately this allows us to view where traffic is getting dropped and identify if there is an operation inside the host that is causing the problem.

Beyond the direction and stage the parameters of pktcap should feel very similar to our old friend tcpdump. A source or destination port can be specified by –srcport and –dstport respectively and the same applies for both source and destination mac and IP address. If you want to output the pcap to a file for analysis later on you can use the -o <FILENAME> parameter. Explaining the help screen is all well and good, but let’s see pktcap in some real world scenarios below :

 

If I want to capture all traffic on vmnic0 for port 22 the command would be :

pktcap-uw –uplink vmnic0 –dstport 22

ehlpc1

 

Neat! That would be the traffic from my current SSH session to the host that’s being echo’d on the screen. What if we want to just capture ICMP traffic that’s going to the vCenter server running on that host?

 

Our first step is to run esxtop and switch to the networking tab by hitting ‘n’ :

elhpc2

From this screen we capture the highlighted PORT-ID for our vCenter server. In this case it’s 50331656. Leave esxtop by hitting ‘q’ and enter the command below: (NOTE** All protocols will need be referenced by their hexadecimal values which can be found here)

pktcap-uw –switchport 50331656 –proto 0x01

ehlpc3

Interesting. We’re not seeing any traffic captured, this makes sense as my constant ping from my desktop is timing out. Let’s see if it’s even making it to the physical interface that the vCenter server is running on. If you reference the esxtop screenshot again you can see that vCenter has bound it’s traffic to vmnic1. Let’s capture all ICMP traffic destined for my vCenter server’s IP ending in .10.117 :

pktcap-uw –uplink vmnic1 –proto 0x01 –dstip x.x.10.117

ehlpc4

So the traffic is at least hitting the physical nic, but we’re not seeing it at the guest. Let’s see if the vSwitch is delivering it to the guest by specifying the capture point PortOutput:

pktcap-uw –capture PortOutput –switchport 50331656 –proto 0x01

ehlpc6

At this point we’ve been able to identify that not only is the traffic reaching the physical interface of the host, but it’s making it’s way through the vmkernel and the out the virtual port of the vSwitch to the Guest OS. Investigation at this point should be refocused on the vCenter server itself to see why the ICMP requests aren’t making it through. Make sure to check your Windows Firewall people 😉 .

There is a plethora of more options for this tool and the capturepoints are specific to the interface you capture on. For example, using the –capture PortOutput when specifying a switchport as the target will show traffic delivered from the vSwitch to the Guest. Using PortOutput when specifying a physical adapter shows traffic being delivered from the vSwitch to the physical adapter.

For more information on the EHLPC check out http://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.vsphere.networking.doc/GUID-C1CEBDDF-1E6E-42A8-A026-0C067DD16AE7.html

 

-Dillon

 

This is a personal blog. Any views or opinions represented in this blog are personal and solely belong to me and do not represent those of VMware, unless explicitly stated.
All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.

by ·Comments Off on vSphere 5.5 Enhanced Host-Level Packet Capture Overview

Working in support, one of the most common questions I receive from Network administrators and VMware admins alike is “What’s going on, on the vSwitch?”. vSwitches (Distributed or not) can be as a colleague of mine says “A black box filled with voodoo inside”. Unfortunately in the past the best way to observe the traffic was a painful process that required SPAN ports on the physical switch, Wireshark VMs or configuration changes made to the vSwitch to allow captures. Combine this with the fact that these processes can often involve multiple teams within the organization working together and it creates a recipe for a slow moving troubleshooting process. Keeping all of this in mind I’m sure I wasn’t the only one that gave a both a sigh of relief and a bit of a celebration when reading about the Enhanced Host-Level Packet Capture command in the  vSphere 5.5 What’s New Document .

If you haven’t heard, or aren’t sure as to why this is a big deal let me outline a few of the benefits below :

  • Available as part of the vSphere platform and can be accessed through the vSphere host command prompt

The key point to take away from this is that it’s included directly on the host, so any networking issues involving vCenter do not inhibit the use of the tool

  • Can capture traffic on vSS and DvS

 

  • Captures packets at the vNic, Uplink and Port level

This is the reason I am writing a blog about the command. The ability to capture traffic at these different levels within the hypervisor not only allows to demystify the the vSwitch a bit, but it will also allow for expedited troubleshooting. No longer will the blame game be dragged out and reliant on SPAN ports and promiscuous mode to get a full view of the environment. As a VMware professional, if this doesn’t put a smile on your face there is something wrong.

  • Can capture dropped traffic

I think this really speaks for itself. If there is traffic being dropped within the hypervisor it helps to actually be able to identify where.

  • Can trace the path of the packet with timestamp details

 

While this tool has been highlighted by VMware, I was not able to find any mention of the actual command anywhere. I decided to go digging myself and found what was I was working for :

pktcap-uw

The new tool handles much the same as tcpdump but allows for additional granularity in what you’re capturing, or not, and how. A quick view of the help screen by appending the -h switch even shows support to capture at the dvfilter level (vShield App anyone?). I for one am very excited to utilize the tool as it will make my job in support that much quicker and easier to identify problems with networking in or outside the host. I will be adding another post in the upcoming days after I’ve had some time to truly understand what each option allows for and best ways to use them. In the mean time there is a public KB that gives the barebones of captures that can be found here.