In a previous post I gave an introduction to a new tool that was added in ESXi 5.5 that allows for additional granularity when troubleshooting Networking issues. I’ve finally found some time to play around and understand how to use the command and wanted to put those notes down so that I could come back to them when needed.
When running pktcap-uw the structure of the command should look something similar to (NOTE** The blog page is formatting the commands with only a single – . All parameters should include two – )
pktcap-uw –capturepoint <capture point> –interface <interface> –dir <0/1> –stage <0/1> –dstport <port> –proto <0xproto>
The name capturepoint is a bit deceiving. I originally assumed this meant the item that we would be dumping (vmkernel, physical nic, switchport etc); however, it is actually an additional parameter to go along with the interface. A great example is the capture point PortOutput which shows traffic being delivered from the vSwitch to the Guest OS. To get a list of available options of capturepoints run pktcap-uw -A. By default the direction of all captures are set to receive (–dir 0) but can be changed to see outbound traffic as well (–dir 1). At this time I have not been able to identify a way to capture both ingress/egress traffic. The stage parameter identifies whether the traffic is captured before, or post, capture point. Ultimately this allows us to view where traffic is getting dropped and identify if there is an operation inside the host that is causing the problem.
Beyond the direction and stage the parameters of pktcap should feel very similar to our old friend tcpdump. A source or destination port can be specified by –srcport and –dstport respectively and the same applies for both source and destination mac and IP address. If you want to output the pcap to a file for analysis later on you can use the -o <FILENAME> parameter. Explaining the help screen is all well and good, but let’s see pktcap in some real world scenarios below :
If I want to capture all traffic on vmnic0 for port 22 the command would be :
pktcap-uw –uplink vmnic0 –dstport 22
Neat! That would be the traffic from my current SSH session to the host that’s being echo’d on the screen. What if we want to just capture ICMP traffic that’s going to the vCenter server running on that host?
Our first step is to run esxtop and switch to the networking tab by hitting ‘n’ :
From this screen we capture the highlighted PORT-ID for our vCenter server. In this case it’s 50331656. Leave esxtop by hitting ‘q’ and enter the command below: (NOTE** All protocols will need be referenced by their hexadecimal values which can be found here)
pktcap-uw –switchport 50331656 –proto 0x01
Interesting. We’re not seeing any traffic captured, this makes sense as my constant ping from my desktop is timing out. Let’s see if it’s even making it to the physical interface that the vCenter server is running on. If you reference the esxtop screenshot again you can see that vCenter has bound it’s traffic to vmnic1. Let’s capture all ICMP traffic destined for my vCenter server’s IP ending in .10.117 :
pktcap-uw –uplink vmnic1 –proto 0x01 –dstip x.x.10.117
So the traffic is at least hitting the physical nic, but we’re not seeing it at the guest. Let’s see if the vSwitch is delivering it to the guest by specifying the capture point PortOutput:
pktcap-uw –capture PortOutput –switchport 50331656 –proto 0x01
At this point we’ve been able to identify that not only is the traffic reaching the physical interface of the host, but it’s making it’s way through the vmkernel and the out the virtual port of the vSwitch to the Guest OS. Investigation at this point should be refocused on the vCenter server itself to see why the ICMP requests aren’t making it through. Make sure to check your Windows Firewall people 😉 .
There is a plethora of more options for this tool and the capturepoints are specific to the interface you capture on. For example, using the –capture PortOutput when specifying a switchport as the target will show traffic delivered from the vSwitch to the Guest. Using PortOutput when specifying a physical adapter shows traffic being delivered from the vSwitch to the physical adapter.
For more information on the EHLPC check out http://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.vsphere.networking.doc/GUID-C1CEBDDF-1E6E-42A8-A026-0C067DD16AE7.html