I was digging through my collection of junk that I had on my Dropbox and I found an older PDF that I made that helped me when troubleshooting View issues.

 

The PDF which is linked here VMware View Logs shows where traffic is flowing and if something fails that the service is leveraging, which logs to look at. I don’t have Blast or anything in there because I think at that point it hadn’t come out yet but I will be updating it over the next few weeks.

 

 

VMware View Logs

 

There are very few times that you will ever need to do this but when you do, you will have the information that you need. I frequently use this when testing HA during a greenfield implementation.  I do this prior to removing a blade so I can test software failure prior to physical failure. Another instance of you having to do this is if you are having to work with VMware support and they need to get some full cores to see if there is a driver / software issue. Either way it’s pretty fun so let’s do it!

SSH to your host

SSH

Warning: If you type the following command and hit enter on a host it will cause it to panic. Use caution when doing this.

Type: vsish -e set /reliability/crashMe/Panic 1

Crash Me

 

After doing so you will see the following screen on the host

PSOD

 

After this you can gather a VM support or wait and watch HA fail the VMs over if it is enabled for this cluster. This will allow you to test a failure without pulling any hardware.

So this is one thing that I saw when I was messing around that really made me wish I knew this sooner. Basically you can access the DCUI via SSH through your client of choice as long as the process is running.

First off let’s SSH to the host

SSH

 

Now let’s type DCUI

dcui

 

Go ahead and hit enter and you will get the following

 

dcui in it

 

So now you can go ahead and hit F2 and login and start changing settings or shut the box down. To get out of this window go ahead and hit CTRL + C and you will be back to the command line.

Opening:

So I finally achieved a goal that I set a long time ago. It still hasn’t set in that I did it and I don’t think that it ever will. My parents still think that I put computers together and that this means I can now do it better than I could before. Either way, these were the major points that I hit going through the process.

Set a Schedule:

I personally gave myself about 2 hours a night during the work week for my design document for the first 2-3 months prior for submission and about 4 hours on weekends. I would ONLY stick to this schedule during that time so that I had a regular review schedule as well as a way to add more meat to the document. I have a whiteboard that sits between my kitchen and my office and this is so that when I get coffee I have to stare at it as my Keurig makes crazy hissing noises. Write down the sections that you need to edit or have peer review and just keep it there as a tabling area. This will remind you of what you need to do and if you are feeling risky and fancy you can even set dates for these. I honestly wish that I thought of this during the earlier parts of the design. I came across this idea with about 2 weeks left and it helped me stay on track. Adjust your schedule accordingly when getting closer to the defense day. I think towards the end I was doing 8-12 hour days just re-reading my design and doing mocks.

Prior to working on your design every night go back and re-read what you have done thus far. This will cement this in your brain as to what you have done and help you memorize it. I also used this policy when I went through and did mocks. Always read your design and know it inside and out.

Side note: If you aren’t extremely intimate with your office suite of choice….get prepared to be. She is a cruel mistress but she is going to be your best friend through a majority of this.

Ask yourself questions:

My design was based off of a real customer environment that I had implemented. If you are going to submit for this beast, you need to make sure that if you are using a fictitious design then you need to treat it like a real one. Either way when I was going through every paragraph either editing or writing I was writing questions for myself that I thought someone would ask me. For example think of it like you have your 5-6 year old nephew next to you and your design is like a popup book. “The customer used the super powerful blade of B200 M3 to say the monster VM”  and he asks WHY, WHY are you doing this? Why did they do this? Could it be done better? Should it be? What features am I risking?

Always keep things like this in mind and be ready to fully explain EVERYTHING that you did. Alot of us in our group for the mocks started utilizing Quizlet for just things to ask ourselves or to keep in mind.

Side note: If you are walking around and arguing with yourself about why you did something, make sure you put in like a bluetooth earpiece or headphones. People thought I was crazy in my apartment complex and still look at me funny.

The Mocks:

Brad Christian @BChristian21 is a huge life saver and to be perfectly honest I don’t think many of us would have passed without his efforts. Brad reached out to all of us via Twitter and got us doing mock defenses with one another every night for 3 weeks straight. Alot of us were used to arguing with ourselves and not doing it against other people. With new people come different questions and different personalities and during our first mocks it made me feel like I knew absolutely nothing and like I wasn’t going to make it. This was a HUGE motivator for me. You need to be ready to asked anything and everything about your design and this group really pushed the limits of us. There were a few times where my fist wanted to go through the monitor but it made me better and it’s best to not take it personally.

Don’t Take it Personally

Mad

There were a few times that I got a bit touchy when people were hammering the hell out of me. Think about it like this, you are in a room with smart people and they all know things that you don’t. They WILL find your weakness and they WILL make you feel dumb about it. We all aspire to be good at what we do and be as knowledgeable as possible and these people are here to push you out of your comfort zone.

Troubleshooting:

During the mocks we didn’t really focus on this and this will make or break you when the time comes. Think of scenarios of when you were fixing something or even ask your friends. It’s all about the process you take and the way that you vocalize it. I worked in support for almost 3 years so I was having a blast doing mocks for the group but some of you may not have this available to you. If I were you I would head over to the VMware Communities  and pick a problem and attempt to solve it with as little information that is in the thread. If it is already solved then see what approach those people took and how you may have done it differently.

Design:

To be perfectly honest….we didn’t practice these and I would recommend doing that haha.

The Defense:

Emotion
You have been preparing for months and you are ready for this. The panelists are just people like the rest of us and they are in the business of making more VCDXs. You need to be confident and know your design inside and out. Everyone that I talked to said that a majority of the anxiety that you will be facing will fade after answering the first question. Use your PowerPoint as your backup for your entire defense. If there is a diagram that you have put it in the back of the deck so you don’t have to draw it. Keep in mind that you may have to draw it anyway so you need to know it inside and out.

What not to do:

Seriously though also look at the official video here: http://vmwarecertificationvideos.com/k34f/vcdx-defense-preparation-what-not-to-say/

Overall:

I felt that I learned a large amount and developed a great group of friends during this entire process. I was opened entirely more to criticism and became a bette consultant because of it. This process is like no other and has a great community behind it that are extremely eager to help anyone and everyone that is attempting.

Did you read anything?

Of course I did! There are several books that I practically lived next to during the entire process

VCDX Boot Camp: Preparing for the VCDX Panel Defense by John Arrasjid, Ben Lin, and Mostafa Khalil
Storage Implementation in vSphere 5.0 by Mostafa Khalil
VMware vSphere 5.1 Clustering Deepdive by Duncan Epping and Frank Denneman

Are you planning on defending soon?

Get a hold of James Bowling @vsentinal and let him know! He is working on getting a group together for the Cambridge defense. Here is the submission document here

If you need anything and I am not a part of the panels I would be delighted to help you. Just tweet me @kalenarndt 

So on Dec 31st I scheduled my VCAP-DTA5 and figured it wouldn’t be that bad compared to other exams I have taken……I was pretty wrong.

First thing is that I went through the blueprint and it looked extremely straight forward on what was going to be tested and how many questions there actually were. I will start off saying that I love the practical labs but with only 23 questions they only give you 3 hours. You may think….well 3 hours that’s a ton of time for 23 questions! You sir are wrong. If you have ever taken any of the practical VCAPs after 4.x you will know that 23 questions only means there are 23 common questions with sub questions for each. So this means you are doing something around like 30ish with all the steps.

The take away is…..KNOW EVERYTHING and work efficiently. If you are stuck waiting for the lab to actually render a page (I will go over this in a bit) then you should move forward and write down what the other question want you to do while you wait. 

My main complaint was the speed of the lab. Since Pearson isn’t running a local environment for this you are at the mercy of their testing internet to RDP into the lab you are going to be working on. If it is slow raise your hand immediately. They can only do so much but just have them report it. Now if the internet speed is fine you still have a few things to worry about. I am pretty sure this is a nested environment. I only think this is true because the speed is a huge issue. I have virtualized View before in a nested environment and it performed at just about the same rate. You are going to need to make sure that you know exactly where things are in the ui and what they want you to accomplish. I personally spent a majority of my time waiting for the lab either in the View flex UI or waiting for tabs to actually render. Please note that my testing center was having massive internet problems and when they moved me (for the 4th time -.-) it finally was a bit better.

Time management is something that you need to focus on here. Don’t guess where things are or what things are in the blueprint…deploy it prior. 

I didn’t actually study for the test prior to taking it. You may think….what the hell is wrong with you? I used to fix View all day every day when I was at VMware so I felt extremely confident since I used to be in support. The other part of that statement is wow there were things in support that I had fixed but I hadn’t actually implemented.The night before I would go through each part of the blueprint with a test Connection Server / Security Server. I would review each part of the blueprint and see how fast I could complete each portion and then move on from that. This test actually tests the broad range of things you can do in a View environment and should know how to do in an environment.

I was very surprised that my labs didn’t break like they had in previous tests. All of my problems were with the Pearson Vue workstations which were usually hard locks, slow internet, hard locks, and an odd comment lock issue.

TL;DR

  • Use this to determine what you don’t know: http://mylearn.vmware.com/register.cfm?course=187168
  • Build a lab environment and just run through the blueprint section by section to see how fast you can do it and learn what you don’t already know
  • Time Management
  • Submit Feedback to VMware! The comment button is there for a reason.It won’t hurt their feelings and it will help them understand the challenges that you face.
  • Don’t freak out if something bad happens in the exam. Raise your hand and let them know! They may give you additional time for their end being horrible.
  • Did you pass or not? I am still waiting on my results.

Let me know if you have any additional questions or feedback for me.

 

 

In a previous post I gave an introduction to a new tool that was added in ESXi 5.5 that allows for additional granularity when troubleshooting Networking issues. I’ve finally found some time to play around and understand how to use the command and wanted to put those notes down so that I could come back to them when needed.

When running pktcap-uw the structure of the command should look something similar to (NOTE** The blog page is formatting the commands with only a single – . All parameters should include two – )

pktcap-uw –capturepoint <capture point> –interface <interface> –dir <0/1>  –stage <0/1> –dstport <port> –proto <0xproto>

The name capturepoint is a bit deceiving. I originally assumed this meant the item that we would be dumping (vmkernel, physical nic, switchport etc); however, it is actually an additional parameter to go along with the interface. A great example is the capture point PortOutput which shows traffic being delivered from the vSwitch to the Guest OS. To get a list of available options of capturepoints run pktcap-uw -A. By default the direction of all captures are set to receive (–dir 0) but can be changed to see outbound traffic as well (–dir 1). At this time I have not been able to identify a way to capture both ingress/egress traffic. The stage parameter identifies whether the traffic is captured before, or post, capture point. Ultimately this allows us to view where traffic is getting dropped and identify if there is an operation inside the host that is causing the problem.

Beyond the direction and stage the parameters of pktcap should feel very similar to our old friend tcpdump. A source or destination port can be specified by –srcport and –dstport respectively and the same applies for both source and destination mac and IP address. If you want to output the pcap to a file for analysis later on you can use the -o <FILENAME> parameter. Explaining the help screen is all well and good, but let’s see pktcap in some real world scenarios below :

 

If I want to capture all traffic on vmnic0 for port 22 the command would be :

pktcap-uw –uplink vmnic0 –dstport 22

ehlpc1

 

Neat! That would be the traffic from my current SSH session to the host that’s being echo’d on the screen. What if we want to just capture ICMP traffic that’s going to the vCenter server running on that host?

 

Our first step is to run esxtop and switch to the networking tab by hitting ‘n’ :

elhpc2

From this screen we capture the highlighted PORT-ID for our vCenter server. In this case it’s 50331656. Leave esxtop by hitting ‘q’ and enter the command below: (NOTE** All protocols will need be referenced by their hexadecimal values which can be found here)

pktcap-uw –switchport 50331656 –proto 0x01

ehlpc3

Interesting. We’re not seeing any traffic captured, this makes sense as my constant ping from my desktop is timing out. Let’s see if it’s even making it to the physical interface that the vCenter server is running on. If you reference the esxtop screenshot again you can see that vCenter has bound it’s traffic to vmnic1. Let’s capture all ICMP traffic destined for my vCenter server’s IP ending in .10.117 :

pktcap-uw –uplink vmnic1 –proto 0x01 –dstip x.x.10.117

ehlpc4

So the traffic is at least hitting the physical nic, but we’re not seeing it at the guest. Let’s see if the vSwitch is delivering it to the guest by specifying the capture point PortOutput:

pktcap-uw –capture PortOutput –switchport 50331656 –proto 0x01

ehlpc6

At this point we’ve been able to identify that not only is the traffic reaching the physical interface of the host, but it’s making it’s way through the vmkernel and the out the virtual port of the vSwitch to the Guest OS. Investigation at this point should be refocused on the vCenter server itself to see why the ICMP requests aren’t making it through. Make sure to check your Windows Firewall people 😉 .

There is a plethora of more options for this tool and the capturepoints are specific to the interface you capture on. For example, using the –capture PortOutput when specifying a switchport as the target will show traffic delivered from the vSwitch to the Guest. Using PortOutput when specifying a physical adapter shows traffic being delivered from the vSwitch to the physical adapter.

For more information on the EHLPC check out http://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.vsphere.networking.doc/GUID-C1CEBDDF-1E6E-42A8-A026-0C067DD16AE7.html

 

-Dillon

 

This is a personal blog. Any views or opinions represented in this blog are personal and solely belong to me and do not represent those of VMware, unless explicitly stated.
All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.

So a few weeks ago I attempted the VCAP5-DTD and I am on the fence about the whole ordeal. I will first say that I do not like any of the design exams as you generally have to stop thinking logically for some of the questions.

Here are a few materials that I used when I was studying:

Chris Bekket’s Study Guide

VMware’s Blueprint

VCAP5-DTD Design Simulator

Make sure that you go through this a few times because it helps with how they want the design questions laid out and how to handle a few of the infrastructure pieces

Implementing VMware Horizon View 5.2

  • I bought this book a while back and I went through it prior to this test and it was great even though it covers 5.2. Most if not all concepts are the same but you need to keep in mind that the configuration maximums have changed along with feature sets.

The product documentation is a great place to start digging in especially the architecture design guide

Make sure that your storage knowledge is up to par for this test and your underlying vsphere knowledge. I would also look at the Storage guide for View 5.1 and this is also covered in the book above.

Rant:
It isn’t clear what version that this exam is on. This may not be a big deal when you are comparing 5.0 to 5.1 since the configuration maximums didn’t change but between 5.1 and 5.2 it is a totally different ball game. I finally caved and stuck to my guns and studied for it based on 5.1 since the exam is fairly old and VMware takes a good amount of time to upgrade their tests when new versions come out.

Working in support, one of the most common questions I receive from Network administrators and VMware admins alike is “What’s going on, on the vSwitch?”. vSwitches (Distributed or not) can be as a colleague of mine says “A black box filled with voodoo inside”. Unfortunately in the past the best way to observe the traffic was a painful process that required SPAN ports on the physical switch, Wireshark VMs or configuration changes made to the vSwitch to allow captures. Combine this with the fact that these processes can often involve multiple teams within the organization working together and it creates a recipe for a slow moving troubleshooting process. Keeping all of this in mind I’m sure I wasn’t the only one that gave a both a sigh of relief and a bit of a celebration when reading about the Enhanced Host-Level Packet Capture command in the  vSphere 5.5 What’s New Document .

If you haven’t heard, or aren’t sure as to why this is a big deal let me outline a few of the benefits below :

  • Available as part of the vSphere platform and can be accessed through the vSphere host command prompt

The key point to take away from this is that it’s included directly on the host, so any networking issues involving vCenter do not inhibit the use of the tool

  • Can capture traffic on vSS and DvS

 

  • Captures packets at the vNic, Uplink and Port level

This is the reason I am writing a blog about the command. The ability to capture traffic at these different levels within the hypervisor not only allows to demystify the the vSwitch a bit, but it will also allow for expedited troubleshooting. No longer will the blame game be dragged out and reliant on SPAN ports and promiscuous mode to get a full view of the environment. As a VMware professional, if this doesn’t put a smile on your face there is something wrong.

  • Can capture dropped traffic

I think this really speaks for itself. If there is traffic being dropped within the hypervisor it helps to actually be able to identify where.

  • Can trace the path of the packet with timestamp details

 

While this tool has been highlighted by VMware, I was not able to find any mention of the actual command anywhere. I decided to go digging myself and found what was I was working for :

pktcap-uw

The new tool handles much the same as tcpdump but allows for additional granularity in what you’re capturing, or not, and how. A quick view of the help screen by appending the -h switch even shows support to capture at the dvfilter level (vShield App anyone?). I for one am very excited to utilize the tool as it will make my job in support that much quicker and easier to identify problems with networking in or outside the host. I will be adding another post in the upcoming days after I’ve had some time to truly understand what each option allows for and best ways to use them. In the mean time there is a public KB that gives the barebones of captures that can be found here.

I recently ran into an issue where I had to do a factory reset on a Gen 5 EMC Recoverpoint appliance. When I would apply the config the appliance would entirely drop off the network and I would have to reinstall using the ISO. After each install I noticed that the config was still being pulled from a partition on the local device…which is an issue.

When hitting Ctrl + G during the boot the appliance would try to load into the raid configuration utility….unfortunately it would hang indefinitely until you reboot the box.  In order to get into the configuration utility I had to do a few things.

  • Reboot the server
  • Hit F2 to boot into the bios
  • The bios is password protected so I used the password I found here
  • Modify the USB legacy compatibility and set it to disabled
  • Reboot the server
  • Hit Ctrl + G

At this point you should be in the Intel configuration utility. This is where I just went and re-initialized the virtual volume and rebooted the server.

Note: After this make sure you got back into the bios and set the USB legacy compatibility to enabled. If you don’t then during the new install of the RP software it will just continually boot loop.

Just install the version of the RecoverPoint software you want and it shouldn’t pull the old config and be good as new.