Recently on a customer site we had a peculiar scenario where we were deploying a VM into a NSX environment via vRA and the operation was failing due to an unknown reason. However, we noticed that the IP address that vRA had allocated for the new VM was still responding to our pings even though the provisioning process had failed and the VM was never actually deployed…… so what was responding to our pings and where was it????
We tried accessing it via telnet, ssh, RDP, VNC, SNMP and a few other methods to figure out what it was but had no luck (we were not the only ones working in the environment so there were machines deployed we knew nothing about). We knew it must be a VM as the particular subnet was associated with a VXLAN, and we could see within the vSphere Web Client all the VMs that were connected to the logical switch in question, but due to the fact the VM template used to build these test machines didn’t have VMware Tools installed, we couldn’t see the IP addresses on the VMs in the vSphere Web Client.
So the question asked was “how do you find out what VM an IP address belongs to, in an NSX-v environment?
Cisco Commands
In a traditional physical network, to figure out where an IP address is configured, I would normally jump on the gateway for the L3 subnet, and look at the ARP tables of the L3 device. On a Cisco device, the command I would use would be something along the lines of:
Router#show ip arp Protocol Address Age (min) Hardware Addr Type Interface Internet 10.29.4.1 0 0050.56c0.0002 ARPA GigabitEthernet1 Internet 10.29.4.222 0 8c70.5a12.8454 ARPA GigabitEthernet1 Internet 10.29.4.101 - 000c.292f.e634 ARPA GigabitEthernet1 Internet 10.29.4.254 13 0050.56f8.b138 ARPA GigabitEthernet1 Internet 10.29.32.254 2 0254.003c.fa14 ARPA GigabitEthernet1
This would give me the MAC address associated to the IP address in question, and also the physical interface for which the router has associated the information to.
Next I would jump onto the L2 switch connected to the interface which the router has associated the ARP information to, and look at the mac-address tables to figure out which port the applicable MAC address is associated to.
switch1#show mac address-table address 8c70.5a12.8454
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
10 8c70.5a12.8454 DYNAMIC Gi0/25
Total Mac Addresses for this criterion: 1
switch1#
Now you know the physical port the phantom device is connected to, and if your switch port descriptions are kept up to date, it should tell you what it is connected to the port. If it connects to another switch, you repeat the process until you find the end device and if required, trace the physical cable. If it went to a blade enclosure or a hypervisor, I would then go and bug the team who looked after those environments so they could tell me what the device is 😉
But what if there are no physical switch ports or cables like in a NSX-v environment and the IP in question is a VM connected to a logical switch?
NSX-v Commands
First, we need to identify the master controller for the vni.
SSH into one of the controllers, or jump on the console of one, and run the following command:
nsx-controller # show control-cluster logical-switches vni 5001 VNI Controller BUM-Replication ARP-Proxy Connections VTEPs 5001 10.29.90.61 Enabled Enabled 3 3
We can see the master controller for vni 5001 is 10.29.90.61.
Now jump onto the controller 10.29.90.61 and we will look at the ARP tables associated with vni 5001. We can issue the following command on the master controller:
(the IP address we are going to follow in this example is 10.29.84.141)
nsx-controller # show control-cluster logical-switches arp-table 5001 VNI IP MAC Connection-ID 5001 10.29.84.141 00:50:56:be:81:d3 10183 5001 10.29.84.142 00:50:56:be:0d:08 10187
Now we have the MAC address (00:50:56:be:81:d3) which belongs to the IP address 10.29.84.141, along with the Connection-ID (10183) of the ESXi host.
OPTIONAL: If we issue the following command we can see which VTEP the MAC address is mapped to.
nsx-controller # show control-cluster logical-switches mac-table 5001 VNI MAC VTEP-IP Connection-ID 5001 00:50:56:be:81:d3 10.29.97.1 10183 5001 00:50:56:be:0d:08 10.29.97.2 10187 5001 00:50:56:be:dd:46 10.29.97.2 10187 5001 00:50:56:be:34:de 10.29.97.3 10188
You can see from both the commands above that IP address 10.29.84.141 has a MAC address 00:50:56:be:81:d3, is mapped to VTEP 10.29.97.1, which has a Connection-ID of 10183.
Next we need a way to figure out which host the Connection-ID belongs to.
Thanks to Dmitri Kalintsev and this post, we know that we can use the following command to display the connection-table, which will show the ESXi management host IP address associated to the Connection-ID:
nsx-controller # show control-cluster logical-switches connection-table 5001 Host-IP Port ID 10.29.91.3 12836 10183 10.29.91.4 50722 10187 10.29.91.2 57147 10188
We can see that Connection-ID 10183 belongs to Host-IP 10.29.91.3.
Now we can SSH into ESXi host 10.29.91.3 and run the following command to look at what VM and vNIC the MAC address belongs to:
~ # net-stats -l PortNum Type SubType SwitchName MACAddress ClientName 67108866 4 0 vSwitch1 74:e6:e2:ba:f9:78 vmnic3 67108869 3 0 vSwitch1 00:50:56:63:d2:53 vmk2 83886082 3 0 vSwitch2 00:50:56:6f:f8:e8 vmk1 83886083 4 0 vSwitch2 74:e6:e2:ba:f9:75 vmnic2 100663298 4 0 DvsPortset-1 74:e6:e2:ba:f8:a4 vmnic1 100663300 3 0 DvsPortset-1 00:50:56:6c:5a:c2 vmk4 100663301 3 0 DvsPortset-1 00:50:56:67:f8:c7 vmk3 100663302 3 0 DvsPortset-1 74:e6:e2:ba:f8:a1 vmk0 100663306 4 0 DvsPortset-1 74:e6:e2:ba:f8:a1 vmnic0 100663310 3 0 DvsPortset-1 00:50:56:6a:06:6a vmk5 100663353 5 9 DvsPortset-1 00:50:56:be:81:d3 sneaku-scan01.eth1 100663354 5 9 DvsPortset-1 00:50:56:be:de:c0 sneaku-scan02.eth0 100663356 5 7 DvsPortset-1 00:50:56:be:ad:f8 NSX_Controller_4e6a2ed1-f438-4791-a8ae-07bc782b3735.eth0 100663359 5 9 DvsPortset-1 00:50:56:be:4d:12 NSX Manager.eth0
And just in case you were wondering, the IP Address in question belonged to a machine which was manually provisioned into the wrong subnet/logical switch as part of some previous testing, but was forgotten about. We ended up deleting the machine 🙂
Hope this helps.