Nsx on SneakU

Monitoring NSX FW Rules per host

Wed, 18 Nov 2020 00:00:00 +0000

Today I was asked for some help in figuring out the number of NSX-T Distributed Firewall rules configured for all filters on a given ESXi host.

Why do we need to figure out this value? If you take a look at the configurations maximums page at configmax.vmware.com, you’ll see one of the NSX-T Configuration Maximums is the number of Distributed Firewall Rules per Hypervisor Host.

Up until recently (NSX-T 3.0.2), this number had always remained the same at 10,000 and was something that you should monitor to make sure you remained within this (soft) limit.

What's New in NSX-v 6.4.7 - L3port-optimization

Tue, 14 Jul 2020 00:00:00 +0000

Late last week, the latest version of NSX vSphere, 6.4.7 was released for General Availability (GA). Although it was mainly a maintenance release, there were a couple of items listed in the What’s New section of the release notes. I won’t list them all here, and instead I will just provide you with a link.

https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/rn/releasenotes_nsx_vsphere_647.html

Once a NSX-v environment has been upgrade to 6.4.7, you may notice there will be a difference in how Distributed Firewall (DFW) rules will look on the dataplane when viewing them using the following command:

Counting Exploded NSX-v DFW Rules

Tue, 09 Jun 2020 00:00:00 +0000

When working with a customer recently, there was a question raised about how to calculate the actual number of NSX-v distributed firewall rules configured for a given VM on the dataplane of an ESXi host.

Whilst the short answer was to jump onto the console of the hypervisor or SSH into the host and look at the rules configured on the filter with the following command:

vsipioctl getrules -f <filtername>

Looks easy enough right? However, It didn’t really work in this specific customers environment as SSH and console access to the hypervisors was managed by a 3rd party and hence it was impossible for the customer to run the above command.

Notify Switches on N-VDS

Mon, 01 Jun 2020 00:00:00 +0000

Recevied an interesting question today about whether a N-VDS supports the “Notify Switches” feature that is available in regular VDS. And as I didn’t know the answer to this off the top of my head, it was time to do some digging.

First we needed an environment where I had a N-VDS configured, and it just so happened I had a 2.5.1 setup running.

Jumping onto a host, first thing was to find what the N-VDS was actually called.

How to count DFW rules per ESXi host.

Wed, 27 May 2020 00:00:00 +0000

If you’ve heard me speak at VMworld on NSX Distributed Firewall best practises, you would have heard me speak about the importance of using the Applied To option when configuring DFW rules. One of the metrics using the Applied To option influences is the total number of rules configured per host.

If you haven’t seen the VMworld session, I’ve uploaded it to YouTube for easier viewing - https://youtu.be/fX9pwiIeMps

As per the published configurations on configmax.vmware.com, the maximum number of rules supported per host is as follows:

Working with NSX-T 3.0 and VDS 7.0 - Duplicate Port Group Names

Wed, 15 Apr 2020 00:00:00 +0000

When working with a colleague recently on an NSX-T 3.0 environment that utilises VDS 7.0 rather than N-VDS, we came across something which initially stumped me.

If you were to try and create 2 Distributed Port Groups with the same name on the same vCenter

the following error is displayed, which is expected.

So what would happen if we created a NSX-T Segment with the same name as one already created through vCenter?

NSX-T Error when trying to configure a VDS port group for a VMK Uninstall mapping

Tue, 10 Mar 2020 00:00:00 +0000

This is one of those pesky errors that I keep stumbling across, and every time It pops up, I forget how to fix it, so I am going to document it once and for all so that I can at easily find the fix for it.

When adding a ESXi transport node into NSX-T and wanting to configure the vmk uninstall mappings (to handle removing the NSX-T VIBs and migrating the VMKernel interfaces back to a VSS/VDS), the following error sometimes appears.

How to sync a dynamic NSX-T group to an external system

Mon, 02 Mar 2020 00:00:00 +0000

One of the things that makes NSX-T such a flexible platform when it comes to creating security policy is the ability to group membership defined by dynamic criteria, or based on objects such as a logical segment.

Once a group has been defined in NSX-T based on dynamic criteria, its membership can grow and shrink based on the specified criteria, which is awesome, and allows for virtual workloads to be added/removed from groups based on other attributes (e.g tags, amongst other attributes). But one of the sticking points of this extremely dynamic behaviour is that the groups are only available within the specific NSX-T environment for which they are configured.

Running nested NSX-T on NSX-T

Wed, 11 Dec 2019 00:00:00 +0000

Recently I have been seeing various posts about running NSX-T on NSX-T, and I thought it was about time that I finished off this post i’ve been sitting on for quite some time.

As part of my teams role, we have a lab environment which we use for customer re-productions, however, we also run enablement sessions internally using our own labs.

These nested NSX-T labs run on top of a NSX-T environment which is being driven by VMware Integrated Openstack (VIO).

NSX-v: Where is my object used that I am trying to delete?

Thu, 07 Sep 2017 00:00:00 +0000

Over the past few years of working with NSX vSphere, one of the more frustrating things that would happen is that you would try to delete an object of some description (IP Set, Security Group, Service etc) and you would get the annoying message in the UI which says the object is in use, but it doesn’t give you any more information or any context of where to even start looking to find out where it is being used.

Log Insight & NSX-v DFW

Fri, 05 May 2017 00:00:00 +0000

I was recently asked by a customer who is running NSX vSphere (NSX-v) and Log Insight, is there a way they can export data from Log Insight that will give them the unique source/destination/protocol/destinationPort from the DFW logs for a given ruleId over a given time period. This exported data could then be fed into a PowerNSX (Get it on GitHub) script to create the required firewall rules.

My initial response to the question was “buggered if I know, but I will send an email to some people that will know”.

Central CLI Update #1 for NSX 6.2.3/6.2.4

Sat, 17 Sep 2016 00:00:00 +0000

Following on from my previous post in regards to getting the Central CLI API working after an upgrade to 6.2.3/6.2.4, it seems that a nice little enhancement has been made which should help verifying and troubleshooting DFW data plane issues.

When looking at DFW filters and you wanted to view the contents of an address set, on the data plane you would issue the following command:

vsipioctl getaddrsets -f filter-name

And it would display the complete list of address sets for the particular filter.

NSXv - DFW Log Format Change

Sat, 17 Sep 2016 00:00:00 +0000

In NSXv 6.2.2 and earlier, the format of the DFW logs has remained relatively the same for quite some time now. The following is a specific sample of the dfwpktlogs.log file from an NSXv 6.2.2 host.

[root@host-192-168-111-11:~] tail -f /var/log/dfwpktlogs.log
2016-09-17T09:41:50.979Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:50.979Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:50.979Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.026Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.026Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.026Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.026Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.089Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.135Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.792Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.792Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:51.792Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137
2016-09-17T09:41:52.897Z INET match DROP domain-c46/1005 OUT 78 UDP 10.2.5.10/137->10.2.5.255/137

The interesting thing about the sample logs shown above is that these are from 2 individual VMs, connected to 2 separated logical switches, but the VMs have the same IP address. This is a situation which can manifest itself when using NSX in either a multi-tenant environment or when using NSX to clone existing topologies.

Central CLI API Update for NSX 6.2.3/6.2.4

Tue, 06 Sep 2016 00:00:00 +0000

When working on a customer site recently I was made aware that their system to use the NSX Central CLI via the API had stopped functioning. The system had stopped working when the upgrade to NSX 6.2.3 happened.

Click here for my previous post on how to use the Central CLI API.

When submitting the request, they would always get the following response:

The resource identified by this request is only capable of generating responses with characteristics not acceptable according to the request "accept" headers.

It turns out that there was a change made to the API, and an additional header is required to make it work with NSX 6.2.3 & 6.2.4. You now need to add the following to the existing headers:

NSX-v - How to find Object IDs for almost everything!

Tue, 12 Jul 2016 00:00:00 +0000

When starting out with the NSX-v API, you will quickly learn that there are times where you are required to reference an object, like when you want to add a member to a security group, you are required to know the object ID of the object you want to add as a member to the security group, as well as the object ID of the security group itself.

When using the NSX-v UI, you can try as hard as you want, but your aren’t going to be able to find the security group object ID. So how do you find it?

vSphere Web Client, NSX and a dreaded black screen

Sun, 03 Jul 2016 00:00:00 +0000

Every now and then I get asked by a colleague or client if there is a known issue with the vSphere Web Client and the NSX Plugin that causes the screen to go black.

My response is to ask them the following questions:

Are you using Firefox as the browser?
Are you accessing the machine your using the browser on via RDP?

So far the answers to the questions is always Yes to both of them!

NSX-v Web Central CLI

Sun, 21 Feb 2016 00:00:00 +0000

While NSX 6.2.x has been out for a while now, most people should be aware of the newly introduced feature called the Central CLI.

I won’t go into the details of what the Central CLI is, as you can read about it on Brett Draytons blog (link).

What I do want to point out though is that the main method for accessing the Central CLI is via a SSH connection to the NSX Manager. As you may or may not be aware, the authentication used for SSH connections to the NSX Manager is not integrated with SSO, meaning it uses local authentication.

NSX-v: The Penny Drops

Tue, 26 Jan 2016 00:00:00 +0000

I was onsite implementing NSX-v for a customer, and part of the installation in this environment requires that we allocate the VTEP pnics to the VXLAN transport VLAN. To do this I had to liaise with the Network Operations guy. After laying it out for him that although these VTEP interfaces are going to be used for “VM Data” (his words, not mine) they will only need to be an access port in VLAN xxx, the penny finally dropped for this guy on why they are going the NSX-v route.

NSX Manager: VTY configuration is locked by other VTY

Fri, 22 Jan 2016 00:00:00 +0000

Have you ever had the problem of working on a unstable network connection and your network drops out.

Well this happened to me this morning, but whilst it happened, I was actually connected to my NSX Manager via SSH and in configuration mode. When my network connectivity returned (4G) and I could SSH into the NSX Manager, I was greeted with the following:

VTY configuration is locked by other VTY
nsxmgr> ena
Password:
nsxmgr# conf t
VTY configuration is locked by other VTY
nsxmgr#

I asked around internally and it turns out that there is currently no elegant way of disconnecting the session which has crashed. I will be following this up so that hopefully an elegant solution will be included in a future release.

How to create a NSX-v API Only User Account

Thu, 21 Jan 2016 00:00:00 +0000

One of the main reasons for customers wanting to implement NSX for vSphere is the fact there is a RESTful API which can be leveraged to drive the whole system.

In a standard NSX-v installation, by default, the only account that has API only privileges (and no vSphere Web Client privileges) is the NSX Manager “admin” account. It is possible to use vSphere SSO accounts to interact with the NSX API, however this will also allow vSphere Web Client access (although they wont be able to view or access anything once logged on without granting specific vCenter rights).

NSX-v: Manually uninstall NSX VIBs

Thu, 21 Jan 2016 00:00:00 +0000

Due to a recent lab rebuild, I needed to manually remove the NSX VIBs that were installed on my ESXi host, and thought I would document the process so I can remember it in the future.

Removing the NSX VIBs from a host manually does not remove the configuration stored on the NSX Manager, so care needs to be taken when doing this.

Depending on the version of NSX-v installed, the number of NSX VIBs installed will vary.

Using PHP to query NSX-v via REST

Mon, 09 Nov 2015 00:00:00 +0000

Whilst working on a little side project of mine, I wanted to be able to submit a REST API call against a NSX Manager. At first I was just going to use shell_exec to execute one of my python scripts, but after a bit of investigation, I found that I could use the PHP Client URL Library (cURL) .

It took me a little bit of time to get it working just the way I wanted, but now that I have the basics, it should be simple to be able to do most things through a webpage.

NSXRAMLCLIENT on Ubuntu Issues

Wed, 30 Sep 2015 00:00:00 +0000

Following on from my last post about the NSXRAMLCLIENT, I decided to setup an Ubuntu machine dedicated to being able to run the NSXRAMLCLIENT in my home lab.

Using my own instructions that I had written previously to get it up and running on my OSX 10.10.5 machine, I came across a peculiar issue when trying to initiate a connection to the nsx manager.

Here is the code I was running which works on my OSX machine without issues:

NSX vSphere RAML Client

Fri, 18 Sep 2015 00:00:00 +0000

Up until now, my programmatic interactions with the NSX API have all been hacked together based on the NSX API Guide, due to the lack of a specific NSX API scripting interface.

For typical vSphere scripting, there has been PowerCli and also pyvmomi and I am sure a few more I don’t know about, but nothing created specifically for NSX. This meant that for individuals like myself, who wanted to write scripts to interact with the NSX-v API through python, we would have to create all the API calls from scratch which is time consuming and then learn how to parse the responses and content.

NSX-v Security Policy Applied To Considerations

Mon, 07 Sep 2015 00:00:00 +0000

When working with the Service Composer in NSX-v, by default, when a firewall rule is created in a Security Policy, the firewall rule, when applied, uses the default Applied To value of Distributed Firewall. This means that even though the firewall rule is part of a security policy which is then applied to a specific security group, the rule would be pushed down to every vNIC within the clusters that have been prepared for NSX DFW capabilities and have the firewall enabled.

NSX-v 6.2 What's New: IP Discovery

Fri, 28 Aug 2015 00:00:00 +0000

One of the new features in NSX vSphere 6.2 is the introduction of new IP discovery mechanisms. The official release notes mentions this as follows:

New IP address discovery mechanisms for VMs: Authoritative enforcement of security policies based on VM names or other vCenter-based attributes requires that NSX know the IP address of the VM. In NSX 6.1 and earlier, IP address discovery for each VM relied on the presence of VMware Tools (vmtools) on that VM or the manual authorization of the IP address for that VM. NSX 6.2 introduces the option to discover the VM’s IP address using DHCP snooping or ARP snooping. These new discovery mechanisms enable NSX to enforce IP address-based security rules on VMs that do not have VMware Tools installed.

NSX-v Unable to install VIBs: Blame VUM

Thu, 27 Aug 2015 00:00:00 +0000

When working with a customer recently to install NSX-v into a lab environment, we were faced with hosts which would fail when we went through the Host Preparation steps to install the NSX VIBs.

As it turns out, it wasn’t the usual problems of incorrect DNS or a firewall causing issues but rather vSphere Update Manager (VUM) which was causing the issue.

So from what I can deduce, it seems that when you Click the install button to prepare a cluster, EAM is used to install a VIB onto the ESXi hosts within that cluster, an agency is created within EAM for that specific cluster and within the cluster specific agency the following config option is set by default.

NSX-v 6.x: Operations and Troubleshooting Guides

Wed, 26 Aug 2015 00:00:00 +0000

Operating and troubleshooting a NSX-v environment can sometimes be a daunting task, especially if the customer had the environment setup by an external party (ie. VMware PSO or a VMware Partner). And so over the past few weeks, VMware have released 2 pieces of collateral which I am finding answer a lot of questions that I am normally asked by customers.

The first is the NSX-v Operations Guide, v6.1 (https://communities.vmware.com/docs/DOC-30079) that is posted on the VMware Communities site. At a high level, this document touches on the following topics:

Scripting: NSX-v – Bulk DFW Rule Creation

Wed, 26 Aug 2015 00:00:00 +0000

Recently when doing some performance testing with a customer, we wanted to have the ability to bulk create a number of DFW rules. By leveraging some python scripting and the NSX-v API, I came up with a flexible and re-useable script which can be used to create a new section and populate it with up to 997 DFW rules. The reason for creating a new section is so that when it comes time to delete the test rules, its as simple as just deleting the section, which will delete all rules within the section.

OSX Base64 Encoded Credentials

Tue, 25 Aug 2015 00:00:00 +0000

For those who have used the NSX vSphere REST API, you will know that with every API call you are required to send the following in the headers:

Content-Type application/xml
Basic <base64 encoded credentials>

So if you have an OSX machine, you can leverage openssl (which is installed by default) to generate the base64 encoded credentials. This means you do not have to find a website and paste credentials into the site to generate the encoded credentials.

NSX-v Quick Tip – Changing Controller Passwords

Tue, 04 Aug 2015 00:00:00 +0000

Changing NSX-v controller passwords is a question which comes up quite a bit with customers. Often because the passwords chosen at deployment time are either too complex to type in on a VM console session, or on the other end of the spectrum, they were set to something easy to type during deployment, and now need to be changed.

When deploying NSX-v controllers during the setup phase, the password is set when deploying the first controller.

NSX L2VPN with Standalone Edge

Tue, 14 Jul 2015 00:00:00 +0000

One of the features of NSX-v is the ability to create a Layer 2 VPN between 2 NSX-v Edge Services Gateways (ESG from now on). But what happens if there is no NSX-v at the destination where you would like to extend your Layer 2 network. As of NSX-v 6.1.0 the concept of a Standalone Edge L2 VPN Client has been made available to deploy into a remote vSphere environment.

Scripting - Resetting NSX-v objects

Sun, 14 Jun 2015 00:00:00 +0000

On various engagements I am involved with, I often need to produce some code to add or delete objects from NSX-v. These objects in general are things like IP Sets, MAC Sets, Security Groups, Security Tags, Services, Service Groups, Security Policies and even deleting the FW rulebase itself.

Until recently I was doing this manually as I was dealing with relatively small numbers of objects, however on a previous engagement I was working on a script to import up to 33,000 objects and when testing the script in a dev environment, we needed a way to go through and delete everything we had just imported and set it back to “defaults”.

NSX-v: Follow the IP/MAC address

Tue, 02 Jun 2015 00:00:00 +0000

Recently on a customer site we had a peculiar scenario where we were deploying a VM into a NSX environment via vRA and the operation was failing due to an unknown reason. However, we noticed that the IP address that vRA had allocated for the new VM was still responding to our pings even though the provisioning process had failed and the VM was never actually deployed…… so what was responding to our pings and where was it????

Scripting syslog server configurations on NSX-v Controllers

Thu, 28 May 2015 00:00:00 +0000

Its well documented that the only way to configure syslog settings on NSX-v controllers is via the REST API. One of the things I find myself constantly doing over and over again on customer engagements is configuring syslog servers details on the NSX-v controllers, whether its because I am at a new client, or someone decides to change the destination for their syslog and we need to update them all.

Broadcom bnx2x driver and VXLAN offload

Mon, 25 May 2015 00:00:00 +0000

The Broadcom bnx2x NIC driver for VMware ESXi when installed on a ESXi 5.5 host is often an overlooked component when working with VMware NSX.

If you have one of the Broadcom NICs which supports VXLAN offload and uses the bnx2x driver, it is important to choose the correct driver version. Not all versions are created equal!

If you look through the release notes provided with all versions of the bnx2x driver download on the VMware website, it will show you a full history of changes and enhancements made to the driver.

Scripting: NSX-v – Displaying Security Group Members

Fri, 24 Apr 2015 00:00:00 +0000

Often when working in customer environments, there is a requirement to define security group members which are not virtual machines within the visualised environment. To achieve this, these security group members must be defined as an IP Set.

The IP Set can then be included as an object in a security group.

However, within the UI, it doesn’t show you all the non-vm members of a security group.

vRA: Network and Security Inventory Data Collection Errors

Thu, 26 Mar 2015 00:00:00 +0000

Whilst working on a vCloud Automation Center integration with NSX-v this evening, I noticed a strange error which had appeared in the logs in regards to the data collection for Network and Security Inventory.

The Network and Security Inventory data collection was displaying the time the data collection last completed, which looked fine to me, as it was inline with what I was expecting, but the status was showing as Failed.

NSX-v: Controller Packet Capture

Wed, 18 Mar 2015 00:00:00 +0000

So following on from my previous post (NSX-v: ESG Packet Capture), today we run through how to do a packet capture on one of your NSX Controllers.

Why would you want to do this I hear you ask?

Well I had a situation recently where I had some unexplained behavior and I needed to make sure that a particular packet was physically arriving at the controller as it had to pass through several VRFs and a FW or two.

NSX-v: ESG Packet Capture

Mon, 16 Mar 2015 00:00:00 +0000

Whilst troubleshooting at a client today, I needed to perform a packet capture on one of the Edge Services Gateways in the environment. Performing a packet capture is often very helpful in diagnosing a range of different issues.

To kick off a packet capture you can jump on the console of the ESG or like I am doing in this example, open up an SSH session to the ESG.

You will need to know what interface to run the capture on, so run the following command to list out all the interfaces (for ease of reading I have removed all the interfaces that were showing down/down from the output)

NSX-v Quick Tip - Where is my NSX Manager?

Wed, 25 Feb 2015 00:00:00 +0000

So you’ve just installed NSX Manager and registered vCenter, but when you log in to the vSphere Web Client, you cannot see your NSX Manager. Where did it go?

When deploying NSX-v, one of the initial steps involves registering vCenter server with NSX Manager. The section in the NSX 6.1 Installation guide reads as follows:

Register vCenter Server with NSX Manager

You must login to the NSX Manager virtual appliance to register a vCenter Server and review the settings specified during installation.

Scripting: NSX-v - Security Groups with Dynamic membership

Thu, 19 Feb 2015 00:00:00 +0000

NSX-v allows the creation of Security Groups to group objects to be used in DFW rules and security policies. Each security group can have a mix of static and dynamic membership (If you want to get picky, you can also statically exclude objects). One of the possible ways to dynamically include members into the security group is to match on a security tag. This allows a VM to have security tags assigned to them, and based on the security tag, a VM can be dynamically added as a security group member.

Scripting: NSX-v - Importing Checkpoint Objects

Fri, 06 Feb 2015 00:00:00 +0000

When faced with migrating from a Checkpoint installation to a VMware NSX-v installation, at some point someone is going to need to get all the objects across to the new firewall platform. I was recently asked to help add some functionality to a script to take some data from an odumper csv output and pump it into NSX-v.

The following script is a bit of a hack I put together. The script is based off one written by Brett Drayton (www.brettdrayton.com) in this post.

Scripting: Querying NSX-v service objects by port

Thu, 05 Feb 2015 00:00:00 +0000

When using the NSX-v distributed firewall (DFW) have you ever need to find out if a service has already been configured in the system for a particular port number?

Recently I was given a sample ruleset from a client to re-create in the DFW, and one thing that stood out was that when creating NSX firewall rules and faced with a random port number that needed to be configured for a rule, there is no easy way to find out if a service has already been configured using that port.