Archives For Networking

Ran into a networking-related issue the other day, and while it wasn’t anything new or necessarily exciting, it taught me a lesson about cut-through switching vs store and forward switching.  Sharing this one for my fellow non-Network engineers:

Symptoms:

Realized that (from the Server Engineer perspective) we were seeing frequent packet receive CRC errors/drops on multiple server interfaces (ESXi hosts, HP virtual connects) in a particular datacenter row.  In this case, these devices were connected to Nexus 5Ks, either directly or via Nexus 2Ks.

Lesson Learned:

Depending on the network switch, it can either use the store and forwarding method or the cut-through method to forward frames.  I believe Cisco Catalyst switches and Cisco Nexus 7000 series typically use store and forwarding.  I believe the Cisco Nexus 5000 and 2000 series use cut-through.  (Network engineers: please correct me if that is incorrect)

Store and Forwarding – When there are CRC errors generated, ie. from a device network interface with a bad cable, this corruption will get detected before the frames are sent and would not get propagated across the network or “switching domain”.

Cut-Through – When there are CRC errors generated, ie. from a device network interface with a bad cable, these corrupted frames may get propagated because cut-through switches aren’t able to verify the integrity of an incoming packet before forwarding it.  This is good because it reduces switching latency, but it can cause CRC errors to “spread like the plague”

Resolution:

Worked with a friendly Network Engineer to identify the device and interface that was the source of these propagating errors, and then replaced the cable.

There were a couple of good sources of info that were very helpful for this one:

 

 

Advertisements

During the setup of a new vSphere cluster, I had to troubleshoot an issue that was causing latency for the NFS datastores. This new vSphere cluster was attached to a newly setup NetApp storage array with essentially no workloads yet hosted on it.

One of the first symptoms of the latency was noticed when browsing the NFS datastores in the vSphere client GUI. After clicking on Browse Datastore, and then clicking on any VM folder, the window would display “Searching datastore…” and often take 40-50 seconds to display the files. Further testing with the NFS datastores confirmed that slowness was also seen when performing file copy operations, or certain disk intensive operations in the guest OS.

Several weeks were spent troubleshooting and working with vendors to try and determine the cause. It was found that when the configuration on the NetApp storage array side (and switches) was changed to use access ports versus trunk ports, the issue went away. In addition, the issue did not occur when one of the hosts and NetApp storage array were connected to the same datacenter switch. No jumbo frames were in this equation.

The cause of the issue was found to be a conflict between the default behavior of NetApp when using VLAN tagging and the Nexus core switch QoS configuration. By default, NetApp assigns a default CoS value of 4 when using VLAN tagging (trunk ports). This caused the NFS storage traffic to get placed in a queue on the router that was limited in terms of bandwidth. A workaround was implemented on the switches for the storage array interfaces that essentially changed the CoS value to fit with the network configuration in the environment.

Here are some links that helped to connect the dots when researching the issue:

https://kb.netapp.com/support/index?page=content&id=3013708&actp=RSS

http://keepingitclassless.net/2013/12/default-cos-value-in-netapp-cluster-mode/

https://communities.cisco.com/message/137658

http://jeffostermiller.blogspot.com/2012/03/common-nexus-qos-problem.html

http://www.beyondvm.com/2014/07/ucs-networking-adventure-a-tale-of-cos-and-the-vanishing-frames/

For those with networks that use Cisco OTV with Nexus 7Ks to extend Layer 2 connectivity between sites, be aware that there is a bug (CSCuq54506) that may cause brief network connectivity issues for VMs that are vMotioned between the sites. The first symptom you may notice is that a VM appears to drop ping or lose connectivity for almost 1-2 minutes after it is vMotioned between sites.  Following a vMotion, a destination ESXi host will send RARP traffic to notify switches and update the MAC tables. When this bug occurs, the RARP traffic/updates basically don’t make it to all of the switches at the source site.  (Note: Since not having portfast enabled on the source or destination host switch ports can cause symptoms that may look a bit similar, it’s a good idea to confirm portfast is enabled on all of the ports.)

Troubleshooting that can be used from the VMware side to help identify if you are hitting the bug:

  • Start running two continuous pings to a test VM:  one continuous ping from the site you are vMotioning from, and one continuous ping from the site you are vMotioning to.
  • vMotion the test VM from one site to the other.
  • If you see the continuous ping at the source site (site VM was vMotioned from) drop for 30-60 seconds, but the continuous ping at the destination site (site VM was vMotioned to) stays up or only drops a ping packet, then you may want to work with Cisco TAC to determine if the root cause is this bug.

 

When migrating virtual or physical servers from one data center to another, especially if you are moving from Cisco Catalyst to Nexus switches, it’s helpful to be aware of the concept of Proxy ARP.  Here is a link to a Cisco article that explains Proxy ARP:

http://www.cisco.com/c/en/us/support/docs/ip/dynamic-address-allocation-resolution/13718-5.html.

If Proxy ARP is enabled on a switch/router, it can hide or mask misconfigured default gateways/subnet masks on servers.  A switch/router with this setting enabled can help servers reach devices in other subnets, even if the configured default gateway on a server is incorrect. Once the misconfigured servers are moved to network equipment that has Proxy ARP disabled, the servers will no longer be able to communicate with devices in other subnets.  Proxy ARP is enabled by default on Catalyst switches and disabled by default on Nexus switches.  Make sure to review the Proxy ARP settings in both the originating and destination data center.  If this setting will be disabled at the destination site, run a script to check default gateways and subnet masks on servers before beginning a migration.

…written from the perspective of a Virtualization Engineer.  A very special thanks to Networking Guru @cjordanVA for being a key contributor on this post.

Overlay Transport Virtualization (OTV), which is a Cisco feature that was released in 2010, can be used to extend Layer 2 traffic between distributed data centers.  The extension of Layer 2 domains across data centers may be required to support certain high availability solutions, such as stretched clusters or application mobility.  Instead of traffic being sent as Layer 2 across a Data Center Interconnect (DCI), OTV encapsulates the Layer 2 traffic in Layer 3 packets.  There are some benefits to using OTV for Layer 2 extension between sites, such as limiting the impact of unknown-unicast flooding.  OTV also allows for FHRP Isolation, which allows the same default gateway to exist in the distributed data centers at the same time.  This can help reduce traffic tromboning between sites.

When planning an OTV implementation in an enterprise environment with existing production systems, here are a few things to include in the testing phase when collaborating with other teams:

  • Setup a conference call for the OTV implementation day and share this information with the Infrastructure groups involved in the implementation and testing, ie. Network, Storage, Server, and Virtualization engineers.  This will allow staff involved to easily communicate when performing testing following the change.
  • Test pinging physical server interfaces by IP address at one datacenter from the other datacenter, and from various subnets.  Can you ping the interface from the same site, but not from the other site? (Make sure to establish a baseline before implementation day.)  Is your monitoring software at one site randomly alerting that it cannot ping devices at the other site?
  • If your vCenter Server manages hosts located in multiple data centers, was vCenter able to reconnect to ESXi hosts at the other datacenter (across the DCI) after OTV was enabled?
  • If you have systems that replicate storage/data between the data centers, test this replication after OTV is enabled and verify it completes successfully.

 

Be aware of a couple of gotchas:

ARP aging timer/CAM aging timer – Make sure to set the ARP aging timer lower than the CAM aging timer to prevent traffic from getting randomly blackholed.  This is an issue to watch out for if OTV is being implemented in a mixed Catalyst/Nexus environment, and will not likely be an issue if the environment is all Nexus.  The default times for the aging timer depend on the Cisco platform.  The default for a Catalyst 6500 is different than the default for a Nexus 7000.

Symptoms of an aging timer issue:  You will more than likely see failures during the pings tests mentioned above or you may see intermittent issues with establishing connectivity to certain hosts.

MTU Settings – Since OTV adds additional bytes to IP header packets and also sets the do not fragment “DF” bit, a larger MTU will need to be configured on any interfaces along the path of an OTV encapsulated packet.  Check the MTU settings prior to implementation, and again if issues arise when OTV is rolled out.  If MTU settings were properly configured, consider rebooting the OTV edge devices as a troubleshooting step if issues are encountered to verify the MTU settings actually applied properly and did not get stuck — (it’s happened).

Symptoms of an MTU-related issue:  If you have a vCenter server in one data center that manages hosts at the other datacenter, it may not be able to reconnect to the hosts at the other data center.  Storage replication may not complete successfully after OTV has been enabled.