Configuring a Cumulus Switch to Send syslog info to vRealize Log Insight

Log Insight is a so much more than a syslog collector, however this post focuses on how to export syslog files from a Cumulus switch.  Today, it doesn’t appear so though Cumulus has a content pack for Log Insight but you may want to check the VMware Solution Exchange.  Until then syslog must be manually configured to export log files.  Fortunately the process is relatively straightforward.

The following steps must be completed:

1. The  switch will need an IP Address on the network where the Log Insight collector is located, this can be done by editing the /etc/network/interfaces file.   You can use cat, nano, or vi to verify that the switch has an interface on the same network as the Log Insight collector.

auto br-rack.3100
iface br-rack.3100
address 192.168.16.1/24

2. If you had to manually add an IP address, the networking service on the Cumulus switch must be restarted using the command service networking restart

3. After successfully restarting networking attempt to ping the IP address of the Log Insight server, in my environment it is 192.168.16.7
Note: Optionally you may wish to establish a Telnet session to verify that you can reach the syslog server.  To do so type telnet 192.168.16.7:514

4. Next configure syslog by creating a file named /etc/rsyslog.d/90-remotesyslog.conf and add the following content:

## Copy all messages to the Log Insight server at 192.168.16.7 port 514
*.*                             @192.168.16.7:514

5. We also want to ship switchd logs to the syslog server by creating a file named /etc/rsyslog.d/24-switchd.conf
## Logging switchd messages to Log Insight
$ModLoad imfile
$InputFileName /var/log/switchd.log
$InputFileStateFile logfile-log
$InputFileTag switchd:
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 5
$InputRunFileMonitor

if $programname == ‘switchd’ then @192.168.16.7:514

6. You may also find it useful to log information for clagd.  I created a file named 20-clagd.conf and added the following

## Logging clagd messages to Log Insight
:programname, isequal, “clagd” -/var/log/clagd.log

7. When finished type service rsyslog restart

8. Next go log in to Log Insight, the default user name is admin.

9. Click on the Interactive Analytics button and filter by source (192.168.16.1), if it has been successfully configured you should see several events from the switch.

Posted in Uncategorized | Leave a comment

Dell FTOS % Error: Port is in Layer-2 mode

I was recently configuring a Dell Force10 S Series S55 1/10GbE top-of-rack (ToR) switch and received the error message “Dell FTOS % Error: Port is in Layer-2 mode” when attempting to remove Hybrid Mode.

The Dell Knowledge Base Article How to configure interfaces in Layer 2 mode on Dell Networking Force10 Switches has a note at the very bottom that states, “Before hybrid mode can be removed from an interface all layer 2 configurations have to be removed. Including any VLANs as tagged or untagged.” Unfortunately it doesn’t provide any details on how to remove the existing configuration.

You are already connected to the switch if you are receiving the error message in the title – so no “How-to connect to the switch” instructions are being provided.

To remove an existing Layer 2 configuration the first step is to look at the current configuration of the port.  You can do this by typing show running-config or simply show run

Screen Shot 2015-12-31 at 1.19.44 PM

Once you have reviewed the current configuration disable the current Layer 2 mode (in my example, Hybrid Mode) enter configuration mode by typing config and typing the commands below. Undo from the bottom up.

First I connected to the interface by typing the following:

interface gigabitethernet 0/4

Next I shutdown the port, this is important since we will also be disabling spanning-tree.  It is a Best Practice to shutdown the port during configuration.

shutdown

Now we are ready to undo the existing Layer 2 configuration type in the following commands:

no spanning-tree 0 portfast

no switchport

no portmode hybrid

end

If you are done configuring the port don’t forget to re-enable it by typing:

no shut

The last step is to save the configuration, type:

copy running-config startup-config

Here is what it looks like from the console

Screen Shot 2015-12-31 at 1.25.51 PM

 

Posted in Uncategorized | Leave a comment

Create a VLAN on a Dell S55 Series Switch for VMware ESXi

Before you get started I would recommend that you get a list of the existing VLANs that have been created on the switch.  You can do this by typing show vlan

Pay particular attention to the Q Ports section.  In my case below, I have already created VLAN 100. VLAN 1 is the default VLAN for the switch.

Screen Shot 2015-12-31 at 2.07.51 PM

There are two types of Q Ports U for Untagged, and T for Tagged traffic.

I’ve connected my ESXi host physical network adapters (vmnic0 and vmnic1) to ports 0/48 and 0/49.   I am “tagging”  virtual machine network traffic at the Virtual Machine Port Group named TrippLite on my VMware Standard vSwitch (vSS).

Screen Shot 2015-12-31 at 2.18.23 PM

Any virtual machine that is connected to the Virtual Machine Port Group TrippLite will have it’s traffic “tagged” with VLAN 100 before it leaves the ESXi host.

Once the traffic arrives at switch port 0/48 or 0/49 it will be sent out any other ports on VLAN 100 – in this configuration those ports are 0/37 -40.

I connected a TrippLite Serial Port/Terminal Server to port 0/37 on the S55 switch.  Unfortunately, VLAN tagging is not supported on the TrippLite device which prevents me from having it’s traffic “tagged” with VLAN 100 before it arrives at the switch.  To ensure that the TrippLite device network traffic is on VLAN 100, I have configured port 0/37 so that it automatically puts “untagged” traffic on VLAN 100.

This allows my any virtual machine connected to the TrippLite port group to manage the TrippLite device.

To configure the switch I used the following commands:

config

interface Vlan 100

description TrippLite

tagged tengigabitethernet 0/48-49

untagged gigabitethernet 0/37-40

no shut

If you need to delete Vlan 100, type the following:

config

no interface Vlan 100

You may also find the Dell Tech Tips article How Do I Use FTOS to Configure VLANs? helpful.

Posted in Uncategorized | Leave a comment

VLAN 101: What are the benefits of using VLANs?

As VMware Administrators take a more active role in Network Design they begin to ask more questions regarding why, how, or when to configure specific network features.  Creating VLANs today has become a common practice, let’s find out why!

There are five primary reasons why VLANs are used today:

  • Cost
  • Security
  • Performance
  • Manageability
  • Availability

Cost

It seems as though everything we do in IT in someway revolves around cost savings.  We constantly hear industry terms such as TCO (Total Cost of Ownership) and ROI (Return on Investment).  Bound by budgetary constraints and the race to zero, reducing costs within an enterprise IT environment has become one of our primary responsibilities.

In the enterprise, rack-mounted Ethernet switches are typically available in 24 or 48 port configurations.  In the past, if you wanted to separate network traffic for the computers of multiple departments, say HR (Human Resources) and Sales, you would have to purchase two physical Ethernet switches.  If both departments had enough computers to occupy all of the ports on both physical Ethernet switches there wouldn’t be any cost savings associated with using VLANs.  However, what if HR had 10 computers and Sales had 12 computers?  Purchasing two 24-port Ethernet switches (One for each department) would waste 24 ports (assuming one uplink is used per switch).  By implementing two VLANs (VLAN ID: 101 – HR, VLAN ID: 102 – Sales) we can use a single switch and isolate each departments network traffic.

Security

VLANs logically separate network traffic preventing devices from listening to any network traffic on other Virtual Local Area Networks.  They also offer additional security by VLAN device assignment.

There are three common methods used to assign a device (computer, PC, printer, etc.) to a VLAN:

  • Port based – A switch port is manually configured to be a member of a specific VLAN(s). Any device connected to this port will belong to the VLAN.  Physical security such as restricted access to the location of the physical switch is required.
  • Protocol based – The Layer-3 protocol being carried by the frame is used to determine VLAN membership, this method is not commonly used today.
  • MAC based – the VLAN membership is based on the MAC address of the device.  This method offers additional security at the cost of increased management.

It is important to note that VLAN Security relies on proper configuration.  An improperly configured environment exposes the customer to exploits such as VLAN Hopping, VTP Attacks, MAC Attacks, and PVLAN attacks.  See SANS Institute Virtual LAN Security: weaknesses and countermeasures and Redscan – Ten top threats to VLAN security for more details.

Performance

Performance is increased by reducing broadcast traffic, leveraging L3 switch capabilities to achieve wire-speed routing between VLANs, and applying VLAN-based QoS policies.

On an Ethernet network, broadcast traffic is traffic that must be processed by every device on the same network segment.  In order to process the traffic the NIC driver interrupts the host CPU.  As a result, CPU utilization would be higher on hosts in a network segment with a significant amount of broadcast traffic.  According to a Fluke Networks Whitepaper VLAN Best Practices, “An average number of broadcasts should be 30 broadcasts per second, or less. While no officially sanctioned quantity is specified in standards documents, field performance monitoring suggests that broadcasts should not exceed about 30 broadcasts per second.”

Common broadcast traffic includes: ARP, DHCP, Routing Protocols (RIP v1), NetBIOS/SMB, IPX/SPX (SAP Broadcasts).

Using a “sniffer” such as Wireshark, you can view broadcast traffic on your network. Ethernet has a MAC level broadcast address (ff:ff:ff:ff:ff:ff), and an IP-level broadcast address (255.255.255.255).

Today there are features natively built into operating systems and switches that help reduce some broadcast traffic.  For example, Windows hosts use an ARP cache.  “Each dynamic ARP cache entry has a potential lifetime of 10 minutes. New entries added to the cache are timestamped. If an entry is not reused within 2 minutes of being added, it expires and is removed from the ARP cache. If an entry is used, it receives two more minutes of lifetime. If an entry keeps getting used, it receives an additional two minutes of lifetime up to a maximum lifetime of 10 minutes.”Microsoft TechNet Entries in the ARP Table of an ESXi host remain for 1200 seconds (20-minutes).

Ethernet Switches use MAC Learning to reduce the amount of broadcast traffic. “MAC learning allows the Ethernet switch to learn the MAC addresses of the stations in the network to identify on which port to send the traffic. LAN switches normally keep a MAC learning table (or a bridge table) and a VLAN table. The MAC learning table associates the MACs/VLANs with a given port, and the VLAN table associates the port with a VLAN.” – Sam Halabi – Cisco Press: Metro Ethernet Services  Using the sh mac-address-table aging-time command on our Cisco Catalyst switch reveals a Global Aging Time of 300 seconds (5-minutes).

CPUs have become so powerful that the impact broadcast traffic has on a host is minimal today.

In order for frames to get from one VLAN to another, a Layer-3 device must route them. This device could be a traditional router, or a Layer-3 switch. Each router hop adds additional latency to the time it takes to get the frame from the sender to the receiver and can act as a bottleneck.  By using a Layer-3 switch, traffic never leaves the switch and could offer better performance.  Using a Layer-3 switch allows you to apply wire-speed routing between VLANs.

You can also use the QoS: Match VLAN feature to classify network traffic on the basis of the Layer 2 virtual local-area network (VLAN) identification number.  QoS—VLAN Tag-Based feature can then be leveraged to apply a single QoS policy, referred to as a VLAN-group policy, to a group of IEEE 802.1Q VLAN subinterfaces.

Manageability

Let’s say you wanted to use of a single subnet (192.168.15.0) for all of the printers at the Corporate Office.  In order to do so, each printer on the subnet would need to be in the same broadcast domain. This would require a dedicated switch for the printers, which wouldn’t be too challenging.  However, if the Corporate Office had five floors, you would now need a switch on each floor for just the printers. These switches would need to be connected to each other to extend the network to each floor. Using a VLAN would allow the printers to be connected to the same switches as other devices on the network.  Fewer switches = less management.

According to Cisco, VLANs also improve IT staff efficiency. “VLANs make it easier to manage the network because users with similar network requirements share the same VLAN. When a new switch is provisioned, all the policies and procedures already configured for the particular VLAN are implemented when the ports are assigned. It is also easy for the IT staff to identify the function of a VLAN by giving it an appropriate name.”  – Cisco Networking Academy Introduction to VLANs

Switch features such as VLAN Trunk Protocol (VTP), make it easy to distribute VLANs across a physical network environment.

Availability

The amount of broadcast traffic generated today is significantly less than it has been in the past.  Protocols such as IPX/SPX, RIP Version 1, and NetBIOS aren’t as common today.  Networking technologies such as vSphere NSX also reduce broadcast traffic through features such as ARP suppression. Furthermore, host CPUs have become so powerful that the additional task of handling broadcast traffic is insignificant host performance.

So why bother with VLANs?  Think “Failure Domain”  VLANs offer the ability to reduce the size your failure domain.  If a device has a damaged Network Interface Card (NIC) it may broadcast enough traffic to impact every host in the VLAN.  For example, if a desktop computer in the Sales VLAN had a bad NIC that had become “chatty” and created a storm of broadcast traffic, only the devices on the Sales VLAN would be impacted.

How do I configure a VLAN?

Before we get started we need to briefly define the difference between an “access” port, and a “trunk” port.

  • An access port can have only one VLAN configured on the interface; it can carry traffic for only one VLAN.  Traffic that reaches an access port is
  • A trunk port can have two or more VLANs configured on the interface; it can carry traffic for several VLANs simultaneously.

Now, let’s use our previous example where the HR and Sales departments are sharing a single 24-port Ethernet switch, ten (10) HR computers and twelve (12) Sales computers:

  • Ports 1-12 are connected to the ten (10) computers for the HR department
  • Ports 13-22 are connected to the twelve (12) computers for the Sales department
  • Port 24 is the “uplink” port for Internet Access

In order for the HR department computers to communicate with each other and the internet we will associate ports 1-12 and 24 with VLAN 101. Ports 1-12 would be designated are configured as “access” ports, while port 24 would be designated as a “trunk” port.

Next we will create a second VLAN by associating ports 13-22 and 24 with VLAN 102. In this case, ports 13-22 would be designated as “access” ports while port 24 would be designated as “trunk” port.

Posted in Uncategorized | Leave a comment

VMware NSX Document Library

This is a collection of all of the Public Documents that are available from VMware for NSX.

NSX Administration Guide

NSX Installation and Upgrade Guide

NSX for vSphere Getting Started Guide – July 2014

VMware NSX: The Platform for Network Virtualization Datasheet – June 2014

The VMware NSX Network Virtualization Platform Technical White Paper VMware Solutions: Designed for Early and Ongoing Success – June 2014

VMware NSX for vSphere (NSX-V) Network Virtualization Design Guide – Version 2.1 – December 2014

VMware NSX for Multi-Hypervisor (NSX-MH) Network Virtualization Design Guide – Version 4.2 – November 2014

VMware NSX for vSphere (NSX-V) Network Virtualization Design Guide – Version 2.0 – March 2014

VMware NSX DFW Policy Rules Configuration Technical White Paper – VMware NSX for vSphere, Release 6.x – September 2014

Data Center Micro-Segmentation White Paper – A Software Defined Data Center Approach for “Zero Trust” Security Strategy – June 2014

VMware and Arista Network Virtualization Reference Design Guide for VMware vSphere Environments – Deploying VMware NSX with Arista’s Software Defined Cloud Networking Infrastructure – March 2014

Connecting Physical and Virtual Networks with VMware NSX and Juniper Platforms – August 2014

VMware and Brocade Network Virtualization Reference White Paper – August 2014

VMware NSX Network Virtualization Design Guide – Deploying VMware NSX with Cisco UCS and Nexus 7000 – February 2014

Network Virtualization with Dell Infrastructure and VMware NSX Reference Architecture – August 2014

NSX-v 6.1 – Security Hardening Guide (Community version 1.2) – November 2014

Securing VMware NSX – May 2014

VMware NSX for vSphere Proof-of-Concept Requirements Checklist – Version 1.6 – October 2014

Posted in Uncategorized | Leave a comment

vSphere NSX: Unable to Remove or Delete Logical Switch

Screen Shot 2014-12-05 at 12.25.00 AM

Notification – The last operation failed for the entity with the following error message.  The object <Logical Switch Name> is in use, this operation cannot be performed.  Remove all configuration referring to this object and retry operation.

I recently working with a customer who was unable to remove or delete a logical switch that had been created.  The first step was verify that no virtual machines were connected.  To do so, I did the following:

Remove all Virtual Machines and NSX Edges from the Logical Switch

  • Click the vSphere Web Client Home icon.
  • On the vSphere Web Client Home tab, double-click on the Networking and Security icon.
  • In the left navigation pane, select Logical Switches.
  • Double-click on the Logical Switch you are attempting to delete.
  • Select the Related Objects tab, then click on the Virtual Machines button.
  • Note: If you have any remaining virtual machines connected to the Logical Switch you are attempting to delete, migrate them to another Logical Switch.
  • Click on the Manage tab, then click on the NSX Edges button.
  • Note: If you have any connections (interfaces) to an NSX Edge you will need to remove them.

In my case the customer had migrated the virtual machines, but had neglected to remove the connection to the NSX Edge.  Once the NSX Edge was removed, we were able to successfully delete the logical switch.

Posted in Uncategorized | Leave a comment

On choosing VMware NSX or Cisco ACI by Brad Hedlund

Brad Hedlund, Engineering Architect with the CTO office of VMware’s Networking and Security Business Unit (NSBU), posted an article to help customers who are evaluating both VMware NSX and Cisco ACI.  You can read more here.

Posted in Uncategorized | Leave a comment