Shut Up and Take My Money

This should be a required item in every engineer’s toolkit.

http://www.thrillist.com/drink/nation/kirin-frozen-beer-slushie-maker-on-sale

Misconfigured Servers and Proxy ARP

I worked a migration this weekend from an older Catalyst 4500 to a Nexus 7004 and ran into a (what turned out to be) a very simple issue to fix, but one that confused the hell out of me trying to troubleshoot it.

The config on the old switch was very basic:  (I’ve included only the related vlan info):

interface Vlan1
ip address 10.68.1.1 255.255.255.0
ip pim sparse-dense-mode
ip policy route-map POLICY10
ip igmp join-group 239.255.255.255

interface Vlan109
ip address 10.68.9.1 255.255.255.0
ip helper-address 10.68.1.4
ip helper-address 10.68.1.22
ip wccp web-cache redirect in
ip pim sparse-dense-mode
ip igmp join-group 239.255.255.255


Looks relatively simple, right? Right. The topology was not complicated…it should be pretty cut and dry.

The symptoms we were having are described below:

• A user in any vlan directly attached to the 7K could not access certain resources on Vlan 1, but some resources on that same vlan were fine. Keep in mind that users had no connectivity issues to these same devices prior to migration.
• We could reach these resources in vlan 1 (that were not reachable from directly attached vlans) from a remote WAN site with no problem.
• A ping from the N7K worked fine by default (source from the same vlan interface as the destination address), but if you sourced it from another vlan, it failed.

Hmph…weird. Any ideas…care to wager a guess?

So here are some of our findings after some additional digging:

• Some servers had a subnet mask of 255.255.0.0
• Some servers had a default gateway of a device that doesn’t exist!?! (but still within the same /16 space)
• Some servers had the proper subnet mask

Do you detect a theme yet? Have a suspect in mind? (Well, two suspects)

If you guessed proxy-arp, you win a shrubbery. If you guessed proxy-arp and server guys that didn’t apparently know understand subnetting, you win two shrubberies.

Proxy-arp, if you are not familiar with it,  is a technique in which one host, usually a router, answers ARP requests intended for another machine. By “faking” its identity, the router accepts responsibility for routing packets to the “real” destination. Proxy ARP can help machines on a subnet reach remote subnets without the need to configure routing or a default gateway. Proxy ARP is defined in RFC 1027 (Definition from: here)

Proxy-arp is disabled by default on NX-OS. On the version of IOS that the 4500 was running, proxy-arp was enabled by default.

For example, one server we were investigating in Vlan 1 had IP address of 10.68.1.21/16. A device in Vlan 9 had an IP address of 10.68.9.5/24. See the problem here?

The server in vlan 1 thinks that the host in vlan 9 is in the same subnet, and since the N7K won’t respond to proxy-arp, the host in Vlan 1 won’t be able to communicate with the host in vlan 9, but it CAN reach devices outside of it’s own /16 subnet. That’s why communication to another branch worked fine, but communication to any directly attached vlan failed. (That is, any directly attached vlan within the same subnet as the server).

There’s a good discussion on the Cisco Support Forums about this (found here), so I won’t go into how the host builds the frame and the related source and destination MAC addresses, but if you want additional clarity, please let me know.

It was really a frustrating issue, and I wish I had thought of this sooner, but it’s one of those cases that you don’t run into it quite often enough to keep it fresh in your mind.

Authenticate Cisco Devices Against Microsoft NPS Using Wildcards

In many deployments we do as consultants, our customers may not have all of the pieces of the puzzle that we need to finish our project. One of those missing pieces is frequently a RADIUS/TACACS+ server. We’ll skip the RADIUS vs. TACACS+ discussion for now, and assume we’re setting up a Microsoft Server with the NPS role. Since a lot of customers are a Microsoft shop already, and NPS is part of the OS, the customer doesn’t have to shell out the cash for Cisco ACS.

One of the things that I would always find annoying (but not annoying enough to spend the time to research – until now of course) is that you had to specify each RADIUS client that you wanted to authenticate. When there’s 5 or 10 devices, that’s not a big deal, but when you have hundreds of devices, it gets old quickly. Here’s a way (I’m sure there are others, but this is how I solved the issue), to use NPS to authenticate any number of devices without having to manually specify each client individually.

We’ll assume that you have already added the NPS role to the server. (If not, click here) The first step is to add a RADIUS client. In our case, we’ll be adding subnets instead of a specific client IP. Adding a specific subnet works well if your customer has a dedicated management vlan or a consistent addressing scheme, but this customer did not, so we had to include all of the RFC 1918 address space.

Open the NPS console, and expand “RADIUS Clients and Servers” then right click “RADIUS Clients” and choose “New.”

The friendly name doesn’t matter…I used the subnet as the name. The IP address of the client in this case, will also be the subnet. Enter the shared secret and click OK. This will cause the NPS server to process requests from any device originating in the 10.0.0.0/8 subnet. I also added 172.16.0.0/12 and 192.168.0.0/16 as “RADIUS Clients” as well. Here’s the results:

Now that we have our RADIUS clients identified, we need to create a Connection Request Policy. So expand “Policies” then right click “Connection Request Policies” and then “New”

• Policy name: Whatever you want to call it. I chose to call mine “Accept Request From Any Device” Leave the type of network access server as default
• Add a condition: Scroll down about halfway and add “Client IPv4 Address” – here’s the import part…in the address field, specify *.*.*.* which is the wildcard for any IPv4 address.
• Accept the defaults for the rest of the wizard.

Here’s what we end up with:

Now we need to tell NPS who should be able to log into these devices. For this, we need to add a network policy. Right click “Network Policies” and create a new one. I called mine “Network Admins.”

• Type of access server – leave as unspecified. Click Next.
• Add a condition: Windows Groups – then choose the AD group where your admin user accounts live. Click Next.
• Leave as “Access Granted” under the “Specify Access Permission” screen
• IMPORTANT: On the “Configure Authentication Methods” screen, we need to check “Unencrypted authentication (PAP,SPAP)” – Click Next
• Accept the defaults for the rest of the prompts.

So now we have created a configuration that will authenticate any network device (that has the correct shared secret, of course) against an AD group of your choosing. In our case, any AD user that was in the group had access to the devices. We didn’t differentiate between privilege levels or anything like that.

You can read more about regex matching in NPS over here: Using Regular Expressions in NPS

NOTES: A follow up tweet from @verbosemode indicates that there is a cap on the number of Radius clients (50)  that  Windows Server 2008 R2 Standard supports. Also, apparently you have to have Enterprise Edition to use CIDR ranges. More info can be found here: http://technet.microsoft.com/en-us/library/dd365355(v=WS.10).aspx

I Think We’ve All Been There…

How often do customers ask you for this…

Per-SSID VLANs on Meraki Access Points

At the company I work for, we’re starting to see more and more Meraki wireless deployments, and the documentation on Meraki’s KB is lacking in this area. So, I threw this post together on how to configure Per-SSID VLANs as a reminder for myself, and hopefully it saves someone a little time down the road.

This assumes you have access to your Meraki Dashboard already and have the APs added to your inventory. The ports connected to the APs need to be trunks.

In this instance, we’ll add a guest wireless vlan which is segmented from the rest of the network.

1. Create the vlan as you would normally on the switch. Nothing special here…
2. Log into the Meraki dashboard and go to “Wireless –> Access Points” and select the APs for which you want the VLAN/SSID to be active (assumed to be all of them in this case) and click “Tag”

3. Type a new tag name or select an existing tag. In this case, I already have a tag “GUEST-ENABLED”. Click Add.

You can now see that all the APs have the GUEST-ENABLED tag applied

4. Click “Wireless –> SSIDs –> Access Control”
5. Choose the SSID you want to modify from the drop down box at the top. (Or create a new SSID if you havent already)
6. Under the “Addressing and Traffic” section (about 2/3’s of the way down the page), select “Use VLAN Tagging” next to the “VLAN tagging” option. (NOTE: The APs must be in bridge mode. That is, clients are part of the LAN  and either the core switch or some other device must hand out DHCP – as opposed to using the Meraki DHCP)

7. Click “Add VLAN” under the “VLAN ID” and enter the Tag you created previously. It must match exactly. Enter the VLAN for that SSID to use, in this case, vlan 170.

8. Click Save to apply your changes and wait 1-2 minutes for them to propagate to the Meraki Cloud

 

Cisco ASA 5500x IPS Management IP Configuration

The ASA 5500x series (other than the 5585x) uses the management 0/0 port on the ASA itself as the port it uses to communicate with the network.

You have to have the management port up in order to reach the IPS sensor management IP, but you don’t necessarily have to have an IP address on the ASA m0/0 port itself.

For example, this configuration works fine:

ASA Management Interface

interface Management0/0
 nameif management
 security-level 100
 no ip address

IPS Management Interface

service host
network-settings
host-ip 172.16.0.251/24,172.16.0.1

Just something to maybe save someone a few minutes…

Cisco ASA IPS Sensor Upgrade Time

For those impatient people among us (me included) …just a note in case anyone is searching for this…it appears that it takes roughly 10 minutes for an IPS sensor in an ASA 5525x to upgrade.

I must’ve killed it two or three times after 5 minutes or so thinking there was a problem…there wasn’t. Awesome.

My VLANs…Where Did They Go?

In my line of work, we tend to do network assessments fairly regular basis. Customers will request them in order to help better understand what’s wrong with their network and how to improve it. This entails digging into the customer’s network and hopping from device to device and reviewing configurations.

One of the things that I cite quite often are VTP misconfigurations or not following best practices. Since many customers aren’t intimately familiar with VTP and how it works, a lot of times I’ll end up in discussions on the some of the finer points of VTP, and one of those finer points usually ends up being how a misconfiguration can blow away all of your VLANs. I’ll usually mention that even as a VTP client, a switch can still update the VLAN info on a VTP server, given the correct configuration. However, I can never seem to remember what the exact configuration is for this scenario to occur, so here it is.

1. A trunk link must be present (either statically or dynamically created)
2. The VTP client switch must have the same VTP domain name
3. The VTP client switch must have the same VTP password
4. The VTP client switch must have a higher revision number than the rest of the network

Given these circumstances, a VTP client switch can update your VTP servers with whatever VLANs the client has on it…and that may be all of the same VLANs plus a few new ones, or just the default VLAN!