Get Your WLAN Ready for Carrier Wi-Fi Calling

To follow-up my last post where I expressed concern about marking cellular carrier Wi-Fi calls with the proper QoS class, I’m please to see that Cisco will include application signatures for Wi-Fi Calling in it’s upcoming AVC Protocol Pack 15 update. Other vendors should follow suit.

Keep in mind that changing the classification of VoWiFi packets on the WLAN only affects downstream packets from an AP. Upstream is up to the client.

Do Wi-Fi Calling smartphones mark upstream VoWiFi packets for the WMM AC_VO queue? If so, that could pose a problem in high density networks, as a large group of these clients will demand immediate airtime and limit other clients’ access to the medium. Imagine a future where Apple, AT&T, and Verizon all support WiFi Calling and enable it by default to off-load data from their LTE networks. This could happen as early as 2016. High density networks that were designed for best effort data suddenly have to deal with these demanding clients who can dominate the 802.11 contention window. Wireless engineers that haven’t handled voice on the WLAN in the past will now be forced to deal with it.

The first thing to consider is making WMM Admission Control (WMM-AC) mandatory for voice to prevent voice clients from dominating a channel’s contention window. I suggest doing this before all the major cellular carriers enable Wi-Fi calling and these clients show up en masse on your WLAN. To date, the Wi-Fi Alliance has certified 77 smartphones for WMM-AC. The elephant in the enterprise room is the iPhone, which lacks WFA certification for WMM-AC, although it may still support it. I suspect that most newer clients that support WMM probably are WMM-AC capable as well, but that is just a hunch. A client that doesn’t support WMM-AC just won’t gain access to the AC_VO queue, but it can still pass voice traffic without higher priority.

Wireless engineers may also choose to tweak the default WMM AC_VO AIFSN and contention window min/max settings to give these packets less airtime priority. Given today’s PHY rates, that may not cause a significant impact on the performance of these applications when channel utilization is low to moderate.

The goal will be to strike a balance between voice performance without significantly degrading the performance of best-effort data clients. WLAN’s that were designed with voice in mind will have an advantage as they provide higher minimum SNR and therefore higher minimum PHY rates, as well as better roaming characteristics. If your WLAN doesn’t provide fast roaming now, expect it to be a requirement in the future. (Queue the lack of client support for 802.11r rant, with a hat tip to Apple)

What other approaches are out there for dealing with a sudden increase in voice clients?

Update 10/9/2015

Yesterday, AT&T enabled Wi-Fi calling for iPhones on its network. AT&T is by far the largest carrier in the US to enable this feature, so expect to see an increase in Wi-Fi calling on your WLAN soon. Twitter user @wirelessguru posted this packet capture, which shows an iPhone with service from AT&T sending Wi-Fi voice packets with WMM AC_VO QoS markings (and some odd layer 3 markings as well).

The time is upon us to flip the WMM-AC mandatory bit for the voice queue, and consider enabling AVC QoS markings for downstream Wi-Fi calling traffic if available.

Advertisements

Layer 7 Firewalls and QoS on the WLAN

Several WLAN vendors offer layer 7, or application layer, firewalls and quality of service tools. The feature has different names depending on the vendor (Application Visibility and Control, Layer 7 Visibility, AppRF, etc.), but they all try to do the same thing. These tools work at the application layer to identify packets for processing through firewall or QoS rules, which is very useful in today’s world where so many applications are served over the Internet on ports 80 and 443. Traditional stateful firewalls aren’t much use when you want to say, ratelimit Netflix traffic.

At first, you may be tempted to identify all the applications commonly used on your WLAN and assign each of them to a QoS queue. Mission-critical applications get higher priority while social networks and video streaming services are deprioritized. Mark everything!

However, like other features of enterprise gear, while it’s tempting to turn it on and go nuts, you should use restraint, and here’s why: Layer 7 traffic analysis can be very CPU intensive, so the more layer 7 rules in your ACL’s, the more work the AP or controller must do to enforce them. That can result in a performance penalty during high traffic periods. At least one WLAN vendor tacitly acknowledges this by providing an undocumented “Turbo Mode” that will “disable QoS policies and improve Wi-Fi performance.”

Also keep in mind that layer 7 traffic analysis is a bit more of an art than the hard science of stateful packet inspection. Traffic flows are compared to vendor proprietary signatures for proper identification, and that’s not always 100% reliable. An application update or backend infrastructure change may require the development of a new signature for proper identification. WLAN vendors need to provide customers with regular updates to their application signature databases to ensure proper identification is occurring.

With that aside, what are some good uses of layer 7 firewalls and QoS?

Background Data Hogs

RF is a shared medium and as such it is often a bottleneck in busy networks. Software update utilities that run in the background on client machines can be problematic when there are a lot of stations sharing a channel. These applications like to all run at the same time, triggered by events like shifting from a 4G connection to Wi-Fi or right after a machine boots up.

In a school environment, this could happen during first period when everyone pulls out their Chromebook and they all automatically check for updates in the background, while at the same time students’ iPhones notice the Wi-Fi connection and decide now is the time to download that massive iOS update. The WLAN can slow to a crawl without any end-user interaction other than walking in the door.

I think this is where layer 7 QoS shines. By marking Apple Software Update and Chrome OS update packets for the background queue (AC_BK), for example, other applications that users are interacting with in the foreground of their clients take priority on the network. Of course, you will customize these rules to your IT environment. A Microsoft shop will want to do this with traffic to their WSUS server, etc. If you have a lot of iOS clients, iCloud traffic is one to look out for. Dropbox might be a big one too. You may want to consider deprioritizing antivirus updates as well, as these applications sometimes update quite frequently in the background.

Chrome and Chrome OS Updates

Google Chrome LogoIncidentally, despite the overwhelming popularity of Chrome OS in K12, I am unaware of any vendor that provides application signatures for Chrome OS updates. If you can define custom applications within your WLAN (I know that Aerohive and Meraki can do this), use these URL’s to identify Chrome OS updates (these also cover Chrome web browser autoupdates for Windows/Mac/Linux):

https://dl.google.com
http://dl.google.com
https://cache.pack.google.com
http://cache.pack.google.com
https://tools.google.com
http://tools.google.com

Or, if you are really strapped for throughput, use firewall rules to block these applications altogether on the guest network, for example. If WAN throughput is really limited you may need to consider end-to-end QoS all the way to your WAN circuit. Most enterprise WLAN gear can translate WMM QoS markings to 802.1p or DiffServ markings on the ethernet network, but remember to configure QoS on every networking device between the AP and WAN. Do packet captures to confirm your configuration is working.

Recreational Applications

Is it standardized testing season and you are worried that students’ use of Pandora and Netflix is affecting your WLAN performance? No need to go to superiors or committees and ask to have them blocked. That’s a bit draconian anyway. Deprioritize those applications with layer 7 QoS rules.

Malicious and Illegal Applications

Stop bad traffic at the AP or controller before it gets to your content filter. This provides an extra layer of filtering and reduces the traffic the content filter must process. If you don’t enforce station isolation, it can also can block some LAN attacks that would otherwise not reach your content filter. At school, peer-to-peer file sharing applications like Bittorrent, proxy applications, Tor, and shady VPN services are all good candidates layer 7 firewall blocking. Just make sure your firewall rules comply with organizational policy.

Looking Ahead

RF design is the most important factor in meeting the needs of voice-over-Wi-Fi applications, and properly configuring QoS for the enterprise VoIP system has always been important as well. But now we’re seeing users making VoWiFi calls via their cellular carrier. Layer 7 traffic analysis can be used to identify this new traffic and push it to the proper WMM queue (AC_VO).

Going forward these tools might prove less effective as more and more network traffic is encrypted by default. In fact, all HTTP/2 traffic will be encrypted. The companies that develop the application signatures used by WLAN vendors have a challenge to do more with less. Our dependence on these products is increasing while at the same time it will become more difficult to identify application traffic on the network.

Wi-Fi Load Balancing Considerations

When deploying a WLAN it’s easy to fall into the trap of enabling features you might not need, just because, well, you paid for them and they are cool. Often times a KISS approach results in better performance, but hey, look at this cool new thing it can do it!

Load balancing is one of those features. While it seems harmless enough, there are some scenarios that can get you into trouble.

For the uninitiated, WLAN load balancing is a feature that encourages clients to associate with the least-loaded nearby AP. Typically, a client will attempt to associate with the loudest nearby AP, without regard to how many clients are already associated (most AP’s don’t share that information anyway, but some do by using the BSS Load element within management frames). Most load balancing algorithms work by suppressing probe and association responses from heavily loaded AP’s so that a client either won’t know that it is there, or it will fail to associate with it. Hopefully the client will then attempt to associate with a different AP that has more capacity available to clients.

There are a couple problems with this to keep in mind. The most important problem is one the affects all clients. The AP has a very different view of the RF environment than the client does. What a highly sensitive, enterprise grade AP is capable of hearing is quite different from what a low-cost, consumer-grade Wi-Fi chipset can, and they are of course not listening from the same location either. It gets worse if that client radio is part of a smartphone tucked into a pocket or purse. In this example, the AP may think it’s safe to ignore probe and association requests from that client because it’s aware of three other nearby AP’s that are less-loaded, but the reality is that the smartphone can’t hear any AP but the one that is ignoring it.

And not all load balancing algorithms do what you think they do.  Some operate by simply limiting the total number of associated clients an AP radio will accept, even some that are described as “airtime-based” from my experience. The problem here is that this doesn’t take into account the actual airtime utilization, the truest measure of the load on an AP radio. Often, the airtime utilization is quite low when an algorithm decides the AP is too loaded and should push clients elsewhere. Say 30 clients are associated to one of the AP’s radios. If they are all idle, there is still plenty of capacity for others to associate as well, as very little airtime is being used. Make sure you know exactly how your WLAN’s load balancing works. Test it to make sure it does what it claims it does, and set your limits high.

Here are some examples where load balancing can causes problems:

  • A high school classroom fills with students.  As they enter the room, their smartphones, which were already configured to join the WLAN, automatically roam to the loudest nearby AP. The teacher asks the students to get out their laptops as part of her lesson. The laptops now try to connect, but the nearby AP already has 30 smartphones associated to it, so it ignores the probe and association requests from the laptops. The best case scenario is that the laptops are able to associate with another nearby AP, albeit at a lower data rate than the louder AP. The worst case scenario is that client’s Wi-Fi radio drivers won’t budge, and continually fail to associate with the loudest AP (which is ignoring them), or the neighboring AP that the loaded-AP is trying to push new clients to is actually too distant for the new clients to hear. But the smartphones are all nearly idle, so it would have been better for the laptops to associate with the louder AP.
  • The school media center is used to store several carts of iPads. The iPads are not powered-down before being stored, so they all associate with the media center AP. Visitors to the media center have difficulty connecting to the network in the media center, because the algorithm believes it is heavily loaded and ignores requests to associate to the media center AP. The media center AP can hear another AP well, but most visiting clients in the media center cannot. The visiting clients cannot connect to the WLAN, yet in this case as well, the AP is actually not loaded at all. The iPads are completely idle and using almost no airtime.

Here is the where load balancing makes sense:

  • In areas where you can reasonably anticipate that a single AP radio may become overloaded, such as a cafeteria, gym, or performance space.
  • In areas where multiple AP’s are very close to one and other and create tightly overlapping coverage cells. This helps mitigate the problem of  clients and AP’s having a differing view of the RF.
  • Nowhere else. Only use load-balancing when both of the above criteria are met.

Yes, Hotspot 2.0 is the Future of Secure Guest Wi-Fi

Since first blogging about Hotspot 2.0 and its application to typical enterprise WLAN guest networks I’ve learned quite a bit more thanks to several helpful tweets from Dave Wright of Ruckus Wireless. Although the large majority of the focus of Hotspot 2.0 still seems to be on integration with cellular carriers for AAA services and all the complexity and exclusivity that entails, there are provisions for simpler, anonymous, and secure Hotspot 2.0 guest networks that are much closer to what the typical enterprise WLAN operator will actually deploy. As I’ve said before, authentication is not a priority for most WLAN operators on their guest network, but encryption certainly is.

Is it the holy grail of guest Wi-Fi? Maybe, but more on that after we look at the Wi-Fi Alliance Passpoint (Release 2) Deployment Guidelines. In all 61 pages of the document, there are these few paragraphs devoted to what I predict to be the most common use of Hotspot 2.0.

12. Free Public Hotspot 2.0-Based Hotspots 

Hotspot Operators may provide Hotspot 2.0-based free, public, hotspot service. In this particular service, Hotspot Operators have the need to ensure hotspot users have accepted the terms and conditions governing their hotspot’s use, but are not interested in knowing (or do not wish to know/track) any particular user’s identity. This functionality is provided by Hotspot 2.0 Release 2 infrastructure. The Hotspot Operator configures their infrastructure as follows:

  1. The user in a Free Public Hotspot initiates the online sign-up registration process with the Free Public Hotspot’s OSU server.
  2. During the registration exchange, the OSU server presents the terms and conditions to the user.
  3. If the user accepts the terms and conditions, the OSU server issues a credential; if the user refuses, no credential is provisioned. Note that the same credential is issued to all users which have accepted the terms and conditions; therefore, the Hotspot Operator cannot track the identity of an individual user during the Hotspot 2.0 Access state (see section 6).
  4. When the user/mobile device returns to the same Free Public Hotspot, the previously provisioned credentials are used to provide secure, automatic access. The mobile device authenticates using EAP-TTLS, which provides for the generation of unique cryptographic keying material even though users share a common password.

If the terms and conditions change, then the user is taken through a subscription remediation
process during which the new terms and conditions are presented. If the user accepts the
changed terms and conditions, then a new credential is provisioned. 

There you have it. Hotspot 2.0 does provide for anonymous and secure guest networks. In short, 802.1X/EAP authentication is accomplished with EAP-TTLS through a common credential that is issued after the signup process. In fact, this has already been deployed by the cities of San Jose and San Francisco. To get an idea of how it works from a user’s perspective, check out the directions here.

Yes, you can do this all without Hotspot 2.0 in a less elegant way: Add a notice to your guest network captive portal that users can login to the secure network with a specific generic credential, and even a link to download a .mobileconfig profile for iOS and Mac OS users. However, the user experience won’t be standardized like it is with an OSU server, and non-Apple users will have to manually configure a connection to the 802.1X network, including adding a cert to their trusted roots. Not good UX. And definitely not fast, free, and easy.

The bad news: With Hotspot 2.0, the guest network captive portal is here to stay.

The good news: Users only have to wrestle with the captive portal once (unless the client credential is changed). And perhaps the technology behind the portal is more mobile client-friendly than today’s captive portals. Hopefully a HS2 client sees the OSU server being advertised by ANQP and immediately presents a notification to the user. If the user doesn’t play ball, the client should disconnect and the SSID should not be saved as a preferred SSID.

The great news: This is a lower-friction way to get secure Wi-Fi to guests.

Is this the holy grail? That depends on what you think that is. To me, the barrier to entry is low enough that I think this is a win for guest Wi-Fi.

Another wrinkle: The Hotspot 2.0 802.1X network can still be configured to automatically connect guests from known realms. That means that you could add eduroam and the coming anyroam realms to the SSID to onboard users from those participating organizations securely and automatically. And yes, no captive web portal either. So if the opportunities to integrate with AAA clearinghouses grow (exist at all?), the number of users subjected to the captive portal shrinks.

I’m sure there are concerns regarding the possibility of new SSID’s. Luckily, a legacy open guest network can serve Hotspot 2.0 incompatible clients while also delivering the Online Sign Up portal to compatible clients. That means no new SSID’s.

For the visual learners among us, your typical enterprise WLAN might look like this now:

A typical enterprise WLAN
A typical enterprise WLAN

To support secure Hotspot 2.0 guest clients, it might look like this in the future:

A Hotspot 2.0-enabled enterprise WLAN
A Hotspot 2.0-enabled enterprise WLAN

I’m looking forward to seeing gear get updated to support Hotspot 2.0 Rev 2 so we can see this in the wild. Ruckus is doing a great job banging the drum for Hotspot 2.0, but other vendors seem to be further behind. Client support is not great (come on, Android), but Apple has supported it since iOS 7, so here’s hoping that will drive others to follow suit.

K-12 Needs eduroam Too

As eduroam sweeps across higher education in the United States, I think it’s worth considering its place in K-12 as well. After all, every university and college that joins eduroam is within the boundaries of a K-12 school district, and a longstanding relationship is likely to exist between those institutions.

Where I work, we have Miami University within our district boundaries. Miami has a highly regarded education program and dozens of Miami students student teach at our schools everyday. Miami faculty and staff send their kids to our schools, they volunteer here, and they regularly attend school functions. We maintain a formal partnership with the university.

Our teachers and staff take classes at Miami, teach classes, send their kids there, and attend events at Miami. Some of our high school students attend post-secondary classes there as well. In other words, there is a regular flow of visitors back and forth between the institutions.

Because Miami is an eduroam member, it made perfect sense for us to join too. Visitors from our district could gain automatic and secure Wi-Fi access at Miami, and visitors from Miami could have the same at our district. That’s why I pushed to deploy eduroam at my district, making us the first K-12 institution to join eduroam in the United States.

But what about K-12 schools that don’t have higher education institutions nearby? I think there is still a case to be made for eduroam for these districts too.

Quoting the eduroam US FAQ:

Our institution already has great guest Wi-Fi, why do I need eduroam?

eduroam is not a replacement to your guest network, it is a complement to make your guest network and your community compatible with other eduroam participants.

Enabling eduroam on your campus provides four main features:

  1.  it allows your campus to welcome eduroam enabled visitors in a strongly authenticated way (the strong authentication also provides a way to authorize users to different resources)

  2. it allows your own users to travel to eduroam enabled locations around the world (some places only have eduroam as a guest Wi-Fi)

  3. it saves provisioning time to your institution and to the visitors since authentication is automatic and access is immediate

  4. it improves security since your visitors use a standard protocol (WPA2-enterprise, 802.1X) that encrypts traffic between their devices and the Wi-Fi infrastructure

Doesn’t all of this apply to K-12 as well?

I think it does. K-12 school districts aren’t isolated islands. Teachers attend professional development at neighboring districts, students travel for field trips and athletic events, teachers and leadership attend meetings at local education service centers. Some even share employees who split their time between multiple districts. There is a lot of educational roaming occurring within today’s K-12 community.

WLAN Vendors: The NBASE-T Ball is in Your Court

I’m going to echo the position of Marcus BurtonLee Badman, and Andrew von Nagy. Despite the marketing push, there really is no need for > 1 Gbps link to a 802.11ac Wave 2 AP. Not at the access layer at least.

This Wi-Fi gauge goes up to 6.8 Gbps, so that means we need a 10 gig port for every AP, right?

Back to reality: 802.11ac Wave 2 is roughly 7 Gbps max with 8 spatial streams and 160 Mhz channel bandwidth, but most will use 4 spatial streams at max, cutting that in half. If you’re lucky you’ll will use 40 Mhz wide channels, cutting that four-fold. Then take off another 40% for layer 2 overhead and you get roughly 500 Mbps at half duplex on the wire. Maybe a gigabit with 80 Mhz channel width and absolutely ideal conditions that don’t exist outside of the lab.

Even if you have room for 160 Mhz wide channels, which might be possible if the FCC expands 5 Ghz unlicensed spectrum and then client adapters are updated to support those new channels, what is it that clients are using that calls > 1 Gbps of throughput? Remember that Wi-Fi is access layer technology. What applications are you supporting today or anytime in the foreseeable future that call for > 1 Gbps to a small group of clients?

Probably nothing coming over a WAN link. That leaves LAN applications. The list of potential applications provided by the NBASE-T Alliance doesn’t establish the need very well. Security cameras and signage systems?

It’s hard to think of many scenarios where it will be necessary to provide > 1 Gbps of throughput at the access layer. I can’t think of any of those that wouldn’t be better served by a wired solution.

Point to point Wi-Fi links in the distribution layer could benefit from NBASE-T, but how short must those links be to support 256 QAM, and is 8 spatial streams really possible outdoors with limited multipath? I don’t know the answers to those questions, but I don’t imagine that many locations exist that need > 1 Gbps of throughput that haven’t already provided that with fiber. We’re talking serious edge cases here, not typical enterprise Wi-Fi.

In any event, despite the hype, it will be a long time before the need for > 1 Gbps switchports extends outside of the network core and distribution layers.

WLAN vendors have an important decision to make here. Because 802.11ac appears to be the major justification for NBASE-T switches, I imagine they are under a lot of pressure right now. To get real interest in these new NBASE-T switches,  AP’s will have to be built with NBASE-T interfaces that use the faster speeds. I assume that those interfaces will be more expensive than standard gigabit interfaces. Given that NBASE-T supports 10 Gbps I bet they will be a whole lot more expensive than the gigabit interfaces used today. Just a guess though. Time will tell what this stuff really costs.

Will WLAN vendors become accessories to this marketing crime and include these potentially expensive interfaces in their Wave 2 AP’s? Aruba and Ruckus recently joined Cisco in the NBASE-T Alliance. We’ll have to wait and see their plans for the technology.

I hope that 802.11ac Wave 2 enterprise AP’s are still made with standard gigabit interfaces. Speciality AP’s like those used in point to point links could benefit from multigigabit interfaces, but the AP’s that are sold by the dozens for typical enterprise purposes do not. The added cost of an underutilized NBASE-T interface is not justified by real world needs.

Perhaps the usual product cycle will repeat itself. The first Wave 2 AP’s will have the highest end hardware and NBASE-T interfaces. Then the mid- and low-range AP’s that follow and actually get sold will have gigabit interfaces.

Whatever the case, it’s going to be interesting to see how the marketing hype about 802.11ac Wave 2 evolves as more people get clued in to its real world performance.

Channel Planning isn’t Easy for Algorithms

If you’ve ever had to create a manual channel plan where spectrum is scarce, you know how hard it is to get it right. You run out of virgin spectrum, then the difficult choice of channel reuse is encountered. Often, what looks acceptable on an architectural plan, doesn’t hold up to post-deployment validation. Two AP’s 6 classrooms away on the same channel can hear each other at a loud and clear -65 dBm RSSI. Reuse the same channel in the classroom directly above, and the signal disappears below the noise floor. An extra inch of concrete make all the difference. To get it right, you have test, change, check, test, change, check, etc.

24channelplan
A 2.4Ghz channel plan

Given the challenge, it’s no surprise that I’ve never encountered an automatic channel selection algorithm that produced better results. At least, not in high density designs where spectrum is scarce, which is more and more just about everything I design. AP’s directly adjacent to one and other end up on the same channel, blasting away at max transmit power.

Speaking of power, I’ve also never encountered an algorithm that satisfactorily handled transmit power control in a high density network. They always turn things up WAY too high. As in, I’m manually taking AP’s that were auto-set from 12-20 dBm down to 4-6 dBm to shrink their cells away from the AP’s that share their channel. That can mean a 10x reduction in power! And even when power levels are auto-set to an acceptable level, I’ve yet to meet an algorithm that proportionally adjusts an AP’s receive sensitivity to accommodate the smaller cell.

Another thing algorithms don’t do well is handle DFS channels. I use DFS channels in many high density designs, but there are always some clients that don’t support them. The best thing to do in that case is to evenly distribute DFS channels throughout the WLAN, and only use them where AP density would otherwise cause non-DFS channel overlap. In those environments I like to alternate non-DFS channels with DFS channels so that clients without DFS support are still within range of a 5 GHz radio they can use. My experience with channel selection algorithms has been that a group of adjacent of AP’s may all be set to a DFS channel, creating a 5 Ghz dead-zone for clients that don’t support DFS channels.

What gives?

This is all too bad because auto channel/power features would be ideal as it dynamically adjusts to changes in the RF environment. A neighbor puts up a new AP on one of your channels and, without intervention, the algorithm moves your AP to clean spectrum elsewhere. In urban environments, this is a highly desirable feature because there is so much RF in your environment that is out of your control.

Every WLAN vendor offers automatic channel/power selection. They all ticked that box a long time ago. But who’s got an algorithm that actually works?