Clear To Send Podcast Episode 62: K12 Wi-Fi Deployments

podcast_logoI recently had the pleasure of joining Rowell Dionicio on the Clear to Send Podcast to talk about Wi-Fi in K12 schools. Clear To Send is a great podcast about enterprise wireless networking and a great way to stay current with the Wi-Fi community.

We talked about K12 requirements, challenges, funding, my design process, security, and everyone’s favorite K12 subject, 1 AP per classroom!

After listening to the podcast, I thought about some other K12 Wi-Fi considerations that I didn’t bring up on the air.

  • K12 often has requirements for mDNS applications like Apple AirPlay for AppleTV or Google Cast for Chromecast. This is a challenge in an enterprise network because mDNS does not cross layer 2 boundaries. It’s important to consider that when designing a new WLAN and selecting the vendor. Many WLAN vendors do have features that can assist with relaying mDNS traffic between vlans. Be careful to limit this traffic to only the vlans where it is required.
  • Excessive multicast traffic can be a burden on channel utilization when it is not controlled. Many WLAN vendors have features that intelligently filter broadcast/multicast traffic, instead of always forwarding it out the AP radio interfaces at the lowest data rate. If you are dealing with mDNS or large subnets (common in K12) it’s worthwhile to understand how the WLAN can manage broadcast/multicast traffic.
  • MSP’s are a great way to get well-designed enterprise Wi-Fi into small to medium size schools that don’t have the internal resources to handle it themselves. MSP’s can be hired to support and operate the WLAN after installing it, which gives them an incentive that VAR’s who just sell the hardware might not have–to design the WLAN properly. E-Rate funding is now available to reimburse schools for managed services contracts with MSP’s.
  • eduroam is available for K12 schools, not just higher education. Check it out!
  • It’s hard to listen to the sound of your own voice.

I really enjoyed talking Wi-Fi with Rowell and I’d love to return to the podcast in the future. Maybe we can talk about healthcare Wi-Fi next? Thanks Rowell!

Have a listen here: CTS 062: K12 Wi-Fi Deployments – Clear To Send

Making RRM Work

There’s been a lot of good discussion within the Wi-Fi community recently about the viability of radio resource management (RRM), or the automatic selection of channels and Tx power settings by proprietary vendor algorithms. At Mobility Field Day 1 there was this excellent roundtable.

Personally, I usually fall into the static design camp, for many of the same reasons as others. I don’t want RRM to change the carefully tuned design I put in place and create an unpredictable RF environment, I’ve seen RRM do some very peculiar things like put adjacent AP’s on the same channels or crank up the Tx power of 2.4 GHz radios in an HD environment, RRM doesn’t disable 2.4 GHz radios when CCC is present, and it doesn’t plan DFS channels properly. Still, I’ve tried to keep an open mind.

Static designs have their limitations too. Statically designed WLAN’s can’t react to new neighboring networks contending for the same airtime, or new sources of RF interference that weren’t there when the static design was developed. It’s a real benefit of RRM that it does automatically correct for these problems.

Let me propose a hybrid approach that uses static design to handle the things that RRM does poorly, while still allowing RRM to react to the changing RF environment.

Static Design Elements

  • Tx power levels should be statically assigned. Once finely tuned as part of the design process, why would they ever need to change?
  • Excess 2.4 GHz radios in high density environments should be manually disabled because RRM simply won’t do this.
  • DFS channels should be statically planned. RRM can clump DFS channels near one and other, resulting in a 5 GHz dead zone for clients without DFS support. Also, because of these clients, DFS channels should only be used when non-DFS channels are all already deployed. Therefore, statically plan DFS channels when needed in areas where non-DFS channels create secondary coverage, and let RRM dynamically plan the other bands. It’s less likely to have a neighbor or transient hotspot appear in the DFS bands anyway.
  • Set channel channel bandwidth statically. The design process includes considering the capacity requirements of the WLAN to determine the appropropriate 5 GHz channel bandwidth. RRM algorithms don’t know what your capacity requirements are. 2.4 GHz should always be 20 MHz.

Things Left to RRM

  • 2.4 GHz channel planning, once excess radios are disabled. Channels 1, 6, and 11 only, of course.
  • 5 GHz channel planning, once DFS channels are statically assigned.
  • That’s all.

The benefit of this approach is that it addresses many of the shortcomings of RRM while still retaining its main benefit: the WLAN can dynamically react to RF interference and transient neighbors by moving affected AP radios to clear spectrum. The things that RRM can’t do or does poorly are simply removed from its control.

Even within these constraints, there are still some vendor’s RRM algorithms I trust more than others. And even those I trust enough to try this with, I’d still want to monitor regularly to make sure the WLAN hasn’t turned into the RRM trainwreck the I’ve seen all too often when RRM is given free reign.

Why K12 Schools Need Wi-Fi Design

Chalk drawing of WIFI

Enterprise Wi-Fi is expensive, very expensive. For schools with limited budgets and a responsibility to be good stewards of tax dollars, it is important to get it right, without spending more than necessary on the initial deployment, ongoing support, or fixing costly mistakes. Any savings can be used in other ways to improve education, so unnecessary spending on Wi-Fi can have an impact on the quality of education in schools.

That’s why it is critical for schools to work with Wi-Fi professionals to develop a sound design for the network before it is purchased and deployed. Fixing mistakes after the fact costs a lot of money. The usual “fix” of installing extra access points in areas where performance is poor can often make the situation worse, when the real solution might be to remove an AP or correct a bad channel plan.

What often happens is this: A vendor talks the school into purchasing one AP per classroom and then the channel planning is left up to auto-channel algorithms (known as RRM, or radio resource management). This is a very simple and seemingly easy way to get Wi-Fi in schools that doesn’t involve the headaches of procuring CAD drawings, performing multiple site surveys, collecting client device data, and other things that delay the installation of the Wi-Fi network and increase the up-front costs.

Don’t do it!

The big problem here is that this is extremely inefficient. Do schools need one AP per classroom? Some do, some don’t. You’ll only find out by doing a proper network design. Maybe the design process reveals that a school only needs one AP per two classrooms. A school like this that doesn’t bother with a design and just does one AP per classroom has spent 100% more money than it needed to.

Capacity issues aside, what about channel planning and radio transmit power control?Nearby AP’s on the same channel interfere with each other. Vendors love to tout their RRM as effective means to automatically set these controls optimally. Just turn it on and let the magic happen.

The truth is, RRM just can’t be trusted. It may work for a while, and then it changes something and it doesn’t. My experience has shown that RRM is fine for simple networks with few neighbors, but in the high density, busy RF environment of K12 schools it often fails miserably. Neighboring AP’s end up on the same channel resulting in interference with one and other. Transmit power goes up and down unpredictably. Your Wi-Fi network is an unpredictable moving target. What you measured and validated at one location one day is different the next day, and so on. The ongoing cost of supporting a network in this state is much higher than one that began with a proper design.

While some vendors’ RRM is better than others, no vendor is immune to this. A better solution is a proper design where channels and transmit power are determined by a Wi-Fi professional who is informed by years of experience and site survey data that RRM algorithms can’t factor into their decision making.

It is critical that schools include a proper Wi-Fi design in their Wi-Fi deployments to save tax dollars that would better be spent on other educational needs, and prevent many future headaches that result from over/under capacity networks and bumbling RRM algorithms. The Wi-Fi design process avoids these issues, and leaves schools with efficient, stable networks and the confidence in knowing that the network was validated against their needs, with the data to prove it.

Beyond the tax dollars, in a 21st century classroom, what is the true cost of poor Wi-Fi?


Layer 7 Firewalls and QoS on the WLAN

Several WLAN vendors offer layer 7, or application layer, firewalls and quality of service tools. The feature has different names depending on the vendor (Application Visibility and Control, Layer 7 Visibility, AppRF, etc.), but they all try to do the same thing. These tools work at the application layer to identify packets for processing through firewall or QoS rules, which is very useful in today’s world where so many applications are served over the Internet on ports 80 and 443. Traditional stateful firewalls aren’t much use when you want to say, ratelimit Netflix traffic.

At first, you may be tempted to identify all the applications commonly used on your WLAN and assign each of them to a QoS queue. Mission-critical applications get higher priority while social networks and video streaming services are deprioritized. Mark everything!

However, like other features of enterprise gear, while it’s tempting to turn it on and go nuts, you should use restraint, and here’s why: Layer 7 traffic analysis can be very CPU intensive, so the more layer 7 rules in your ACL’s, the more work the AP or controller must do to enforce them. That can result in a performance penalty during high traffic periods. At least one WLAN vendor tacitly acknowledges this by providing an undocumented “Turbo Mode” that will “disable QoS policies and improve Wi-Fi performance.”

Also keep in mind that layer 7 traffic analysis is a bit more of an art than the hard science of stateful packet inspection. Traffic flows are compared to vendor proprietary signatures for proper identification, and that’s not always 100% reliable. An application update or backend infrastructure change may require the development of a new signature for proper identification. WLAN vendors need to provide customers with regular updates to their application signature databases to ensure proper identification is occurring.

With that aside, what are some good uses of layer 7 firewalls and QoS?

Background Data Hogs

RF is a shared medium and as such it is often a bottleneck in busy networks. Software update utilities that run in the background on client machines can be problematic when there are a lot of stations sharing a channel. These applications like to all run at the same time, triggered by events like shifting from a 4G connection to Wi-Fi or right after a machine boots up.

In a school environment, this could happen during first period when everyone pulls out their Chromebook and they all automatically check for updates in the background, while at the same time students’ iPhones notice the Wi-Fi connection and decide now is the time to download that massive iOS update. The WLAN can slow to a crawl without any end-user interaction other than walking in the door.

I think this is where layer 7 QoS shines. By marking Apple Software Update and Chrome OS update packets for the background queue (AC_BK), for example, other applications that users are interacting with in the foreground of their clients take priority on the network. Of course, you will customize these rules to your IT environment. A Microsoft shop will want to do this with traffic to their WSUS server, etc. If you have a lot of iOS clients, iCloud traffic is one to look out for. Dropbox might be a big one too. You may want to consider deprioritizing antivirus updates as well, as these applications sometimes update quite frequently in the background.

Chrome and Chrome OS Updates

Google Chrome LogoIncidentally, despite the overwhelming popularity of Chrome OS in K12, I am unaware of any vendor that provides application signatures for Chrome OS updates. If you can define custom applications within your WLAN (I know that Aerohive and Meraki can do this), use these URL’s to identify Chrome OS updates (these also cover Chrome web browser autoupdates for Windows/Mac/Linux):

Or, if you are really strapped for throughput, use firewall rules to block these applications altogether on the guest network, for example. If WAN throughput is really limited you may need to consider end-to-end QoS all the way to your WAN circuit. Most enterprise WLAN gear can translate WMM QoS markings to 802.1p or DiffServ markings on the ethernet network, but remember to configure QoS on every networking device between the AP and WAN. Do packet captures to confirm your configuration is working.

Recreational Applications

Is it standardized testing season and you are worried that students’ use of Pandora and Netflix is affecting your WLAN performance? No need to go to superiors or committees and ask to have them blocked. That’s a bit draconian anyway. Deprioritize those applications with layer 7 QoS rules.

Malicious and Illegal Applications

Stop bad traffic at the AP or controller before it gets to your content filter. This provides an extra layer of filtering and reduces the traffic the content filter must process. If you don’t enforce station isolation, it can also can block some LAN attacks that would otherwise not reach your content filter. At school, peer-to-peer file sharing applications like Bittorrent, proxy applications, Tor, and shady VPN services are all good candidates layer 7 firewall blocking. Just make sure your firewall rules comply with organizational policy.

Looking Ahead

RF design is the most important factor in meeting the needs of voice-over-Wi-Fi applications, and properly configuring QoS for the enterprise VoIP system has always been important as well. But now we’re seeing users making VoWiFi calls via their cellular carrier. Layer 7 traffic analysis can be used to identify this new traffic and push it to the proper WMM queue (AC_VO).

Going forward these tools might prove less effective as more and more network traffic is encrypted by default. In fact, all HTTP/2 traffic will be encrypted. The companies that develop the application signatures used by WLAN vendors have a challenge to do more with less. Our dependence on these products is increasing while at the same time it will become more difficult to identify application traffic on the network.