I have been a long-time Comcast customer, but when Ziply fiber offered free installation and a decent plan for $20/month, I decided to give them a try. However, I wanted to keep Comcast and simply load-balance my internet traffic across both providers and making it look like a single network to my family. In this article, I’ll explain how I did it, and how I can control the relative traffic to each.
I’m using a Ubiquiti EdgeRouter-X, a $60 box, to do the heavy lifting. Here’s how I have things configured:
There are a few things to note about this setup:
Benefits
- Google Wifi Mesh. I have a 3-puck configuration, with them spread around the house and connected together with CAT5E cable. It works really well. I don’t show it in the diagram, but I also have several devices hardwired and there are a few switches in the topology to allow the devices to share the wires.
- WAN load balancing with weighting. This configuration lets me control the weight of each provider (e.g. Comcast 30%, Ziply 70%). This is useful because Ziply has no monthly caps, whereas Comcast limits me to 1.2 TB/month. I can control the traffic flow and throw more over to Ziply if I get near my Comcast caps.
- WAN failure tolerance. If Comcast goes down, no one in the family will even notice unless the network is pretty busy. (In fact, 4 days after getting Ziply installed, my Comcast line went down overnight and did not come back up on its own; I had to contact support to get it fixed up. Traditionally it has been very reliable. Unfortunately I did not yet have the ER-X so I had to shuffle things around manually to Ziply during the outage.)
Drawbacks
- Triple NAT. There are 3 layers in here, all acting as routers. While this is working for me, it is not ideal. More about this later.
- Additional cost. I can upgrade to Comcast’s unlimited plan for $11/month; with Ziply at $20/month, it’s a little more expensive (with the upside of increased reliability). The ER-X, at $60, is very affordable, but an added cost.
Initial Setup
Update the firmware on the box first thing; mine was way old from the factory.
I used the ER-X “WAN load balancing” wizard to do the initial configuration, using the first two ports (eth0 and eth1) for my ISP connections. The providers are a little bit different; Ziply’s box (the ONT) is not a router, it’s basically a fiber-to-Ethernet converter box, and provides only two public-facing IP addresses. If you’re a Ziply-only customer, you need an additional router, either your own or one you rent from Ziply.
Load Balancing the Traffic
Here’s what my current WAN transmit traffic looks like (red is Comcast, orange is Ziply). I have the traffic weighted 70/30 towards Ziply:
Here’s how to see the current load balance configuration. From the CLI, enter
show load-balance config
In the case where you’re pulling a lot of traffic, both interfaces will be fully saturated at 100%; over time, the box will try to get you back to the weight you set, but this won’t restrict the full bandwidth if you need it.
Telling the Box About Your Connection Speeds
In my case, the connections are very different. Comcast is 175 mpbs down, 5 up. Ziply is 30 down, 30 up. To provide this information to the ER-X, go to the QoS tab at the top of the interface:
Changing the Weighting
Here’s an example of how to set the weighting for each of the WAN interfaces. Let’s say I want to set Comcast to 75% and Ziply to 25%:
configure
set load-balance group G interface eth0 weight 75
set load-balance group G interface eth1 weight 25
commit
save
exit
show load-balance config
Writing Scripts on the ER-X
To make my life easier, I want to store the above commands into a script on the box. The “officially supported” scripting methods only support operational commands (meaning you can’t do configuration within the scripts). Example:
#!/bin/vbash/opt/vyatta/bin/vyatta-op-cmd-wrapper show load-balance config
You can use an editor such as vi from the CLI to create the file above. Save this as /config/scripts/showlb. You can then make it executable by typing
sudo chmod +x /config/scripts/showlb
Now you can execute it by typing:sudo /config/scripts/showlb
And you should get the same output as the image above.
Scripts that Change the Configuration
The script above, when run, does not change anything. For safety reasons, Ubiquiti has made it a little more difficult to find the instructions on how to write scripts to actually change the configuration. Here’s the link to how to write scripts to change configuration; use at your own risk. Here’s an example one I wrote to change the load balancing weights:
#!/bin/vbashif [ $# -ne 1 ]; then
echo "usage: $0 ";
exit 1;
fi
COMCAST=$1
ZIPLY=$((100 - $COMCAST))
source /opt/vyatta/etc/functions/script-template
configure
/opt/vyatta/bin/vyatta-op-cmd-wrapper show load-balance config
set load-balance group G interface eth0 weight $COMCAST
set load-balance group G interface eth1 weight $ZIPLY
commit
save
exit
show load-balance config
I’ve saved this as /config/scripts/weight, did the chmod +x on it to make it executable, then I can change the weights by running:
sudo /config/scripts/weight 70
That’ll make Comcast 70% and Ziply 30%.
Routing, and Possible Improvements to the Setup
Router #1: Comcast Cable Modem
In the Comcast case, the cable modem (either Comcast-rented, or provide your own) acts as a router by default. So, in my case it has an external 70.x address and serves up non-routable NAT addresses in the range 10.0.0.x.
Router #2: EdgeRouter-X
The ER-X also acts as a router, serving up addresses to the LAN port (eth2) in the 192.168.1.x range.
Router #3: Google Wifi
Google Wifi acts as a router, serving up addresses in the 192.168.86.x range. The Google Wifi units must remain in router mode or they won’t mesh. See this article for a more complete explanation: https://support.google.com/wifi/answer/6240987
In short, to get the correct traffic handoff throughout the house, I really want the GWF mesh, so I’m keeping it as the main router, with all of the wired and wireless devices behind it in the 192.168.86.* range.
Improvements
This setup is less than ideal, but it’s working. Here are a couple changes I’m considering that would simplify the picture:
- Put the Comcast cable modem in bridge mode, eliminating one of the NATs. Depending on your cable modem manufacturer, your instructions may vary. Mine’s by Arris; here’s an example Google search to turn up the bridge mode instructions.
- Change the ER-X configuration to bridge mode and create a bridge group for the 3 ports. This would eliminate a second NAT. See the EdgeRouter-X documentation for instructions on creating a bridge group.
- Ideally, use Wifi access points that can be in AP or bridge mode and still mesh. This would allow the ER-X to be the one and only router, which would have the advantage of having all of the clients show up in the ‘Traffic’ page and allow a lot more control over client routing, QoS, etc. (Right now, that traffic page is empty for me, since everything’s hidden behind the Google Wifi NAT and the router can’t see the individual clients.) Google Wifi won’t allow me to do that, and other than that, it works really well, so I’m sticking with it.
Better Shell Access
Using the CLI built into the EdgeMax web-based interface doesn’t support cut/paste with other external apps. There’s a better option, using your own client (such as PuTTY for Windows). Assuming you have SSH Server enabled (on the ‘System’ tab at the bottom of the EdgeMax screen), you can simply use an external terminal app to connect (in my case, it’s 192.168.1.1).
In addition, you can add keys to make this even easier – see this post for instructions. Here’s a quick summary:
(Windows side using Git bash:)cd ~/.ssh
ssh-keygen -t rsa -C "your.email@whatever.com"
scp ~/.ssh/id_rsa.pub ubnt@192.168.1.1:/tmp
(ubnt side:)configure
loadkey /tmp/id_rsa.pub
commit
save
exit
Running Scripts Remotely
With this setup, I can now run scripts from my Windows box directly, e.g.
ssh -l ubnt 192.168.1.1 “sudo /config/scripts/weight 70”
And, in one final act of laziness, I can wrap it with a Windows batch file so all I need to do is type weight 70 from my Windows machine.
weight.bat (Windows):ssh -l ubnt 192.168.1.1 sudo /config/scripts/weight %1
ssh -l ubnt 192.168.1.1 sudo /config/scripts/showlb
Using UNMS for Remote Monitoring
Ubiquiti now offers free access to its cloud-based remote monitoring and management platform (now known as UISP). However, it looks like they’ll bounce you out after 30 days if you don’t have 10+ devices, so I think this one is short-lived for me. Because of this I won’t bother providing instructions, but it’s pretty easy (button in the top-right corner).
The one nice thing they have is a year’s retention on the traffic:
As an alternative, this script looks promising for retrieving traffic stats via the existing CLI. I haven’t tried it yet.
Summary
Although I have way over-engineered my home network, it’s a really nice setup. Because I’m having to work from home because of Covid for the foreseeable future, I can’t afford to be without internet access, so having a cheap backup feed that also offloads my primary line and keeps me from hitting the cap is very nice. For $60, the little ER-X box has been a great addition to make it all seamless for the rest of the family, who also relies heavily on the home network these days.
(Not employed or sponsored by any of the companies mentioned in this post)