Home Networking. Part 1 – The Beginning.

This is a multi-part series on what I’ve learned from my home lab configuration and troubleshooting of Ubiquiti Unifi gear and Velocloud Orchestrator, gateways, and edge devices. Some of this information is already available on the internet, but it took a lot of searching to find it. Some of this information required conversations with the engineers who created the product and is not documented publicly (yet). This blog series is an attempt to consolidate information and links in a single location and simplify some of the mystery of what happens behind the scenes of a Velocloud implementation. I’m happy to answer any questions you might have to the best of my abilities. If it is about Velocloud, I can check with our engineers too.

Phase 1. Requirements.

Cloud and internet. Due to my lack of reliable and highspeed internet connections, security concerns, and my desire to nerd out, everything must be hosted locally and cannot use cloud services.

Wired networks. My home was cabled with cat6, cat6a, and speaker wire to all the locations I might have electronics by professionals. I cannot recommend this enough. Since someone else did it, if a cable doesn’t work, it is on them to fix it. Totally worth it! I also had them install conduits to major locations so it is easy to run new cables in 5-10 years when everything I’ve designed and implemented is obsolete.

I chose to run ethernet everywhere because I find Wifi to be unreliable and generally a pain. With the amount of bandwidth and reliability we need for 4k content and gaming and beyond, ethernet is necessary. Plus, devices like access points and cameras need ethernet for power and security.

Whole home A/V and automation. I didn’t want a closed system (Control 4, Crestron, Savant, etc.) that I was not allowed to configure myself per the manufacturer. And, I didn’t want anything that relied on the cloud to function or stored my data in someone else’s cloud. I choose automation software that has a very large community and runs 100% local. Of course, if I automated something that is controlled in the cloud, that part of the script would be dependent on internet connectivity and the service being online, otherwise, everything is stored and executed locally.

Gear location. I had a closet made specifically for racks of gear with independent cooling. Due to the cost of some cabling types (HDMI over Ethernet), limitations of products on the market, and the laws of physics, some gear couldn’t be stored in the closet which required more cabling to media locations because gear would be mounted there.

Security. I attempted to follow all of the normal best practices: IoT devices can’t talk to the internet or other systems, untrusted guests can’t access internal systems, etc. Any product such as doorbells or cameras that have Bluetooth or Wifi are usually hacked quite quickly and updated for security rarely, so I didn’t choose them. I didn’t buy “smart” appliances or other “smart” home items because security is not those companies’ priority.

Exceptions. Unfortunately, not all of my requirements could be met. And, I still have cell phones, tablets, and a job to do! So, we’re not completely walled off from the world. However, if the internet does become unavailable, we’re able to enjoy media, I can observe and manage my network, and automation still functions.

Phase 2. Setting up the network equipment.

If you follow me on Twitter, then you probably already know that my first issue was the fact that I didn’t have a computer prepared to install Unifi Controller software. Luckily, ESXi runs on pretty much everything these days, like my ancient gaming PC. Within an hour (thank you rural download speeds and my inability to find a large enough USB stick for the iso), I had an ESXi standalone host online. It took a few more hours to have an operating system iso downloaded. I chose to deploy Windows because that’s my most comfortable OS, and I didn’t want to spend time dealing with a less supported version of the controller software. I haven’t had any issues with it yet (after the initial setup as you’ll see below)!

My original choices of gear were the Ubiquiti Unifi Secure Gateway (USG), a Unifi 48 port managed switch, a Unifi 24 port managed POE switch, and multiple Unifi AP Pros. I could have read the product details a closer before placing my order, but a lot was going on in my life, and I made some assumptions I shouldn’t have. Here they are:

·      The USG has deep packet inspection (DPI) capabilities. Yay security and observability! The throughput advertised of the USG is 1 Gbps. Since my rural internet connections come nowhere close to that, I thought I’d be fine. Now, if you turn on DPI, your throughput drops to about 100 Mbps. This is still fine for me, but if you live where you can get fast internet, you will want to get the USG Pro or maybe go a different route altogether (more on that in another post). Also, the USG doesn’t have a rackmount option. The USG Pro does.

·      The Unifi 24 port managed POE switch only has 16 POE ports! The rest are not powered. I did the math on how much power each of my POE devices would probably draw and calculated that the switch could handle that load before ordering. However, I did not check to see how many ports on the switch actually provide power. I ordered a few POE injectors for the remaining devices after realizing my mistake, but I would have purchased different switches if I read the specs closer before placing my order. One cannot return gear to Ubiquiti due to their own mistakes.

·      I did not order a top of rack switch for the second gear rack. When my AV installers showed up and we started discussing the roll out of equipment, I discovered that the cable terminated in different parts of the closet based on a predetermined device layout in the racks. It was expected that the second rack would have devices that needed a switch, so they cabled it that way. I thought there’d be a bit more flexibility. They did provide 4 patch cables between the racks when they ran the rest of the cabling into the closet. If I factored this and my POE switch mistake into the design along with limitations I have since discovered in the USG, my purchase certainly would have looked different.

·      Cat 6a has huge connectors! I already knew Cat 6a didn’t bend well, so we only ran it for certain locations and devices. But, due to the connector size, you cannot have two Cat 6a cables next to each other in the Unifi switch. This is another factor for how many switches to buy and how many ports are needed. Maybe there are switches with ports farther apart than Unifi gear for this reason, I haven’t looked into it.

The Unifi gear was simple to deploy. I was able to just wire up my internet connection to the USG WAN1 port, the USG LAN port to any port on the 48 port switch, and the ESXi host NIC to any port on the 48 port switch and obtain an IP address from the USG DHCP for my host and VMs. The WAN1 port on the USG was configured to use DHCP out of the box as well. After figuring out how to put my satellite modem in bridged mode (Pro tip: disable any VPN software so local addresses can be accessed =), I had internet connectivity. I even placed the Velocloud Edge 510 in line between the USG and modem and had internet connectivity without it being activated and configured. This was nice.

After installing Windows Server, Java, and the Unifi Controller software, it was time to see if setting up the Unifi gear was as easy as everyone said it was. Short answer: Yes! Long answer: not if you think you can change the Unifi devices’ IP addresses!

The Unifi controller software easily discovered the USG and 48 port switch without my having to do anything. There is a simple function called “adopt” so the controller can then manage the Unifi devices it finds. This is all very easy. Except, I didn’t want my hardware to use factory defaults, and I certainly didn’t want it to be using DHCP. The first time, I changed the IP address of the 48 port switch. After the controller software pushed the change, it could no longer find the switch to manage it. I made the change in the controller software so it obviously knows the new IP address. Strange, since the Windows VM was running on a host directly plugged into the switch. I “forgot” the device in the controller, but the software wouldn’t discover it again for me to force an adoption. I spent about an hour looking at community posts on forums about their issues when changing IP addresses (I should have got the hint when every final solution was do a factory reset and adopt the device in the controller software with the defaults) and reading through manuals. I thought I had a plan ready for try number 2 and performed a factory reset on the switch.

This time, I would start with re-IPing the USG. The controller lost its ability to manage USG after I changed its IP address. At least it could talk to it, but it failed when trying to manage it after the change provisioned. My attempts to force the management were thwarted with an authentication issue. Strange again, since the controller software sets the password on adoption. After working through various other documented and community suggested troubleshooting attempts, I gave up. I performed another factory reset and just used the factory defaults. I thought of the other switch and APs that I planned on bringing online and counted the hours of my life I would spend trying to get it to work. I decided it wasn’t worth it. Unless you are extremely lucky, patient, enjoy a cli, and have a lot of extra time on your hands, I don’t suggest wrestling with these devices that definitely do not like their IP addresses changed.

With all these IP changes and abandoned devices, the controller software became very unhappy and couldn’t discover the devices that just went through the second factory reset. I tried a few troubleshooting steps and decided that I may as well just reinstall the software. Minutes later after an uninstall, a reboot, a reinstall, a reboot for good measure, and answering the initial questions, the Unifi controller software was online and able to discover the devices again. I powered up the POE switch, connected it to the 48 port switch, and adopted it in the controller software. I set the DHCP reservations in the USG, performed any remaining updates, and called it a night. If I accepted my factory default fate from the beginning and had faster internet speeds, I probably would have spent no more than two hours on all of this. But hey, I had event tickets to get and dungeons to run in ESO while I waited.

Thanks for reading! Part 2 will be about VLAN creation, firewall rules, and the beginning of the Velocloud edge activation.