Pioneering Edge Computing in the 1970s - "A Little Intel 8008 Computer"

Hello everyone! The post below was not written by me. It was written by my Dad, Charles (Chuck) Blevins. I’m proud to share it because (1) it is super cool tech, especially for the mid 1970s! and (2) after reading it, I realized not only did my Dad teach me so much about computers when I was a kid, but I also have the same curiosity and grit he showed throughout his academic and professional career in my own life and professional career.

This post has something for everyone - solving a new a complex problem, hardware design, chip architecture, custom logic, humor, and inspiration! I hope you enjoy reading it as much as I did.

In the fall of 1974 I began my formal career in electrical engineering. I majored in chemistry as an undergraduate and for a Master’s degree. But after working a few years as a chemist and even inventing an electronic device for measuring the pasting temperature of modified corn starch, I knew that I wanted to switch fields and develop electronic hardware. Hence, I enrolled in the Ph.D. program at the University of Arkansas.

I told them that I wanted to learn about microprocessors. The professors agreed that they too wanted to know more about microprocessors. Intel had made a big splash with the 4004 and 4040 a couple of years earlier. Now the 8008 microprocessor was available and there were rumblings about the venerable 8080 microprocessor. I chose the 8008 microprocessor as it was easy to get. I chose the Intel 1101 SRAM as it was available and the least expensive. However, the total cost of these parts was about $200 back then. Since I didn’t have an application in mind when I started the design and having no computer experience, I didn’t realize how little memory 256 bytes was.

I had to design the circuit and layout and etch the circuit board. If you look at the 8008 datasheet, the most obvious thing is that there are only 14 pins. I had to add latches to capture the upper and lower address bits. This requires that the three state bits must be decoded. I used a SN74138 3to8 decoder to track the state machine. The processor state machine has 5 states to execute an instruction fetch, data read or data write if no wait states are inserted. The upper address bits are latched on the T2 state. The two most significant bits must be decoded to determine if memory or I/O space is being accessed. The bits also decode whether the data is being read or written. The decoder consists of SSI AND, NAND, and inverters like the SN7408, SN7400 and SN7406. (Note that other logic devices have 3 or 4 inputs and decode with fewer devices.) Modern processors have multiple states but these are hidden in a pipeline. They also supply memory read/write and I/O read/write signals. They also have more pins per device so that address and data lines aren’t multiplexed. One very important lesson from examining the 8008 is that it is a state machine and from extension, later processors are just more complex state machines. (As a personal aside, state machines are very important when implementing complex logic with field programmable gate arrays (FPGAs). FPGAs first appeared about 1985 and are my favorite devices when designing custom logic.)

Considering that the clock rate was 500 kHz and two clocks per state meant that the minimum instruction execution time was 20 microseconds. Instructions that needed data required 40 microseconds. Although this is very slow by modern standards, it is fast enough to be useful and that will be apparent later in the story.

My computer did not have any non-volatile memory, any program had to be entered manually before releasing the processor. I had 8 switches to set the SRAM address and 8 more to set the data. A data load or write switch entered the data. Since I did not have a compiler or assembler, I manually converted my assembly level code to machine instructions. By December 1974 I had a computer but no application. Other students had been developing a weather station that would monitor weather and flooding on the White River. I don’t remember which governmental agency wanted the data. The White River was the only waterway in Arkansas that wasn’t dammed and controlled by the Army Corp of Engineers. When everything was ready to deploy, someone realized that the magnetic tape cassette would be filled with data in little over a week. Since it would be a day long trip to drive 150 miles to the selected site on the river, launch a small boat to get to the site, exchange the tape, boat back to the car, and drive back to Little Rock, it was determined that recording data 4 times faster than the current design would make it possible to service the site only once per month. The designer of the recording hardware said that the 4x speed up was not a problem. The designer of the playback unit which also punched paper tape via a teletype machine could not read the data at 4x speed. After hearing about the issue, I decided to see if my computer could read the mag tape and punch the paper tape. Paper tape was the medium that the customer wanted.

Of course, these tasks are I/O. After looking at the digital waveform from some of the tape read hardware, I realized that the computer could count the periods of the high and low signal and decode the data. The computer could also start and stop the tape drive as needed. The computer had enough memory to store a record of one set of readings. Punching the paper tape was a more straightforward task. The binary data needed to be converted to hexadecimal ASCII characters and transmitted serially at 110 baud to the teletype machine. However, there was no UART available to me. The computer program sent the characters by bit banging. A software timing loop was used to set the bit rate. Start and stop bits were prepended and appended to the ASCII characters to complete the formatting. An I/O instruction sent a bit to an RS232 driver. A second printed circuit board held the I/O hardware and was connected to the computer board by many discrete wires. The program was only about 69 instructions long. The program read a record from the mag tape, paused the tape, converted the data to ASCII and sent the data to the teletype, started the tape to read the next record. There was no error handling or end of tape recognition. The setup of the teletype machine was completely manual. Most importantly, it worked and showed everyone the flexibility and utility of small computers. The department was just getting its first minicomputer. This was a very big deal. Computers were just coming online outside of the university data center.

Since my major professor was in charge of the project and we were great friends, we usually serviced the weather station. It was fun boating on the river except for the mosquitoes. When the river was flooding, we could boat all the way to the tree where the hardware was strung up. A long plastic pipe served as a float for measuring the depth of the flood water. Flood data was measured in feet rather than inches. I don’t know details of the measurement electronics because I didn’t do any of the design. One time we brought the mag tape to the lab and found that no data was recorded. We made another day trip to the river and brought the measurement electronics back to the lab. The electronics were encased in a .30 caliber steel ammo box. These boxes are waterproof. The failed component was replaced, and the hardware tested okay. It was returned to the river. We decided to check it in a week rather than a month.

My professor asked if the computer was portable. I decided that a large 12V car battery could replace the power supply. Hence, we could take the computer to the river. The circuit boards with their interconnecting wires and I/O cables were laid in the back of his hatchback along with the battery. We drove to the river and fetched the magnetic tape. As we were pulling the boat up the launch ramp, a game warden showed up. He wanted to see our fishing licenses and our catch. We explained that we weren’t fishing. He was incredulous and looked more than a little spooked when we showed him the computer in the car. After he left, I hooked up the battery to the computer and it lit up okay. Whew!

Who knew how it would survive the ride to the river. I could tell by how the LEDS were sequencing and the tape drive being started and stopped that the computer was reading valid data. We returned to the lab having field tested one of the first portable computers. It wasn’t pretty and it wasn’t rugged but that computer and those experiences remain fond memories.

I have lived my professional life by taking on challenges that I have no experience solving. No one in the Department of Electronics and Instrumentation nor I had ever built a computer. I didn’t worry about failure. I don’t think that I have ever failed to deliver the product, but I have made many mistakes as I was learning something new. I hope whoever reads this anecdote feels the fun and excitement of doing something new. Mistakes are small failures on the path to learning and succeeding. May you enjoy your life and your work product as much as I have.

Amanda's Sessions at VMware Explore 2022

In just a few days thousands of people will be at Moscone in San Francisco and thousands others online for VMware Explore! With 100s of sessions, Hands On Labs (HOLs), and other events, it is hard to build a schedule to maximize your time and learn about all the cool, new stuff VMware is announcing.

I have the privilege of being part of many events and sessions in at Explore! My session involvement ranges from the General Session Keynote to the future of manufacturing. I’m posting my sessions here for you to find and add to your agenda from the content catalog.

General Session
The Multi-Cloud Universe: Bold Innovations and Insights to Accelerate Your Business [GEN2906US]
Tuesday, Aug 30, 9:00 AM - 10:30 AM PDT
Moscone South, Lower Level, Hall F

Solution Keynote
Explore VMware Cross-Cloud Services [MCLK2539US]
Tuesday, Aug 30, 11:00 AM - 12:00 PM PDT
Moscone South, Level 2, Room 206

Breakout Session
From SE to VP: Amanda’s Journey to the Americas CTO at VMware [PCB1232US]
Tuesday, Aug 30, 2:00 PM - 3:00 PM PDT
Moscone West, Level 3, Room 3014

Breakout Session
Open Process Automation for Industrial Controls [VIB1233US]
Wednesday, Aug 31, 10:30 AM - 11:30 AM PDT
Moscone West, Level 3, Room 3018

Each session is interesting and special. The General Session Keynote is where you’ll hear an overview of many new VMware announcements. The Cross-Cloud Solution Keynote will be a level deeper from the general session where we’ll demo some of our new tech. Plus I’ll have a chat with Kelsey Hightower on career progression from vSphere admin to SRE and DevSecOps! Don’t miss that!
My first breakout is a professional development session where I share what I’ve learned over the past 12 years at VMware as I climbed from Senior Systems Engineer to the Americas Chief Technology Officer (CTO) and became VMware’s first woman CTO!
My second breakout represents one of the coolest tech projects I’ve worked on in my career. It is about the shift in industrial controls from proprietary, closed systems to decoupling hardware and software to provide flexibility and new capabilities in manufacturing, oil and gas, energy, and other industries where automating process control lines is key to their business. Open Process Automation is the biggest innovation in industrial controls in 40 years!

I hope to see you all at each session - they’re unique, fun, and showcase innovation across cloud, edge, many industries, and career growth.
Safe travels to San Francisco or enjoy the content online!

Configuring Ubiquiti Unifi Gear for Starlink

This is the final blog post in a three part series describing my experience with SpaceX’s Starlink to date. Here are the links to the first and second blog posts.

My tweet showing 200 Mbps download speed while testing my dmark cable.

My tweet showing 200 Mbps download speed while testing my dmark cable.

After thawing out from the Starlink dish install with my brother-in-law, sending in the support ticket about the mount (that I still have not received a reply to), and investigating dinner options, I sat down on the couch with my laptop to figure out how I was going to leverage the Starlink uplink to meet my user experience requirements. The first step was to make sure that my dmark work was successful. I went downstairs to the gear closet, powered up the Starlink PoE injector and router and ran another speed test on my phone. 200 Mbps down! I’d say my home wiring was working just fine. I unplugged the ethernet from the Starlink router and plugged it into WAN2 on the Ubiquiti Unifi Secure Gateway (USG) PRO 4 and headed back upstairs. An important note: if you are not using the Starlink WIFI router, you can no longer see any information or stats about your Starlink connection. No more debug stats or other useful insight. I don’t know if this affects things like service upgrades or data collection for SpaceX. But, I was not interested in having a double NAT setup that might impact user experience.

Ubiquiti Unifi Controller ISP Load Screenshot. The large throughput usage spike is during an ESO raid (ESO is Elder Scrolls Online. A raid is a run with 12 people to beat a Trial).

Ubiquiti Unifi Controller ISP Load Screenshot. The large throughput usage spike is during an ESO raid (ESO is Elder Scrolls Online. A raid is a run with 12 people to beat a Trial).

If you are responsible for your household’s connectivity to the outside world like me, you know how any blip in service can result in a crisis. We’ve been getting by with 26 Mbps down and 6 Mbps up on point-to-point WIFI for most of the pandemic. However, when my home users are pushing the limits of my service and my Internet Service Provider (ISP) is loaded with users at the end of the day, gaming is almost impossible. I’ve pulled out my hotspot so I don’t lag out during a raid and cause a group wipe. I’ve been extremely impressed with Zoom working well in the worst of internet connectivity. I can’t say the same for Teams or Google Meet. There’s plenty of calls I’m on where I don’t see anyone’s video on those platforms. I assume this is how they deal with my poor connectivity.

This blog outlines my design that I’ve implemented to maximize my home’s user experience. As with any design, yours should align to the requirements you have. The other people in my household use the internet mainly for streaming Netflix, YouTube TV, Hulu, and Amazon Prime. They also have the occasional telehealth or fun virtual gathering. There’s definitely a lot of web browsing, social media, and messaging platforms use. My user profile is that of any home worker plus online gaming requiring real time inputs. I don’t stream due to many factors, but mostly because of my internet speeds. I do record my screen and audio locally to review the runs, and I log the raids which increases the work my client does, but it doesn’t seem to be a major factor when considering the connectivity requirements. Maybe one day with Starlink I will be able to stream!

My rural ISP and Starlink have the same latency, 40-60 ms. This is completely satisfactory for online gaming unless you’re a professional esports athlete, then I wouldn’t recommend it. I’d probably sit in the data center with a cross-connect to the cage where the game is hosted if that was my job! However, the biggest issue with Starlink right now is that I only have about 73% coverage. And I experience a 30 second disconnect every 15-20 minutes during the day. How do I know I have 73% coverage? There are a number of sites with Starlink maps and other Open Source projects. Satellite Map Space and Starlink Coverage in GitHub are the two I have seen.

Clearly, 30 second disconnects multiple times an hour does not work for online meetings and gaming. Therefore, Starlink cannot be my internet connection for those activities. But I really want to have the much faster upload and download speeds for everything else! Based on the rest of the household’s user profile, the majority of their activities can handle those disconnects. Plus, if all of that traffic is removed from the rural ISP connection, it frees up bandwidth and lowers latency. My plan was to route all traffic except for meetings, virtual gatherings, telehealth, and online gaming over Starlink.

Screenshot from the Unifi Secure Gateway Details page. The IP Address has been removed from the image.

Screenshot from the Unifi Secure Gateway Details page. The IP Address has been removed from the image.

Enabling Starlink on WAN2 on the Unifi USG PRO 4 was easy. Navigate in the UniFi Controller to Settings -> Create New Network. Label the link for WAN2, enable Failover only, accept the rest of the defaults, and Save. After that, I was able to see that WAN2 received a Starlink IP address and traffic was moving over the link. Things were moving quickly and easily. Red flag! The pain started as I was clicking around the Unifi Controller GUI to figure out how to route my different subnets over WAN1 or WAN2 based on my preference aligned to the user experience requirements. This is called Policy Based Routing (PBR). After Googling I found that configuring PBR is not an option in the GUI and it must be configured via a JSON file stored on the controller. I also noted the irony that I only learn about and configure Policy Based Routing on Sundays from the couch. The last time was for a VMworld demo!

Most of my Googling resulted in reading about people’s disappointment that PBR was not in the GUI. The complaints on the Ubiquiti forums began years ago. Rather than including the capability in the GUI, Ubiquiti wrote a Support and Help article on how to configure PRB through the Unifi Secure Gateway interface and JSON.

After reading the above article and the related article about USG Advanced Configuration with JSON multiple times, Googling more to see if people posted their JSONs for PBR, and sighing loudly, I went to work. I have multiple subnets for various purposes in my house. But none of VLANs were for users I wanted to give uninterrupted internet access to (since this is usually a given). The first step was to create a new network in the Unifi Controller GUI. I also created a WIFI SSID for the new VLAN to enable easier testing since I wasn’t near an ethernet port. The first test of my capabilities was to route the new VLAN over WAN2.

I am not a fan of static routes. They feel lazy (Like GOTO statements in BASIC. Clearly my elementary school coding experience was traumatic.) and should only be used as a last resort. Instead, I wanted to properly configure failover groups between my two uplinks and designate a primary route for my VLANs based on user requirements. I also wanted to use pointers to groups of VLANs rather than having to explicitly define subnets, but from what I could find, subnets must be explicitly defined in JSON Policy Based Routing. Working with what I was given, I opened an SSH session to my USG and started typing. This part was straight-forward. I scrolled to the “Routing Traffic to Different Load Balancing Groups Based on the Source Network” part of the Policy Based Routing help article, substituted in my subnet for the session state VLAN, and typed the commands into my USG. I connected my phone to the newly created WIFI SSID and ran a speed test. 110 Mbps down. Success!

If you don’t want to navigate to the Unifi help article at this time, here is a copy paste of the example code to run directly on your USG:

configure
set load-balance group wan2_failover interface eth2 failover-only
set load-balance group wan2_failover interface eth3
set firewall modify LOAD_BALANCE rule 2503 action modify
set firewall modify LOAD_BALANCE rule 2503 modify lb-group wan2_failover
set firewall modify LOAD_BALANCE rule 2503 source address 192.168.1.0/24
commit ; exit

Note: eth2 is WAN1 and eth3 is WAN2 on the USG PRO 4. Use the show interface command when connected via SSH in your USG to verify which ethernet interface are your WAN ports.

Second note: if you are curious why we couldn’t stop here, whenever the USG reboots, any configuration applied in the command line interface is erased. This is why we need the custom JSON file stored on the controller. The controller applies the configuration from the JSON upon a provisioning event.

According to the help article, the next step is dumping the USG config to a text file and using this text file to populate the JSON on the controller. This is where the fun really began. Here’s why:

1.     I force myself to use Linux without a GUI at home to improve my skills. Generally, doing anything requires learning something new on Linux first.
2.     JSON is not fun.
3.     Editing JSON in Nano is really not fun.

The Ubiquiti article mentioned above goes into detail on creating and using the config.gateway.json file. If you copy and paste the entire config dump on the USG into your config.gateway.json file, you can no longer control your Unifi setup through the controller. With my obvious Windows background and preference for a GUI, this was a no go for me. Therefore, I had to delete everything from the JSON file that the USG created except for the areas that the above commands generated for my Policy Based Routing. Also, I had to keep track of all the {s and }s and still retain the section headers for the JSON to work.

In an effort to avoid all of this editing, I Googled some more to find any JSON file posted that already accomplished what I wanted to do. Unfortunately, those examples used static routes rather than load balancing. The one example I found that did use load balancing wasn’t valid even though it was a valid JSON file. More on this later.

So, back to Nano over SSH. You can see my true procrastination personality here! After avoiding editing by Googling, then editing, and counting brackets, I was ready to apply my new JSON configuration that would accomplish (hopefully) what the six simple commands above did. Spoiler alert: It didn’t.

To apply the new JSON file, there are two steps.
1. Restart the unifi service on the controller machine. In Ubuntu the command is

sudo service unifi restart

 

Note: if this takes you as many tries as it does me, press the up arrow a few times and enter to restart your service.
2. Navigate into the Unifi Controller and force provision to the USG under Devices -> USG -> Config -> Manage Device -> Provision (or trigger this from the USG CLI).

If your Unifi Controller is pushing an invalid JSON file to your USG, you’ll notice your USG is stuck in the provisioning phase. The only way to stop this is to delete or rename your json file and the provisioning phase will complete successfully. Most times, the USG will reboot after this. The result of the USG reboot is the internet is unavailable, and it will elicit a loud huff from anyone trying to watch YouTube TV while you curse at your screen. There are many ways to ensure your JSON file is valid. Since I was using my high-tech Nano editor, I preferred a SaaS solution rather than wrestle with Linux to install a development tool. This site is recommended by Ubiquiti: https://jsonlint.com/

After the dreaded USG reboot and subsequent cursing, I pasted my json file into the SaaS verification tool. I was missing curly brackets. I added those in, hoped all my commas were right, restarted the service, and forced a provision. Still stuck in provisioning.

I reviewed the one JSON example for a similar load balancer setup that I found online. I edited my file to match that format (which the author said was verified by a JSON validator) and pasted my file into the JSON verification tool on the site above. It did return that the JSON was valid. When I force provisioned this version of the config file, the USG was still stuck in provisioning in the Unifi Controller. Then I started reading about USG error logs.

One person online was of the opinion that verbose / debug logging had to be enabled to troubleshoot JSON config issues. That sounded like a lot of work, so I kept Googling. Another person suggested connecting via SSH to the USG and run

tail -f /var/log/messages


That I was down for! Even though the USG is in a provisioning state, you are able to SSH to it and run commands if you weren’t already connected. My tail -f command flooded my terminal window with error messages that hurt my brain to decrypt. Googling these messages provided no insight. So, I stared at them longer. Finally, I figured out what was happening. Even though the JSON file was valid according to the JSON verification tool and copied from a forum post of someone who was quite confident in their JSON abilities, it was wrong. It needed another set of curly brackets because the USG was interpreting my load balancer rules as an entry under the firewall section! @#*($%&*(^%*(@#!!!

Back in Nano I counted curly brackets again, added in another set, moved around a comma or two, crossed my fingers, and copied and pasted it to the JSON verification tool. It told me it was valid. I restarted the service for the umpteenth time, force provisioned the USG, and threw up my arms with a victory cry after it provisioned successfully. I tested my download speeds on my phone that was connected to my special VLAN, sure enough, they were FAST. I tested the download speeds of my computer that was supposed to route through the rural ISP, sure enough, they were SLOW!

Feeling smug, I tweeted my success.

Then I thought, wait, I actually want all of my VLANs to route through WAN2 and fail to WAN1, and I want my special VLAN to route through WAN1 and fail to WAN2. You’re probably thinking, why not swap ports for the uplinks and call it a night? Well, I would, except my rural ISP authenticates by MAC address. If I swapped ports, I’d have no rural ISP connectivity because it sees a different MAC address. And it was 9 pm on Sunday - there isn’t anyone to answer the phone and make this change.

Therefore, I had to add more firewall rules to my oh so fragile JSON file. I didn’t want to create yet another network to test with, so I figured the guest network subnet could be the second one to move. Remember how I tested if I was my traffic was going out the correct uplink? Yes, a speed test. The difference between 8 Mbps and 150 Mbps is a pretty good test… or so I thought. Well, I forgot that I had setup a bandwidth restriction on the guest network subnet a month ago to prevent video uploads to YouTube from taking down my internet. I spent another 15 minutes of complete frustration and Googling to determine if Unifi Guest networks are compatible with Policy Based Routing. I couldn’t find information stating one way or another. I decided to go all in and added in a firewall rule for the subnet that all of my NVIDIA Shields are on. I braced myself for the fifth loud huff of the night as YouTube TV was interrupted. But, it never happened and that picture sure looked crisp running over Starlink!

It was only then that I remembered my bandwidth limit rule on the Guest network and went through the exercise of removing all references to it in the Unifi Controller GUI so I could delete it FOREVER! Once deleted, my speed test for the Guest network showed 80 Mbps. I guess that was fine with two NVIDIA Shields streaming and the Ubuntu updates downloading. Way better than the 6 Mbps download speed I enforced before. I made a few more changes to switch ports connected to RJ45 jacks in my office to use the special VLAN in preparation for work and leading a raid the next day and called it a night.

I know you’ve been waiting with bated breath to see my masterpiece of a JSON file so you can avoid everything you just read here. Ta da!

{
    "firewall": {
        "modify": {
            "LOAD_BALANCE": {
                "rule": {
                    "2606": {
                        "action": "modify",
                        "modify": {
                            "lb-group": "wan2_failover"
                        },
                        "source": {
                            "address": "10.10.0.0/16"
                        }
                    },
                    "2607": {
                        "action": "modify",
                        "modify": {
                            "lb-group": "wan2_failover"
                        },
                        "source": {
                            "address": "10.50.1.0/24"
                        }
                    }
                }
            }
        }
    },
    "load-balance": {
        "group": {
            "wan2_failover": {
                "flush-on-active": "disable",
                "interface": {
                    "eth2": {
                        "failover-only": "''"
                    },
                    "eth3": "''"
                },
                "lb-local": "enable",
                "lb-local-metric-change": "enable"
            }
        }
    }
}

Seems rather insignificant for all that work, huh? It could certainly be fancier to include rules around testing the connection and how many ping failures before initiating the failover. But, that requires working out some math considering how frequently the Starlink connection goes offline to prevent unnecessary failovers. With today’s and future SpaceX Starlink launches, my experience might change, so I just left it as it is.

Someone with far more patience than me might figure out IP address ranges and ports associated with Zoom, Team, Google Meet, Telehealth sites, Elder Scrolls Online, and everything else that needs a stable connection and have a much longer JSON than you see here which would forgo the need to manually change networks / SSIDs when I want to do something other than download and upload large files or stream content. But this works for me. And, I’m optimistic about Starlink improving their service quickly!

SpaceX Starlink Roof Installation on the Colorado Plains

This is part two of the SpaceX Starlink Blogs Series. Read part one here!

My brother-in-law texted me early afternoon to say he was coming over to do the install in an hour. I couldn’t hear the wind howling, the snow was melted from the roof, and the grass was swaying in a normal breeze. Everything was a go!

Most of what is necessary for the install is included in the Volcano Roof Mount kit – including a carrying bag to safely bring the dish onto the roof! The state of drills and drill bits in my house is questionable. Where do those batteries keep wandering off to?! I suggested my brother-in-law bring his own drill, drill bits, and stud finder. There was no need to purchase silicon for the roof as the kit comes with sticky rubber to place in the drilled-out holes on the roof and set beneath the mount before the lag bolts are fastened to the roof. The only other items I needed were a socket for the lag bolts, socket wrench, and 6-inch extension. There’s no shortage of these in the shop!

I collected the Starlink dish in its carrying case, Volcano Roof Mount install kit, and the POE injector and WIFI router, and carried it outside for the install. As I was setting up and reviewing the cabling the dmark box on the side of the house, I felt the wind pick up. Uh oh. My brother-in-law hadn’t arrived yet and mid-afternoon on the plains can turn into torture with cold, biting wind. I hoped for the best and hunted for the tools I needed to clean up the cabling at the dmark box and make room for the large Starlink ethernet cable.

Upon his arrival, I showed him the layout for the cabling and how we will run it down the roof and the side of the house. We discussed the dish positioning ad nauseum while he was standing on the ground and after he climbed onto the roof. I was probably being extra paranoid about obstructions based on my first impressions of the Starlink service. The dish needed clear visibility to the northwest, north, and northeast. However, we can see the Starlink satellites flying past the house to the west at night, so I didn’t want to block that view either. Once we selected the location for the dish, then it was time to find a stud to attach the Volcano Roof Mount. I have no idea what kind of stud finder works through roofing material, but it wasn’t the one he brought. In fact, I can’t think of a time a stud finder has confidently located a stud in a wall. I’ve always just knocked. And that’s what he did on the roof. He couldn’t hear the different between hollow versus solid because of the wind howling in his ears. Acoustics are interesting. I could hear it perfectly standing on the ground. After locating the studs on the roof with our high-tech methods, it was time to mark and drill.

My brother in law torquing down the lag bolts for the Volcano Roof Mount on the very windy Colorado plains.

My brother in law torquing down the lag bolts for the Volcano Roof Mount on the very windy Colorado plains.

At this point I had been standing in the shade, enduring the increasing wind for an hour. My role was mainly moral support and safety supervisor after the drilling began. Even wearing Carhartt work gloves and shoving my hands into the pockets of my Carhartt heavy duty jacket, my fingers were feeling numb and stinging from the cold at the same time. I regretted not putting on proper winter gear. But, it is impossible to do cabling with winter gloves on. Plus, I wasn’t the one on the roof trying to use power tools with uncooperative Starlink “sealing tape” without protection from the elements. I had to tough it out!

The first test through the Starlink WIFI Router after the dish was mounted on the roof.

The first test through the Starlink WIFI Router after the dish was mounted on the roof.

I was feeling concerned about his safety with the strong gusts while he was attaching the dish to the roof mount. He dropped the ethernet cable down to me and I quickly plugged it into the POE injector that was ready to go in the garage. A few minutes later I was connected to the internet and ran a speed test from my phone through the Starlink WIFI router. 130 Mbps down! Success!

Now we were working as fast as we could because the cold wind was miserable. He was attaching the cabling to the roof and down the side of the house with the Volcano Roof Mount kit cable ties. The screws for the cable ties were entirely too short, so he switched to zip ties to run it down the side of the house. In theory, zip ties were easy because he ran the Starlink ethernet cable along the existing ethernet cable from my other internet service provider. However, the zip ties in the package were brittle from their multiple outdoor projects and kept breaking when tightened. &%#(*&@#$^! I was cutting the opening of the dmark box a bit more to shove the Starlink RJ45 end into it without damaging the other cables inside. My fingers weren’t functioning, and the pain from the biting cold was almost unbearable. I had to stop a couple of times to run into the shop to run my hands under hot water to be able to use them again. Finally, the Starlink ethernet cable fit into the dmark box. I connected it to an existing Cat6 cable with an RJ45 coupler, verified none of the cables were pinched, screwed the cover back on, and sealed it with silicone. We used a few more zip ties to make sure the excess cable wasn’t flopping around, and he quickly got into his car, cranked the heat, and sped away. My sister had checked in twice, so it was clear he was expected home. I will clean up the cabling when it isn’t winter!

I put all the tools away, cleaned the silicone from my fingers as much as I could, and headed into the house feeling extremely proud of what we accomplished and very excited about heat and no wind. The first thing my partner says to me is, “What took you so long?”. I tried to explain, but it is clear from her reaction and my sister’s text and call, that this is not a partner approved Sunday afternoon project when it extends into dinner time.

Before I end this blog, I want to note that I submitted a support ticket to Starlink as soon as I thawed out. We were very deliberate every step of the way to secure the install because it has to withstand sustained 40+ mph wind weekly. The two buttons that “lock” the Starlink dish to the Volcano Roof Mount base is a weak point in the design. The other dish on my roof uses bolts and nuts to attach to the base and it has stayed in place for a year and a half. I don’t want my next blog to be about the $500 Starlink dish tumbleweed that has detached from its mount, damaged my house, and is flying across the property. Hopefully SpaceX Starlink will respond positively and can send me an updated design as soon as possible!

Read part three of this blog series for details on how I configured my Ubiquiti Unifi Secure Gateway Pro 4 to work around the connectivity issues of Starlink while maximizing user experience.

SpaceX Starlink First Impressions

Many people have asked about my experience with Starlink, so I am writing a blog! As someone who lives in a rural area and began the pandemic with 10 Mbps down and 3 Mbps up internet speeds, I’ve been counting the days until Starlink was available to me. Even though I submitted my information at each stage of the Starlink announcements and beta, I did not receive a notification that Starlink was available for me to order. I found out from a coworker instead. After reading his Slack message, my order was confirmed in 10 minutes. I also placed an order for the Volcano Roof Mount at the same time.

Three days after my order was placed, the Starlink dish kit shipped. It was supposed to arrive 2 days after that. FedEx has been delivering packages to my house for about 1.5 years now. Every once in a while, they cannot find my house, so I have to call and give them directions after I see my package was on the truck for delivery and returned to the warehouse that evening. Of course, this was one of those packages. After the additional 2 days for delivery due to a snowstorm and the new driver, I finally had the Starlink box in my possession.

Starlink POE Injector and WIFI Router in a plastic bin on my deck.

Starlink POE Injector and WIFI Router in a plastic bin on my deck.

I immediately shoveled the 4 inches of snow off my deck and setup the dish with the router outside. It was entirely too cold to crack a window or door to run the ethernet cable inside unless I wanted to pay $2000 for my propane bill that month. So, I used a large plastic storage bin to house the PoE injector and router and plugged the power into an outdoor outlet on the deck. They kept each other warm in their cozy plastic bin with the lid snapped shut!

 

First speed test of Starlink.

First speed test of Starlink.

The instructions said the dish needs to face north. That isn’t exactly true. The dish needs a clear line of site from slightly NW to NE. My deck is on the west side of my house, so the house was blocking line of site for the dish most of the time. But I was able to connect and do a speed test. Considering the conditions of the “install”, I was impressed to see 79 Mbps down, 18 Mbps up, and 42 ms latency. I left it online in this state after weighing it down with a duffel bag and backpack full of cans from the pantry. My partner and I noted we need to purchase sandbags.

 

I left it online overnight so we could stream Netflix since my current internet service was down. In the morning I peeked at the app and saw the stats for how much time the dish was connected with internet access and how long it was down “due to beta”, upgrades, and obstruction in the app. It was obstructed almost 50% of the time from the moment I brought it online the afternoon before. I was impressed we could watch Netflix almost uninterrupted from a WIFI router in a plastic bin outside 20 feet away with 50% downtime! Unfortunately, the wind was picking up, and I didn’t want a $500 tumbleweed. I packed it up and brought it inside that morning.

 Two days after the first test on the deck, my internet from service provider for the last 1.5 years was still down. The radio burned out on their dish. They don’t send out technicians on the weekend, so I was without internet for 4 days as, of course, this failure occurred late Thursday night. The technician was supposed to arrive Monday morning to repair the radio, but my roof was covered in snow and ice. We rescheduled to Tuesday afternoon. Tuesday was Global Leadership Forum at work, and I had at least 6 other meetings on my calendar. Spotty 4G hotspot on my phone wasn’t going to cut it for a full day of meetings. I hauled my plastic bin to the east side of the house and ran the cable for the dish as far north and slightly east from my front door as possible.

Starlink dish without obstructions, or so I thought!

Starlink dish without obstructions, or so I thought!

At this point, it was physically impossible for there to be an obstruction. But the app still reported an obstruction about 10% of the time. There must be a lot of space garbage blocking the signal. My theory is they use “obstruction” as a catch all for no connectivity. Interesting tidbit: in the stats in debug mode in the app during the “obstruction” the dish was still connected and showing stats for upload speeds and ping, but it showed no data for download speeds. This meant I couldn’t get through a Zoom meeting without a significant audio and video freeze or full disconnect. Starlink was fine for watching others present on Zoom, but it wasn’t ready for interactive Zoom meetings. It also wasn’t ready for online gaming with these short, frequent outages.

Luckily my internet technician showed up around the time I found out that Starlink was not ready for Zoom meetings, and he replaced the radio on my other dish. I was back in business with 26 Mbps down and 6 Mbps up on my point-to-point WIFI internet service. The wind was picking up again, and I didn’t want any animals chewing on the ethernet cable overnight, so I packed up the dish and bin until my roof mount arrived.

I checked my account on Starlink to see if the roof mount was scheduled to ship. There was no update in its status, so I emailed support through the app. I heard back from support about 24 hours later telling me my roof mount will ship in the next 7-14 business days. I only had to wait 5 days to receive the shipment notice and delivery was only delayed by 1 day due to snowstorm. This gave me 3 days to get my brother-in-law to agree to do all the hard work of the installation on the roof. I’m grateful he is the type of person who always shows up when you need him. He agreed to install the dish on the roof, and we waited until the weather was nice enough to work on the roof safely.

Check out Part 2 of this series to see how the roof installation went!

VMworld Call for Proposals: Professional Development Track. Everyone belongs!

VMware has announced the Call for Proposals for all VMworld US submissions. This is a very exciting time as folks organize their thoughts to create proposals about technology they’ve been working with over the past few months as well as what they’ll be working on this year. As technologists, we all want to share our knowledge with others and do our best to enable others. And this is what VMworld is about – a week of sharing and learning!

It was only 2018 when I submitted my talk, Achieving Happiness: Building Your Brand and Your Career, in the VMworld Call for Proposals process. I didn’t know where this type of talk would land, if at all, since VMworld never had sessions (to my knowledge) on personal and professional growth. I selected the Leadership track for my submission. This track was created for technology leaders and executives. Most of the content was for CIOs, CTOs, and VPs around business processes and organizational change. However, my talk was accepted and well-received at both VMworlds in 2018. In 2019 we had a Professional Development session type in the CFP portal due to the very positive response we saw in our VMworld surveys in 2018. In VMworld US 2019, we had 9 Professional Development sessions and in VMworld Europe 2019, we had 6 Professional Development sessions that are available in the on-demand catalog today on vmworld.com. 

I am honored to announce that I am the track lead for Professional Development at VMworld 2020 US and Europe. We are creating more session slots than ever before specific to professional development. The talks might be technology related, and they might not be! That is the beauty of professional development. It isn’t just about professional life. It is about being a good human being. Human development is professional development.

This is my first time being a track lead at VMworld. I don’t know what to expect, but I do know the following:

·      This appointment is an honor, and I take it very seriously.

·      I will give no less than 110% to ensure the Professional Development track is the best it can be.

·      I want this track to be the shining star for what diversity looks like at a conference.

After accepting the track lead role a few weeks ago, I immediately began thinking about how to get the word out for this new track and inspiring the best possible speakers on the subject of being a good human being and professional development to submit a proposal for VMworld. I don’t have all the answers, but I know as a community, we can figure it out together. I need your help in encouraging and supporting yourself, your friends, your colleagues, and people you don’t know to submit their ideas during the CFP process. I’d like to see people of every under-represented group that exists and more as speakers in this track. We all know how vital it is to have diversity of thought and opinion for any subject, and I can’t think of a subject that needs full representation of all people more than human and professional development.

I was thrilled when I saw an internal email from our VP of D&I, Shanis Windland, this week to everyone involved in VMworld planning and execution. She gave me permission to include this quote on my blog:

 “All of you have the opportunity to showcase VMware’s commitment to Diversity and Inclusion. VMware is a diverse, multinational, multigenerational and multicultural company. As you prepare potential submissions, we encourage you to consider speakers that truly highlight the diversity of our employee population and the customers, partners and communities they represent.”  

At VMware we have employee led Power of Difference Communities, or PODs, to represent everyone. We have demographic PODs such as Black@VMware, PRIDE@VMware, Asian@VMware, Disability@VMware, Latinos@VMware, Veterans@VMware, and Women@VMware. We also have PODs based on Geographic Regions. You can learn more about these pods on our VMinclusion page here.

I will be enlisting the PODs help in encouraging people to submit proposals to the Professional Development track. And I’m asking all of you in our community to also go above and beyond to inspire the best speakers you know to submit. If you have an idea, ask someone that doesn’t look like you to co-submit and co-present so we can hear their thoughts and experience on the subject as well. Absolutely everyone is different, and my approach when learning is to listen to all perspectives and employ the saying “Take what you like and leave the rest.”. This doesn’t mean I only retain what is easy or already fits my mindset. It means I integrate what information and actions work for me as an individual with a unique personality to my being.

I hope to meet many people and learn new things throughout this process. Please submit your VMworld US proposals here! VMworld Europe CFP opens in a couple of months. And when that happens, I will re-surface this blog, possibly with edits based on what I’ve learned since it was originally posted, and have the same call-to-action for all of our European-based friends, colleagues, and community members.

Home Networking. Part 3 - VeloCloud Architecture

Before I blog about my experience in configuring VeloCloud from Orchestrator to Edge, it is important to understand the architecture and how the VeloCloud SD-WAN platform functions. With this knowledge one can make the best decisions about how to configure their SD-WAN. SD-WAN solutions provide the software abstraction to create a network overlay and decouple network software services from the underlying hardware.

There are three major components to the VeloCloud Platform: Orchestrator, Gateways, and Edges. I will describe and summarize their functions and relationships to each other in this blog.

VeloCloud Orchestrator Operator Menu

VeloCloud Orchestrator Operator Menu

Orchestrator (VCO).

The VCO is the portal that is used to create, configure, and monitor VeloCloud SD-WANs. VeloCloud Orchestrator is multi-tenant and very powerful. Through a single Orchestator and its associated Gateways, one can create SD-WANs, or Software-defined Wide Area Networks, for Customers or Partners. A customer is able to manage and monitor their own VeloCloud Edges, Network Profiles, Business Policies, Firewall Rules, and more through the VCO. Partners are able to create their own customers within the VCO and manage their customer environments directly. The VCO is also used to activate and configure Edges. The VCO is a virtual machine that can run on vSphere, KVM, or AWS.

Gateway.

A VeloCloud Gateway, or VCG, is the device that an Edge routes traffic through when the traffic is defined to take a “multi-path” route (there will be more on route types in a future blog) or for non-VeloCloud VPNs. There are two main types of configurations for a Gateway, default and Partner. In the VCO, the VeloCloud Operator creates one or more Gateway pools and then places Gateways into that pool. Gateways are virtual machines that can run on vSphere, KVM, or AWS.

Gateway Pools are then assigned to Partners and/or Customers.

VCO Gateway Pools

VCO Gateway Pools

VCO Customers

VCO Customers

In a Cloud Hosted Model where Gateways are in default mode, an Edge is assigned a primary and secondary Gateway based on Geo location through the Maxmind database. The Edge’s peered Gateways are geographically closest to that Edge. The Edge device sends all “multi-path” traffic to its primary VCG and the Gateway then sends the traffic on to the intended destination. Return traffic is sent to the Gateway and then back to the Edge device. If the Edge identifies one of its Gateways as unreachable after 60 seconds, it marks the routes as stale. If the VCG is still unavailable after another 60 seconds, the Edge removes the routes for this Gateway. If all gateways are down, the routes are retained, and the timer is restarted. If the Gateways reconnect, the routes are refreshed on the Edge.

An SD-WAN with Partner Gateways gives the Operator the ability to route traffic to specific VCGs from Edges based on subnet. This is a value-add beyond the Cloud Hosted Model. A Partner will place Gateways geographically close to the services that they offer. When an Edge peered with a Partner VCG wants to access that service, the Edge leverages the tunnel to the Partner Gateway assigned for that service by subnet. Often Edges that are peered with partner Gateways have an average of 4 Gateways manually assigned. This number generally equals the number and locations of the services that the Partner is providing the customer such as SaaS offerings, Cloud services, etc.

You can see in the screenshot below that I checked the box for Partner Gateway during the Gateway creation and was given an option to define which subnets should be routed by that Gateway.

It is important to note that VCGs do not talk to each other and are not aware of each other’s state. Traffic is not routed between Gateways. The Edge sends “multi-path” traffic to its Gateway, that traffic is sent to its destination. When the destination responds, it is routed back through the Gateway to the Edge.

Gateways can be assigned to multiple Gateway Pools. Gateway Pools can be assigned to multiple Customers and Partners within the VCO. Partner Gateways should be placed closest (within 5-10 ms latency) to services that the Edges will access. Default Gateways should be geographically close to the Edges deployed in the Customer SD-WAN. It is not ideal for an Edge on the west coast of the US to send traffic to a Gateway on the east coast of the US before it is routed to its destination, for example.

Edge.

VeloCloud Edge, or VCE, devices are where the magic happens! Edge devices can be physical or virtual. They are implemented in enterprise datacenters, remote locations, and hyperscalers. Edge devices are able to aggregate multiple WAN links from different providers and send traffic on a per packet basis through the best WAN link to its peered Gateway. An Edge can aggregate the multiple WAN links and remediate issues found on public Internet providers such as loss, jitter, and latency. Even if just one WAN Link is connected to a VCE, improvement can be seen because of remediation capabilities of the Edge device.

In this screenshot you can see that VoIP traffic quality was greatly improved by the VCE. This VCE only has one WAN link.

VeloCloud Voice Enhancements

VeloCloud Voice Enhancements

All VCE Management is performed via the VCO in the customer portal. The Enterprise Administrator uses Profiles to manage Edge devices. This makes it very easy for thousands of VCEs to be managed with the modification of a single profile. Enterprise Administrators can also override the profile settings to give individual VCEs a unique configuration that is necessary for it specific site.

Edges can be configured in three main functions. As a default VCE, a Hub, or Internet Backhaul. The default VCE routes traffic as described above leveraging its profile rules and Business Policies. It might connect to a Hub or Internet Backhaul. A Hub is when one or more VCEs act as a central location for other VCEs to connect over VPN. A Hub is generally created at major data centers. An Internet Backhaul is a destination for traffic is routed via Business Policy rules from VCEs back to a single location such as a data center. This is often used for security or compliance purposes. I will provide more information on Business Policies in a future blog.

VCEs are created within the VCO by the Enterprise Administrator and assigned a profile. This profile includes all configuration items for interfaces, Wi-fi, static routes, firewall rules, business policies, VPNs, security services, and more. When the VCE is activated by the VCO, all configuration is pushed to the VCE by the VCO, and the VCE is peered with its primary and secondary gateway and Partner VCGs, if any.

Once the VCE is online, the VCO displays data about the traffic type, source, destination, and quality that passes through each VCE. A world map is displayed that shows all VCE locations and their status in the customer portal.

VCE Monitoring. Applications Tab.

VCE Monitoring. Applications Tab.

There are three ways that the VCE will route traffic. The way that traffic is routed is determined by Business Policies in the Edge Profile. These three routing types are defined as Network Services. They are Multi-Path, Direct, and Internet Backhaul. Multi-Path means that the VCE determines the best carrier for each packet from all WAN links. Each packet is routed to a Cloud or Partner Gateway. Direct is when the Enterprise Administrator routes specific application traffic by defining a single WAN link and does not route through a VCG. Internet Backhaul is described above.

The VeloCloud platform is extremely robust and easy to use at the same time. The ability to configure VCEs and provide security and services to 1000s of sites with a few clicks is nothing short of amazing. If you are looking to improve WAN quality, move away from expensive MPLS, aggregate multiple WAN links, create VPNs across the enterprise, provide security services, and have an easy to use portal to accomplish it all, definitely look at VeloCloud.

Thank you for reading! I will provide details on how to deploy and configure VCO, VCGs, and VCEs in the next blog of this series.

I want to give a shoutout to Cliff Lane at VMware for spending a lot of time answering my numerous questions about how VeloCloud works. Without him, this post would not be possible (or at least correct)! Thanks Cliff!

Home Networking. Part 2 – Foundational Configuration.

Now that the UniFi Security Gateway, or USG, and switches were online and updated to the latest firmware, I was anxious to really start using my VeloCloud Edge. I have access to a VeloCloud Orchestrator that is hosted and managed by VMware. But as an Enterprise Administrator, I can only configure and monitor Edges in a customer environment. There was a lot to the platform that I hadn’t seen. I would have Operator privileges in my own environment!

However, my home lab wasn’t ready because the VeloCloud Orchestrator, or VCO, is distributed as an OVF that requires vCenter to setup the VCO before it boots. I was hoping that I could deploy the OVF through the direct host management that I had been using. I gave it a try and was able to deploy the OVF. However, because I was not deploying through vCenter I wasn’t able to set the host name, password, or SSH keys. After the VCO booted up, I couldn’t log in or do anything. I deleted the VM and turned my attention to VLANs.

Velocloud OVF Configuration

Velocloud OVF Configuration

Because the UniFi switches are only layer 2 capable, they cannot route traffic between VLANs. This means all inter-VLAN traffic must be routed through the USG. Because I planned to have at least 5 separate VLANs, I began to feel concerned about the CPU utilization on the USG. It already would be performing DPI and other security features. Now, it will need to route most of the packets on my network. At the time of writing this, less than 25% of my devices are online. Every day a few more are connected. It will be interesting to how chatty these devices are with just a few human users. Here is the latest usage chart. CPU is sitting at about 25% with a few devices streaming and a couple of people using cell phones and laptops.

USG Performance Chart

USG Performance Chart

Setting up VLANs in the controller software is very easy. It’s configuring firewall rules that I find to be kludgy because they give you about 6 different ways to make the same thing happen. For software that is for home and small/medium business use, I think they should make this simpler and more intuitive. In the screenshot below you see I am creating a new network named Demo. I typed in 10.10.200.1/24, and it automatically populated everything below the Gateway/Subnet box. If the UniFi Site is setup with the correct DHCP and DNS servers, you won’t need to change those settings unless you wish. You’ll notice there are multiple purposes when creating a new network. To create a VLAN as one might expect to use it in an enterprise environment, select Corporate.

Network Creation in UniFi Controller

Network Creation in UniFi Controller

A network set with the Guest Purpose is used for Guest networks where you do not want those devices to access everything such as visitors who want to use Wi-Fi instead of data on their phones. If you want to use tokens or hotspot authentication, that is built into the guest profile and enabling it takes only a few clicks. This is certainly easier than manually setting up those firewall rules.

After creating VLANs for the different types of devices I would have on my network, it was time to prevent communication between the VLANs where it is unnecessary. When I looked at the site settings for routing and firewall, I was amazed. Why does it need to be so complicated? Nine different places I could create a rule seems excessive. To make matters worse, members of the Ubiquiti community give misinformation in the forums as to how to create a firewall rule such as creating an “IN” rule when there should truly be an “OUT” rule. I don’t think this is their fault, it is due to how the GUI is built and possibly to how the USG handles rules. For example, you must create the rules in order in the GUI that you wish for them to be enforced by the USG. You do not get to edit them to reorder them or even set the rule index during creation. This is just silly. There is more flexibility in the CLI, but then we have to get JSON involved for the settings to be remembered whenever the USG is rebooted or provisioned with a new setting. I would suggest a different product if you want simple firewall management at home. I don’t know what that would be. It seems a lot of people like pfsense. I’ve never used it, so I can’t recommend or not recommend.

UniFi Firewall Rules

UniFi Firewall Rules

It has been a very long time since I did anything that resembled real network administration. Many years ago, I spent a few days in San Jose to take Cisco ACE training. I am pretty sure administering the ACE was more intuitive than creating firewall rules in the USG’s GUI. This is saying a lot. But I prevailed and my IoT devices no longer had access to internal systems or the internet. No botnets coming from my house! Not that they’d have the bandwidth to do much destruction to the world.

USG Threat Management

USG Threat Management

Another feature that I’ve decided to turn on in the USG is Threat Management. We all accidentally click a wrong link every so often. Limiting my internet speed to 85 Mbps? No problem! This is another opportunity to look closely at the specs of the USG Pro if you can pull more than 80 Mbps. Since I do not have a Pro, I don’t know what its throughput would be reduced to.

I thought I was finally ready to install vCenter. But alas, I didn’t have a DNS server running on my network. And if I’m going to have a home lab with a bunch of VMs, I certainly need Active Directory. Creating a domain controller for a new forest in a home lab in 2020 is far less nerve-wracking than running DCPromo.exe in 2001 in an enterprise environment, that’s for sure!

After creating A and PTR records in DNS, it was finally time for the VCSA. As all of you probably know, the tiny deployment of vCenter requires 10 GB of RAM. That certainly wasn’t going to fly with my hardware limitations!

Gaming PC ESXi Host

Gaming PC ESXi Host

VCSA Config

VCSA Config

My host came to a grinding halt. I reduced the VCSA to 6 GB of RAM. It could barely boot and could not load the UI. I set it to 8 GB, and at least it ran with minimal complaining long enough for me to deploy the VeloCloud Orchestrator. After that, it was shut down until it was time to deploy a VeloCloud Gateway and subsequently powered off after that.

I was certainly happy to see this login screen after going through host resource gymnastics!

VCO VM

VCO VM

A production VM of VCO wants more resources than my host can provide.

VCO VM Resource Consumption

VCO VM Resource Consumption

Luckily, it is well behaved and only consumes what it needs while powered on.

One last note for before closing out this blog. VeloCloud Orchestrator must have a publicly accessible IP address. The default route must egress to the internet.

VCO OVF Network Selection

VCO OVF Network Selection

This means if you want to do this in your own environment at scale for true internet routing purposes, you might want to have a separate NIC that isn’t hidden behind NAT from something like, a USG for example! There will be many more things that you would need in addition to this, so it is unlikely that an individual would be running their own VCO instance for true SD-WAN multi-pathing into the world. But running it in your home lab to familiarize yourself with the platform behind a firewall and NAT is just fine.

Thank you for reading Part 2! Part 3 will address the VeloCloud architecture. I will describe what the individual components do and how they talk to each other.

Home Networking. Part 1 – The Beginning.

This is a multi-part series on what I’ve learned from my home lab configuration and troubleshooting of Ubiquiti Unifi gear and Velocloud Orchestrator, gateways, and edge devices. Some of this information is already available on the internet, but it took a lot of searching to find it. Some of this information required conversations with the engineers who created the product and is not documented publicly (yet). This blog series is an attempt to consolidate information and links in a single location and simplify some of the mystery of what happens behind the scenes of a Velocloud implementation. I’m happy to answer any questions you might have to the best of my abilities. If it is about Velocloud, I can check with our engineers too.

Phase 1. Requirements.

Cloud and internet. Due to my lack of reliable and highspeed internet connections, security concerns, and my desire to nerd out, everything must be hosted locally and cannot use cloud services.

Wired networks. My home was cabled with cat6, cat6a, and speaker wire to all the locations I might have electronics by professionals. I cannot recommend this enough. Since someone else did it, if a cable doesn’t work, it is on them to fix it. Totally worth it! I also had them install conduits to major locations so it is easy to run new cables in 5-10 years when everything I’ve designed and implemented is obsolete.

I chose to run ethernet everywhere because I find Wifi to be unreliable and generally a pain. With the amount of bandwidth and reliability we need for 4k content and gaming and beyond, ethernet is necessary. Plus, devices like access points and cameras need ethernet for power and security.

Whole home A/V and automation. I didn’t want a closed system (Control 4, Crestron, Savant, etc.) that I was not allowed to configure myself per the manufacturer. And, I didn’t want anything that relied on the cloud to function or stored my data in someone else’s cloud. I choose automation software that has a very large community and runs 100% local. Of course, if I automated something that is controlled in the cloud, that part of the script would be dependent on internet connectivity and the service being online, otherwise, everything is stored and executed locally.

Gear location. I had a closet made specifically for racks of gear with independent cooling. Due to the cost of some cabling types (HDMI over Ethernet), limitations of products on the market, and the laws of physics, some gear couldn’t be stored in the closet which required more cabling to media locations because gear would be mounted there.

Security. I attempted to follow all of the normal best practices: IoT devices can’t talk to the internet or other systems, untrusted guests can’t access internal systems, etc. Any product such as doorbells or cameras that have Bluetooth or Wifi are usually hacked quite quickly and updated for security rarely, so I didn’t choose them. I didn’t buy “smart” appliances or other “smart” home items because security is not those companies’ priority.

Exceptions. Unfortunately, not all of my requirements could be met. And, I still have cell phones, tablets, and a job to do! So, we’re not completely walled off from the world. However, if the internet does become unavailable, we’re able to enjoy media, I can observe and manage my network, and automation still functions.

Phase 2. Setting up the network equipment.

If you follow me on Twitter, then you probably already know that my first issue was the fact that I didn’t have a computer prepared to install Unifi Controller software. Luckily, ESXi runs on pretty much everything these days, like my ancient gaming PC. Within an hour (thank you rural download speeds and my inability to find a large enough USB stick for the iso), I had an ESXi standalone host online. It took a few more hours to have an operating system iso downloaded. I chose to deploy Windows because that’s my most comfortable OS, and I didn’t want to spend time dealing with a less supported version of the controller software. I haven’t had any issues with it yet (after the initial setup as you’ll see below)!

My original choices of gear were the Ubiquiti Unifi Secure Gateway (USG), a Unifi 48 port managed switch, a Unifi 24 port managed POE switch, and multiple Unifi AP Pros. I could have read the product details a closer before placing my order, but a lot was going on in my life, and I made some assumptions I shouldn’t have. Here they are:

·      The USG has deep packet inspection (DPI) capabilities. Yay security and observability! The throughput advertised of the USG is 1 Gbps. Since my rural internet connections come nowhere close to that, I thought I’d be fine. Now, if you turn on DPI, your throughput drops to about 100 Mbps. This is still fine for me, but if you live where you can get fast internet, you will want to get the USG Pro or maybe go a different route altogether (more on that in another post). Also, the USG doesn’t have a rackmount option. The USG Pro does.

·      The Unifi 24 port managed POE switch only has 16 POE ports! The rest are not powered. I did the math on how much power each of my POE devices would probably draw and calculated that the switch could handle that load before ordering. However, I did not check to see how many ports on the switch actually provide power. I ordered a few POE injectors for the remaining devices after realizing my mistake, but I would have purchased different switches if I read the specs closer before placing my order. One cannot return gear to Ubiquiti due to their own mistakes.

·      I did not order a top of rack switch for the second gear rack. When my AV installers showed up and we started discussing the roll out of equipment, I discovered that the cable terminated in different parts of the closet based on a predetermined device layout in the racks. It was expected that the second rack would have devices that needed a switch, so they cabled it that way. I thought there’d be a bit more flexibility. They did provide 4 patch cables between the racks when they ran the rest of the cabling into the closet. If I factored this and my POE switch mistake into the design along with limitations I have since discovered in the USG, my purchase certainly would have looked different.

·      Cat 6a has huge connectors! I already knew Cat 6a didn’t bend well, so we only ran it for certain locations and devices. But, due to the connector size, you cannot have two Cat 6a cables next to each other in the Unifi switch. This is another factor for how many switches to buy and how many ports are needed. Maybe there are switches with ports farther apart than Unifi gear for this reason, I haven’t looked into it.

The Unifi gear was simple to deploy. I was able to just wire up my internet connection to the USG WAN1 port, the USG LAN port to any port on the 48 port switch, and the ESXi host NIC to any port on the 48 port switch and obtain an IP address from the USG DHCP for my host and VMs. The WAN1 port on the USG was configured to use DHCP out of the box as well. After figuring out how to put my satellite modem in bridged mode (Pro tip: disable any VPN software so local addresses can be accessed =), I had internet connectivity. I even placed the Velocloud Edge 510 in line between the USG and modem and had internet connectivity without it being activated and configured. This was nice.

After installing Windows Server, Java, and the Unifi Controller software, it was time to see if setting up the Unifi gear was as easy as everyone said it was. Short answer: Yes! Long answer: not if you think you can change the Unifi devices’ IP addresses!

The Unifi controller software easily discovered the USG and 48 port switch without my having to do anything. There is a simple function called “adopt” so the controller can then manage the Unifi devices it finds. This is all very easy. Except, I didn’t want my hardware to use factory defaults, and I certainly didn’t want it to be using DHCP. The first time, I changed the IP address of the 48 port switch. After the controller software pushed the change, it could no longer find the switch to manage it. I made the change in the controller software so it obviously knows the new IP address. Strange, since the Windows VM was running on a host directly plugged into the switch. I “forgot” the device in the controller, but the software wouldn’t discover it again for me to force an adoption. I spent about an hour looking at community posts on forums about their issues when changing IP addresses (I should have got the hint when every final solution was do a factory reset and adopt the device in the controller software with the defaults) and reading through manuals. I thought I had a plan ready for try number 2 and performed a factory reset on the switch.

This time, I would start with re-IPing the USG. The controller lost its ability to manage USG after I changed its IP address. At least it could talk to it, but it failed when trying to manage it after the change provisioned. My attempts to force the management were thwarted with an authentication issue. Strange again, since the controller software sets the password on adoption. After working through various other documented and community suggested troubleshooting attempts, I gave up. I performed another factory reset and just used the factory defaults. I thought of the other switch and APs that I planned on bringing online and counted the hours of my life I would spend trying to get it to work. I decided it wasn’t worth it. Unless you are extremely lucky, patient, enjoy a cli, and have a lot of extra time on your hands, I don’t suggest wrestling with these devices that definitely do not like their IP addresses changed.

With all these IP changes and abandoned devices, the controller software became very unhappy and couldn’t discover the devices that just went through the second factory reset. I tried a few troubleshooting steps and decided that I may as well just reinstall the software. Minutes later after an uninstall, a reboot, a reinstall, a reboot for good measure, and answering the initial questions, the Unifi controller software was online and able to discover the devices again. I powered up the POE switch, connected it to the 48 port switch, and adopted it in the controller software. I set the DHCP reservations in the USG, performed any remaining updates, and called it a night. If I accepted my factory default fate from the beginning and had faster internet speeds, I probably would have spent no more than two hours on all of this. But hey, I had event tickets to get and dungeons to run in ESO while I waited.

Thanks for reading! Part 2 will be about VLAN creation, firewall rules, and the beginning of the Velocloud edge activation.

Building Your Brand - Like a Blevins

I recently presented this at the Kansas City VMUG UserCon, and the session was a hit. Here is the blog version of that session.

Over the past couple of months, the opportunities I’ve had to discuss careers and brands have increased exponentially through others’ or my own requests. I’ve been mulling over those conversations as well as doing personal reflection. This blog organizes my insights into something that can help anyone who is thinking about an adjustment in her career or life. What I have learned is that the growth and change necessary to build a career is like a second job! It is highly rewarding though. The lessons I’ve learned about improving myself, my outlook, and my actions have a positive influence on all aspects of my life.

The most important thing that you will hear from any self-help book, spiritual guide, or someone who has rebuilt herself from the ashes is that you must do what brings you happiness. Someone recently shared with me that he ranked various parts of his life in order of importance to him like work, family, values, etc. He thought that his rankings might not be “right” and he should have a different order. My response was a very colorful, short, Amanda-like phrase which amounted to, “Don’t listen to that voice.” I reminded him, “You need to do what brings you happiness. You need to take care of yourself.”

Ok, now that we have the obvious yet often elusive happiness thing out of the way, let’s move on to some details. One, we figure out where to start. To do this, we need to ask ourselves questions:

1.     What do I want to be when I grow up (or for now)?
2.     Is it possible and probable?

It is important to be honest with yourself when you answer these questions. Weigh in factors like: Will I find it fun? Am I passionate about it? How will this affect my time with family and friends? Is there too much or not enough travel? What are the work hours? Does it align with my personality? How will it affect my hobbies? What technical skills do I need? What soft skills do I need?

There are no right or wrong answers here. We are gauging the true interest we have in that position and determining if it is the best choice for us based on what else we want out of life. If we think it is a match, great! If not, great! No is always the second-best answer. Just try again with another possibility until the answer to number two is yes.

This list sums up my brand:

1.     Be Tenacious and have Grit. I don't let something go if I believe it is the right thing, and I'll speak up to whoever will listen. I will come with solutions, not just problems. I don’t stop at the first roadblock or the second. There is always another path.

2.     Identify gaps in the company, offerings, or myself. I look for a gap in our offerings. It might not exist. Or if it exists, it might need a massive overhaul. Then I come up with a plan and allies to create the solution. Then I act with #1.

3.     Be confident. This is related to personality, but truly it is an effect of being prepared. I like to feel solid about the subject at hand. If I’m not, I don't feel as confident.

4.     Act as if. If I think I am a good fit for a different position, I start doing that work. At some point, it just makes sense to give me the title. If it doesn't, then I need to be doing something else.

5.     Expect to work a lot of overtime. Mediocrity doesn’t go far. And neither does only forty hours a week – inspiration arrives at any time. Still, work smart. Don’t put in unnecessary hours.

6.     Relax and decompress your brain whenever possible. This means in the middle of the “workday” too – since every day is a workday! Have fun. And I don’t take things too seriously, especially myself.

7.     Be vulnerable. People want to be able to connect with someone on a personal level. This might mean sharing a bit about your weekend, family, hobby, friends, personal interests, or something else that is important to you before diving into business.  You might be lucky if you’re talking to a fellow geek and the technology is your connection, but there is more to share. I find this easier said than done for me, and I am working daily to do this more.

8.     Find a way to communicate with each individual. My default style of communication is straightforward mixed with dry humor. This is fun for a lot of people. Others do not appreciate it as much. It could be off-putting to them. Therefore, it is important to identify how people like to communicate and adapt to their style. This is another item that is easier said than done!

The list above is about me. Some of them might work for you, others might not. It all depends on your personality, your style, what you’re willing to sacrifice, etc. But the list should give you a great place to start on building your own brand. Be sure it is true to yourself and you are comfortable with the characteristics. Most of it shouldn’t be too hard. If one or two things are, that’s ok. If it were easy, it would have already have happened.

Once we know where we want to be and how we will act to get there, the final thing is execution. Gaps might need to be filled in around technical knowledge, articulating strategy, executive presence, business acumen, relationships, or others. Do this with as much help as possible! Grow your network and gain as many mentors as you can. Mentors are people who possess something you want to have. This isn’t necessarily a job. It is more often a quality or characteristic. I’ve also found that I learn from every conversation I have. This might be because of something the other person shared, or it could be something I realized about myself while sharing my experiences. Most importantly, don’t be scared to approach someone. I believe it is my responsibility to start the conversation. It might feel uncomfortable at first, but I have never regretted the action.

Finally, whatever you learn, share! Blog, tweet, give talks, have roundtable discussions, go to lunch, and video blog. Whatever your favorite medium is – go for it! It is important not to hoard information. It doesn’t give you job security or make you better than the next person. Sharing your knowledge with others helps everyone. When other understand what you know, it lifts you up to the next level and frees you for what is next

The Thing about Internet of Things

Internet of Things is no longer a buzz-phrase. It is a full-blown race in every industry to capitalize on the total addressable market trending towards $1 Trillion by the year 2021. The technology industry has not seen this type of opportunity in a long time, possibly ever. Internet of Things gives a chance to companies who have never been in the business of servers, network, security, cloud, data center, or internet to interrupt established companies in these previously dominated arenas. Many companies, large and startup alike, want to have the largest impact on the ecosystem first so others will follow the same design and implementation path.

I have mixed feelings about this boom in IoT. While the exponential technology improvements are exciting, it seems we are ignoring important factors in trying to create the next big thing. My main concerns are security, on-premises analytics, data management, and security. Yes, security is that important and mostly being ignored! In IT we talk about technology debt. If we ignore these important factors, there will be more to resolve than the technology debt of trying to redo poorly implemented systems. Without a design focused on security, there will be a large increase in successful device and network compromises. These compromises will result in a range of outages. The start of the range would be an amusing inconvenience, while the other end of the spectrum would be a complete disruption of important services across the globe.

My recommendations for successful IoT deployments are as follows:

1.     Security. Security should be designed into the IoT device and ecosystem from the beginning. There must be a baseline for IoT security and compliance. The IoT Security Foundation has published an IoT Security and Compliance Framework. All devices should adhere to these practices whenever possible. IoT devices should be scanned for vulnerabilities just like Operating Systems and Appliances are today and kept up to date. This will be an ever-evolving area as more systems are linked to gain more value from the data.

2.     On-premises analytics. There are many platforms for analytics in the cloud. Companies have improved processes and saved money with these platforms already. My favorite story is how Hershey’s will save millions of dollars on Twizzlers production improvements. As a previous employee of a manufacturing company, I do have a soft spot for this industry, but analytics applies to all industries. The more proprietary the data or the more isolated the IoT networks, companies will need to run analytics on-premises for security, compliance, and design reasons. This will require edge computing and storage beyond anything we’ve seen before. IT will need to design the best combination of edge and core analytics to accomplish what their lines of business need. Many companies are trying to consolidate their edge computing instances. Instead, they should be thinking about using SDDC to quickly make their edge computing more uniform in preparations for its growth.

3.     Data management. Hershey’s has 22 sensors on each Twizzlers tank. These sensors collect 60 million data points. These astronomical numbers represent a very, very small fraction of IoT data on the planet today. And those numbers will increase exponentially monthly. It is a good time to be in the data management business! We will need a new approach to this compared to how we’ve managed and retained databases, files, and email in the past. The data that isn’t valuable today, might be valuable in the future. Educated guesses will determine which historical data should be retained either at the edge or within the core at the beginning. As IoT matures, deep learning can be used to identify which values will be useful. The size of the data points will be small, but they will be generated quickly and often. This will also require vast improvements to networks – 400GbE and 5G implementations are right around the corner. We’ll see requirements for IPv6 in every organization when IoT proliferates. Design teams must be created around data management solutions and processes to solve these problems.

There are many more perils than those listed above when pioneering a new era of technology, but those three are the ones that I believe are most often overlooked by product teams and engineers. I look forward to more conversation about the blind spots you have identified in this space.

Women Engineers and More

This week I had the pleasure of attending DevPulseCon hosted by CodeChix. It was impressive to be in a room with about two hundred of the smartest minds in Silicon Valley. There were presentations about choosing the right database for the application, DC/OS, and IoT. It is important to note that all of the attendees and presenters were women. Women who freelance, sit on boards of OpenSource projects, start MeetUps, maintain technology communities, organize conferences, begin StartUps, and pioneer research on technologies I can barely begin to understand. They do all of this on top of their day jobs.

I was a part of two panels focused on discussing the difficulties women face in the technology industry. For those that know me, I’ve never been a fan of “playing the woman card”. I’ve always believed that I’ve gotten to where I am in my career because I am the best for the position. And this is true. My personality and drive has allowed me to overcome the situations that might have deterred others. But it doesn’t mean I haven’t had my own challenges.

Over the past few weeks through conversations with many women whom I respect, I’ve had my eyes opened to the fact that even though I might have thought these obstacles aren’t really there, they are. Yesterday, I was once again reminded how many women truly are discriminated against and thought of as “less than” just because they are women.

There were discussions of pay inequality, team dynamics, general disrespect, and even managers believing that the women didn’t do the work themselves simply because they weren’t men. These weren’t stories that I could dismiss because I read it on the internet and felt removed from the author. These were real women struggling with these issues today.

I’ve gained even more respect for the women that live through this silently to maintain employment, support a family, or to obtain a visa. Know that you aren’t alone. If you are someone who believes that this isn’t real or it doesn’t happen at the company you work at, you are wrong. It is 100% real and as big of an issue as ever. It might be happening to a woman on your team, one you sit next to, or one you see in the halls.

None of us have a magic wand or a simple answer to solve this immediately, but we can all do something – speak up. This isn’t ok. This behavior isn’t humane. It is wrong. Approach this subject with the most compassion and understanding you can muster. Take the time to evaluate your behavior. You might think you’re not someone who perpetuates this environment, and maybe you aren’t. But maybe you are. How do you speak with your men colleagues versus your women colleagues? What does your body language show? Do you do your best to be inclusive at all times? These questions apply to everyone, including women.

I write this blog with a heavy heart. But I am also determined. I’m a Chief Technologist at one of the most respected companies in Silicon Valley. I am the only woman Chief Technologist in the company. I was the first woman Field Principal Engineer. I will continue to advance my career because I love what I do – the technology, the problems I get to solve, the companies I help, and the people I meet every day. But I will also advance my career for every woman in this industry, the ones considering this industry, and the ones who gave up on this industry. I will do it for all of you. I believe in each and every one of you. Together, and by together I mean every single human being, however you identify or don’t identify your gender, can fight this.

If you feel alone and don’t know who you can reach out to, please contact me. If it isn’t me, find someone you trust. You aren’t alone.

Digital Transformation and the Death of Bimodal IT

I recently heard this at a conference: “If you haven’t started thinking about Digitalizing Business in November 2016, you are too late.” The speaker was dramatic for effect; But, I don’t completely disagree. Many enterprises I speak with might be in the early stages of focusing on Digital Transformation. Generally, it is something they have already been doing, sometimes unknowingly. 

Almost every business today has at least one mobile app to reach their customers. It is much faster for me to board a plane, request a car, or order delivery food with an app. Recently I experienced ordering in restaurant with an app on an iPad mounted to the table. I usually prefer the human interaction that goes along with restaurant dining, but this type of interface makes sense for high traffic restaurants with quick diners, such as in an airport.

The above examples are about digitalizing business. In the case of the in restaurant ordering, this restaurant chose a company that sold a platform for the restaurant customers. This platform company had to think about not their own customers, but the customer’s customer to be successful. My ordering experience wasn’t completely intuitive, but I think it was good. I was able to navigate the menu in a reasonable amount of time, and still have plenty of time leftover to browse through the remaining money-making (shopping, travel, etc) apps while my food was prepared.

Let’s think through the steps the company took to create the iPad platform. First, they identified a gap in the market by watching trends and understanding their customers’ businesses. Next, they had to develop their application and provide it in a way that it can be quickly delivered and updated. This generally is done by building microservices-based applications that rely on application resiliency versus infrastructure redundancy. Because of this architecture, they are often run in public clouds and are known as cloud native applications. This is Mode Two of IT Applications and Operations. Mode One refers to running the traditional applications that are installed on Windows and Linux machines and have the standard architecture of Database Server, Application Server, and Web Server. 

Digital Transformation is underway, and Bimodal IT is extinct. While there might be two different types of application architectures, IT should support both of them with the similar toolsets and processes. The idea of a different team to run a select few applications will stretch already thin resources and cause difficulties when it comes to standards, governance, and security. I speak with many IT leaders that already have business units and application owners who use public clouds without partnering with IT. This is dangerous as eventually IT will be responsible for these applications. When such an important group for architecture, operations, and security is left out of the planning, design, and implementation process, there will be gaps in the final solution. These gaps could potentially mean extreme downtime or large security holes in the applications.

There have been many instances in the history of technology where we have transitioned to different types of computing: Analog to Digital, Mainframe to Client-Server, Client-Server to SaaS-based, Data-Center to Cloud, and now Traditional Applications to Cloud Native Applications. Through any transition time, there will probably be two separate teams – one to get started and the other to continue with the previous technology. However, once the organization has started, it is imperative to bring everyone up to speed on the new way of doing things, as it will soon be the standard.

Cloud Is Not Just Cloud Native Applications

This past year, I’ve run across several misconceptions that hinder a company’s adoption of cloud.  This affects their ability to impact the market in their industry.  My blog has been the platform for sharing these barriers with the world.  The latest blockage has been around third platform applications or Cloud Native Applications.

For the record, I believe Cloud Native Applications are absolutely the future.  Enterprises do need to develop their latest solutions as a CNA.  They will be left behind and unable to compete if they do not.  If any company provides their solutions as a SaaS not only to their customer base, but also to others in the same industry as a platform for purchase, they will be well positioned to take the lead.  However, this applies to one, two, or possibly three applications at most today.  What happens to the other hundreds or thousands of platform two or legacy applications left behind?

These applications can still take advantage of many principles of cloud.  They can leverage virtualization at all layers, not just for compute.  They can run in multiple locations around due to hybrid cloud.  Cloud management and monitoring will still apply because of the world that these applications now exist in.  Older management and monitoring tools can no longer understand the abstraction the software-defined data center and hybrid cloud introduce.  Legacy applications can be packaged and deployed in an automated fashion via the cloud management platform.  Platform two applications can also be first class citizens in the hybrid cloud.

While developing cloud native applications and operationalizing the infrastructure team to support this brave new world is a huge task, building a hybrid cloud for the rest is slightly less daunting.  The older applications do not need to change operating systems, development platforms, etc. to run in the cloud.  The same virtualization technology that allows them to co-exist on physical servers is the foundation of cloud.  The software-defined data center is the next step in this evolution.  Virtualizing compute, network, and storage lowers the operational touches exponentially when deploying a new system.  Enterprises that were previously uncomfortable with virtualization due to the black box nature of the network, can freely run their application on virtual machines with network and security virtualization protecting their applications with full transparency.  Also, the security of a system is automated and instantiated when the application is deployed based on pre-determined profiles.  Whether this application is running on-premise or in a public cloud, the same rules apply.  This results in the same security throughout the hybrid cloud.

The solutions are out there to accomplish this.  This isn’t a futures discussion.  Adopting this style of operations is still no easy feat.  When it comes to the trifecta of people, process, and technology, the first two continue to be the most challenging.  However, if they are not equally considered and executed upon, any type of cloud project or IT transformation will not be successful.  The incomprehensible dependency map of applications that exists today would only become more complicated by a partially implemented cloud solution.  Cloud Native Applications are important for the future, but so are the other thousand running today

How to Get What You Want

I have many conversations about IT Transformation.  For the most part, everyone is in agreement, nodding their heads, and saying, “You’re preaching to the choir”.  With all of these technology folks on board, why aren’t more companies building out third generation architectures and creating third platform applications?

It all comes down to one thing:  Money.  IT is usually not a revenue-generating group; more often it is a budget black hole.  The roadblock to deploying new technology is IT’s status of a cost center rather than an innovator and contributor to the company’s profitability.  New possibilities mean IT can rise above its cost center status.  But, this is a bit of a chicken and the egg problem.   The lines of business must trust us to make the right decision.  But usually, they don’t see how these technologies correlate to drilling more oil, selling more merchandise in a store, increasing wealth management, etc.  This is why we must education them in their own language.

We must stop speaking about technology and widgets to these groups.  Their eyes will gloss over at words like Cloud Management System and Software-defined.  Instead we must think of terms of what is important to them - reduced cost and increased speed.  If we can truly provide this in a tangible way to the business, they will be our allies.

For example, a retail store cares nothing about self-service provisioning and highly virtualized data centers.  But as soon as you say, I can help you open the new location 30 days faster than the last and at half the cost, now they are listening.

The difficult aspect of gaining buy-in for this new approach is trust that we can deliver.  Budget must be allocated for the self-service project.  But how do we prove the worth without the dollars to begin?  Everyone will want to use it once it is built, but no one wants to take the risk of funding the implementation.  Below are my methods for kick-starting a rollout of a new concept.

1.    Find a “Friendly”

Enlist a business or application partner who will take a chance on you and a new technology. There will be at least one proponent of leaping into the future in the application side of the house.  Usually, you have helped this person in a previous project, and she sees the value you can bring to the table.

2.    Understand the Friendly’s Next Project

She is probably trying to achieve something new and great as well.  She might also be having a hard time getting the resources for the project or require technology that isn’t readily available in the current environment.  Show how your project will speed up the delivery of hers.  For example, if the Friendly believes that developing a new website or mobile application will unlock billions of dollars in revenue, connect the dots on how a self-service portal will allow her to develop on any platform she wants, instantly, without waiting the ten days for a request ticket to go through or swiping a credit card at AWS.

3.    Uncover “Keep the Lights On” Expenses

Another type of Friendly is the budget owner.  This one may even be more powerful than the first.  Statistics show that 70-80% of an IT budget is spent on keeping the lights on versus innovation.  If the new technology can reduce that spend by 5-10% that is a huge win.  Sit down with this VP to understand line item by line item how the budget is spent in maintenance activities.  This can be a support contract for solutions that can be removed due to redundancy, labor costs associated with break/fix, or replacing older technology on a one to one basis without improvement.  Create a matrix that shows how these numbers are reduced via specific technology capabilities.  Armed with this information it will be easy to show how the project can be self-funded and lower costs for the organization year over year.

4.    Agree to Success Criteria

Your Friendly is your best chance at proving the new way is the best way.  She might not understand what it is you will be able to provide and ask for things that seem blatantly unreasonable.  However, it isn’t malicious behavior on her part.  She is now a kid in a candy store and wants anything her imagination conjures.  It is important to clearly define the features of the new services.  Be very specific about what it can and cannot do in the proof of concept phase.  I do suggest painting the picture of the final vision, but keep her within the guardrails so she can be your advocate upon a successful completion.

5.    Request Good Press

After a job well done, it is imperative your Friendly sings your praises.  Create a project write-up and presentation to outline the specific and tangible benefits you both experienced as a result of this new technology and methodology.  Share this will leadership and request an audience with the CTO and CIO.  Be prepared with a business plan on how this will help the larger organization, not just the single use case from test.  Also, use informal lines of communication.  You want your Friendly to be the advocate for this new technology with her lines of business coworkers.   Illustrating how quickly she was able to begin coding in her brand new development environment, free of all the restrictions of “old IT” will be of large benefit to you.  Larger numbers requiring the same level of service are harder to ignore.

6.    Have a Plan for Production

In addition to the business plan mentioned above, it is important to identify the steps for a quick rollout of the new platform.  If the momentum is continued, there is a higher likely-hood of success.  If the technology is not implemented quickly, the folks who were on board will look for similar possibilities outside of IT.  Often this will be found in the ungoverned public cloud.  If they believe their needs are met, justification for implementation is once again difficult.  And the issue of rogue clouds are introduced or increased, which create management difficulties for IT in general.

The above steps illustrate the effectiveness of an application and business centric approach of IT.  We are now measured on innovation gained, not just cost avoidance.  However, without attention to dollars spent, we cannot gain the support of the business to move forward with the deployment of cloud technologies.

The Path of Cloud

Usually the title of this blog is referred to as the path to cloud.  Based on recent conversations, cloud is already leveraged in most organizations.  Thus, we are in the cloud, not moving to it.  But what is the best way to weave these pockets of cloud together?  I hear questions around the nature of Platform as a Service or public cloud versus private cloud versus hybrid cloud.   Sometimes discussions reflect on how to reign in rouge public cloud use or how to provide seamless access to multiple Software as a Service offerings.  The success of those efforts is the end state.  They should not be goals for the beginning of the journey.

As a former athlete and forever competitor, my experience in developing any skill is to begin with the basics.  All training is focused on ensuring form is correct before leveraging complex movements or thoughts.  This is one hundred percent true of cloud as well.  The purpose of enterprise cloud is for IT to become a service broker, or IT as a Service.   To achieve this, we need to automate everything IT has been trying to do for the last 30 years. 

These are my guidelines for a successful path of cloud:

1.    Solidify virtualization

If your organization is not comfortable with virtualization, this will be a large barrier to cloud.  Ensure the standard architecture for virtualization can support all workloads in performance and capacity.  Implement reference architectures across main data centers and remote sites.  Reduce the variables of physical hardware, software versions, patch levels, and virtual machine templates to a handful of approved types.  Remove the process of negotiating for resources with application owners and developers.  Leverage templates only.

2.    Implement cloud management and monitoring

Once virtualization is standardized across the enterprise, performance and capacity management is key to ensure internal and external customer satisfaction.  Older monitoring tools were not built for cloud agility and hypervisor capabilities.  Capacity management is also different for virtualized environments as the various workloads and peaks running on the same hardware is difficult to account for in manual methods.

This step does not mean all previous tools are discarded.  Instead, the ones providing unique data should be retained and rolled into the main analysis engine.  But, where features overlap, it is an opportunity to prune down the number of solutions in the environment to a manageable number.  This will also lower CapEx and OpEx costs for IT by not maintaining support on redundant software.

3.    Implement configuration management

The solution required for this is not unique to cloud environments.  The important requirements are that all hypervisors and operating systems are supported.  The configuration management system should tie into cloud management and monitoring for a complete root cause picture when diving into issues.

4.    Implement log aggregation and analysis

This toolset is also not unique to cloud environments but should support all standardized infrastructure and applications.  Integration with the management and monitoring tool is ideal for root cause analysis.  Understanding system and application logs is the only way to know the underpinnings of any environment.

5.    Implement cost analysis

The main purpose of cost analysis is not to “Chargeback” to the business units for their overall environment consumption.  Chargeback is a great feature, but more importantly IT must understand its spending.  Also, IT must strive to become an innovation center instead of the cost center that exists today.  Recognizing the least expensive location for workloads and reinforcing workload sizing is critical to success in later stages of cloud.

Once these five steps are complete, or at least well underway, then it is time to automate.  Cloud is automating the request and delivery of a service.  If an empty Operating System is deployed, this is Infrastructure as a Service.  If it is a development platform, a web server, a database server, etc, then it is Platform as a Service.  If the service is to a ready to use application, this is Software as a Service.

The reason I am providing these definitions is I have found, to my surprise, that they differ among the industry.  The delineation of these terms is not to require folks to adhere to them, but to instead show the complexity of each.  Each service builds on the former mentioned in the paragraph above.  Thus, when I am asked where to start with automation, I always suggest Infrastructure as a Service because this is the simplest to provide.  Simple is not to be confused with easy.  When integrating requests with approvals, ticketing systems, IP management systems, networks, security, global DNS, operating systems, backups, monitoring, change management databases, and other tools, automation is a big step.  Do not let this list of integrations deter you.  It is the end state.  The beginning is self-service access to basic virtual machines. 

Keep in mind that the rewards of cloud are vast.  The new measurement for an IT organization is not percent virtualized or time for deployment of a template.  It is not cost per GB of storage or amount of network ports used in a data center.  Innovation percentage versus “keeping the lights on” is the new measurement.  The early adopter organizations are reaching 50% to 75% on the innovation side of the equation.  This is only accomplished through the path of cloud.

Documentation. The Necessary Evil.

Documentation might not be evil, per se, but I have not heard the words, "Oh my gosh!  I get to document my implementation.  That is so exciting!"  Or "I love filling out tickets, change control forms, CMDBs, etc.!"  If someone who feels this way is out there, you are certainly a rare breed and could probably find a job anywhere, at any time.  For the rest of us, documentation is an often-dreaded afterthought, once the great fun of designing and deployment is over.

However, documentation is very important.  Without it, infrastructures, applications, and security function as a black box.  The environment is a mystery to anyone who did not implement the solution from the virtual ground up.  While some might consider this job security, it is not the best way to run an enterprise.  At this point, you might be thinking of the many times documentation would have been useful to you.  But, let me supply some examples nonetheless.

A prime example is the datacenter move.  Whether this move is due to a merger or acquisition or it is time to move to a better-suited location, knowing the ins and outs of the infrastructure and applications is critical to a successful migration.  So many times the project starts with the question, what is running over there?  Followed by, who is using it?  Once those questions have been answered satisfactorily, discovering the physical and logical layouts of said datacenter and applications must be tackled.  Generally, the project is launched into an investigative phase that lasts months or years.  The actual move usually does not take longer than a few scheduled weekends.  Of course, these scheduled outage windows do not imply success.  Surprises occur, and the IT organization spends thirty hours with all hands on deck delving into the system looking for the cause of the issue.  If the answer is not found, the application must be rolled back to the original datacenter.  Sometimes the entire migration project does not move past the discovery phase because the entire task is too daunting.  Enterprises are then stuck in costly datacenters on aging and insecure infrastructure.

Another important situation is the moment disaster strikes.  Maybe a single system or application failed.  Maybe a natural disaster or power event affects the entire datacenter.  Regardless, it is time to bring production workloads online in a short amount of time.  But how do we do this?  We might have backups to restore (fingers crossed that the restore actually works).  Or data could be replicated to a secondary or tertiary site, waiting to be powered on as systems.  If we are successful bringing the individual server online, this only sets our environment into a crash consistent state.   The application might not yet be functional in this scenario.  In addition to assessing the health of the environment, we must identify where IPs and host names are set, what the effect is if those values are changed, what inter-system communication occurs and how, what startup dependencies exist either internally or externally, and so on and so forth.  Stress is already high in a disaster, think how peaceful it would be to have the recovery work completely documented.

The above examples were extreme.  However, other very important needs for documentation arise, no matter how trivial they seem at first glance.  It might be time to promote an application into production that developers have tinkered with for the last six months in a test environment.  Go-live is in a week.  How is that accomplished without thorough documentation?  An application might already be in production, but it is time to integrate it with another system, enhance the security, or just upgrade the application.  Without proper documentation of the system configuration, well, the discovery work will far surpass the originally planned project.

I could provide statistics around how much money companies can save in datacenter migrations and consolidations alone - millions of dollars, often just in power savings.  Or we can consider the likely hood of a large enterprise sustaining a profitable business model after unsuccessfully recovering from a disaster – in the low 10s of percentages.  But it is more interesting to share a really irritating experience I had because I did not provide detailed and clear instructions on an upgrade I was leading.

Years ago when Active Directory integrated DNS was only four years old, I joined a company that still used BIND on their UNIX servers for internal DNS with  Active Directory Forest and Windows DHCP.  After convincing everyone AD integrated DNS was secure and reliable, I migrated the company to AD DNS during a Windows 2003 Forest-wide upgrade.  I performed the upgrades at the main datacenter and then provided instructions to my other team members to replace the Windows 2000 domain controllers in the field.  I basically said, “Check the box to install DNS and choose AD integrated.”

A few weeks after the transition, support calls started rolling in that e-mail wasn’t working.  We all know e-mail is the most important application in a company, regardless of where it is tiered on the official application inventory.  So, this was a big deal.  It was discovered that e-mail was malfunctioning because the mx record mysteriously disappeared in the forward lookup zone.  I quickly manually entered the record on the management console and e-mail was back online.  However, this kept happening every week or two.  My credibility was on the line, not to mention I was not getting enough sleep during my on-call rotation due to email being down at a global company.

I finally pieced together that this would happen after domain controllers were replaced in remote locations after reading through many, many AD log files.  In speaking with the person who touched the last DC, I discovered he created the Forward and Reverse Lookup zones manually after the promotion instead of letting them replicate on their own (as AD Integrated DNS is designed to function).  The a and ptr records would remain, but other useful records in AD DNS, like mx records, would not reappear.  You can imagine my fury at the revelation!  But, it could have all been avoided if I explicitly documented the replacement steps for the administrator instead of telling them in passing, “Install AD Integrated DNS.”

Why am I writing about documentation in the cloud era?  This is the age for 3rd generation applications, and mobile and social apps are the only types that we wish to develop.  Self-healing infrastructures are available, and environments automatically deploy more systems to support the user load as thresholds are reached.  This is all true, but it does not excuse us from documenting.  One cannot automate something until the process is defined.  And how is the process defined?  It is documented; otherwise it is ambiguous and open to interpretation.  This is true for any deployment to the hybrid cloud, as many options exist.  Explicit documentation ensures the system configuration is clear.

I recently visited with a few customers who confirmed my thoughts that new services such as self-service provisioning, automation, and network and security virtualization will causes issues without proper documentation.  Once the application is brought online and runs in the cloud, then the infrastructure and security teams must reverse engineer the implementation to ensure they understand the building block components, the configuration, and the overall support mechanisms required.  I wouldn’t go as far to say the cloud era brings more work to these teams, but it does leave them in a precarious position for day two operations and support.

During a presentation about cloud native applications, I recently asked a co-worker, how does developer and application promotion automation include documentation?  He thought for a moment and said, “That is a really good question that I don’t know the answer to.”  He is one of the brightest minds I know on this subject, so if he is not aware of a solution, it might not exist.  And we must resolve this soon.  Otherwise, we will continue to suffer uncharted applications and datacenters.  As folks retire or move on, the knowledge of how systems function will be lost.  Future workers will start from the beginning; they will reverse engineer the systems that generate millions, if not billions, of dollars in revenue.  They will not able to focus on innovation to neither help the company’s profits nor implement emerging technologies.  And this hurts everyone – the individual and the company alike.

If you are going to change your solution, why not change all of your solutions?

I was reading a blog by a coworker a month ago, and he said “If you aren’t going to change your process, why change your solution?”, or something to that effect.  It stuck with me, obviously.  It was a great blog on IT transformation from a people/process perspective.  I often say “technology is the easy part” when discussing IT transformation, but I’d like to retract that.  Implementing a new piece of technology is the easy part.  Displacing a set of existing solutions to build the new nirvana is not as easily achieved.  In fact, it is so hard, that most companies ignore it.  Forever.

So many sigh with relief when the words “green field” are used as the basis of an upcoming project.  And most everyone cringes at the idea of integrating with and upgrading the “brown field”.  There are business critical applications responsible for thousands of dollars in revenue daily that often function magically running in this black box. Unraveling the rat’s nest could mean dire consequences for the company and could require updating your resume if all is unsuccessful.  However, we cannot continue to look the other way and hope someone else addresses this issue.  Unlike the first days of Active Directory and Exchange, where it was easier to stand up a new system and migrate everything over, now we have thousands of servers and applications to address.

Regardless of the complexity or sheer magnitude of this undertaking, it is imperative that we solve this.  If not, we are only digging ourselves into a deeper hole.  But, the business and IT benefits that justify moving forward with our exciting projects will be canceled out by the costs of managing and maintaining the untouched environment if it remains.

I was recently working with a customer who has a great private cloud implementation, one that ranks in the top 5% of true private clouds in existence.  We were strategizing about the three year roadmap as it relates to business initiatives.  One of the metrics was mentioned was that the CIO wishes to reduce OpEx spend in IT by X amount.  Improving their private cloud would be one place to reduce OpEx by increasing automation, etc.  However, we discovered they still have thousands of physical x86 servers in locations other than the main, strategic datacenters.

We could easily meet the CIO’s reduction in OpEx goal by performing a large datacenter migration while virtualizing those systems rather than trying to squeeze out more efficiencies from their current private cloud build.  It won’t be new and fun, but it is the path with the greatest return.

You might be thinking, I am 95% virtualized, P2V is so 2009.  Great!  You can consider the next iteration in this process when most systems have been virtualized and the organization is moving down the IAAS and PAAS path.  It is likely the management and monitoring tools for these systems are actually from 2009.  Also, it stands to reason these tools do not understand nor function with cloud constructs.  It is very common that these ancient tools are only used for 5% of their, albeit questionable, abilities.

Rationalization of management and monitoring tools is a phenomenal way to reduce CapEx and OpEx spend.  The yearly spend on that software’s maintenance bills and footprint expansion will be avoided.   A rationalization project might not be met with enthusiasm like the deployment of automated day 2 operations.  However, it is the right thing to do.  And it will make the lives of everyone managing the cloud far easier.  This approach will ensure the success of the new platform because the new cloud management system will be able to understand the concepts and flexibility that is cloud.

I will summarize with the title, “If you are going to change your solution, why not change all of your solutions?”  If we only cherry pick the exciting, new technology projects and limit ourselves to deploying a portion of these architectures, we are not meeting the end goal of this new IT.  We want to provide choice, agility, faster time to market, and a reduced cost for everyone, not just those with brand new cloud-ready applications.  Welcome to operating in 2015!

Doing the Right Thing

We are faced with many visions, trends, and buzzwords daily.  It is difficult to decide where to focus.   In addition, so much needs our attention.  Where do we begin?  What deserves the most focus?  I am a firm believer in doing the right thing.  And, by targeting areas where we are most knowledgeable and passionate, we have the greatest impact.  Fortunately, I work in an industry and for a company where my knowledge and passion align.  It is my privilege to work with enterprises around the world and to determine the best path for their technology services teams.

The first step is to understand the business.  Now, don’t roll your eyes.  This isn’t a training class for selling or executive conversation.  This is how we do the right thing.  It is imperative to understand the business.  We cannot think in vague terms such as “reduce CapEx spend” or “become agile”.  We would only be playing boardroom bingo.  We need to roll up our sleeves: read annual reports and press releases, listen to quarterly earnings calls, talk to representatives from each line of business, read consumer feedback, read partner feedback, try it out ourselves, and live it as much as possible.

Are you working with a manufacturing company?  Walk a plant floor.  A hospital?  Follow around a doctor or nurse, carefully.  A financial institution?  Leverage a service from the website.  A retail store?  Go buy something.   A technology company?  Install a product and use it as anyone else would…  You can see where I am going with this.

All of this is hard work.  But, it is worth it!  With this newfound knowledge, we can do the right thing.   Instead of regurgitating sayings like “time to market” and “lower costs”, instead we can make meaningful conjectures about what will truly help the institution we are supporting.  This type of engagement is appropriate for anyone, at any level.  Are you a Cloud Architect wanting to create IT as a Service?  Are you a VP looking to achieve your MBOs?  Are you in sales trying to meet a quota?  By aligning ourselves to exactly what the business needs to be successful, we in turn, will excel at our jobs.  It is a win-win for us all.

If you are thinking, “What’s in it for me?” or “I’ve come this far without doing those things.,” that’s ok.  It is possible to continue in your career without proactive business alignment.  However, you will most likely be limiting yourself in advancement and opportunity.  If you map yourself, a project, or an initiative to a revenue stream, you will greatly increase your chance of success.  There are many things that motivate people, such as money, recognition, impactful contributions, self-satisfaction, et cetera.  I cannot think of a dimension of motivation that would not be fulfilled by aligning to what the business needs. 

By taking this path, you could begin a new program inside your company, create a new position for yourself, sell more products and services to your customers, or be a good corporate citizen.  It would be very difficult for anyone to say no to your proposal when it clearly meets the needs of the enterprise as a whole, not just an area of IT.  And if the company you work for or with vetoes your suggestions in favor of those that do not directly enable the business, then it is possible you are ready to expand your horizons to somewhere that is forward-thinking.

For those looking for more detail around questions you might ask yourself or others, these are a good start:

  • What are the lines of business in the enterprise?
  • How does each line of business generate revenue?
  • What metrics does each line of business use to report success or failure?
  • What type of  industry or consumer insight does each line of business need to be more successful?
  • What types of new ventures is the company considering?
  • Who or what is the target consumer for each line of business?
  • How do the various business units interact today?
  • How do the business units need to interact in the future?
  • What types of merges and acquisitions occur and with what frequency?
  • What new markets is the enterprise entering?
  • What are the CEOs most important initiatives?
  • What are the CIOs most important initiatives?
  • How are the executives measured and what are their compensation plans?
  • What are the company’s competitors doing?  Can the enterprise react quickly enough to or surpass these competitors with new offerings?
  • What will attract new talent to the company?
  • How does the enterprise measure employee satisfaction?
  • How does communication occur to external partners and consumers?
  • What is the biggest concern of the business three years out?

There are many more questions that you may think of after reading this list - please share them!  While the above thoughts are by no means exhaustive, it will begin a great journey of discovery.  Once we are armed with this information, we can create a roadmap of technology solutions and transformation activities that align to these needs and goals.  This roadmap will be the guide for all of us to do the right thing,.

Welcome!

Thank you for visiting Amanda Blevins.com! This blog is mainly a landing page for my thoughts and experiences from my professional life.  I often encounter organizations navigating technology buzzwords and shiney objects, and they wonder if their approach is correct.  Everyone wants to reach the other side, even if that only equates to the successful completion of a single project.  But what is success? And what are the long-term effects of short-term wins? I enjoy crafting the strategy that benefits everyone in the end: the C-suite's goals, shareholders' value, IT leadership's measurable business objectives, and the IT Administrator's enjoyment.  Snippets of these strategies will appear on this blog as inspiration strikes.  Thank you for your current and future visits.  I hope my ideas spark your own and together we shift the industry!