How to Get What You Want

I have many conversations about IT Transformation.  For the most part, everyone is in agreement, nodding their heads, and saying, “You’re preaching to the choir”.  With all of these technology folks on board, why aren’t more companies building out third generation architectures and creating third platform applications?

It all comes down to one thing:  Money.  IT is usually not a revenue-generating group; more often it is a budget black hole.  The roadblock to deploying new technology is IT’s status of a cost center rather than an innovator and contributor to the company’s profitability.  New possibilities mean IT can rise above its cost center status.  But, this is a bit of a chicken and the egg problem.   The lines of business must trust us to make the right decision.  But usually, they don’t see how these technologies correlate to drilling more oil, selling more merchandise in a store, increasing wealth management, etc.  This is why we must education them in their own language.

We must stop speaking about technology and widgets to these groups.  Their eyes will gloss over at words like Cloud Management System and Software-defined.  Instead we must think of terms of what is important to them - reduced cost and increased speed.  If we can truly provide this in a tangible way to the business, they will be our allies.

For example, a retail store cares nothing about self-service provisioning and highly virtualized data centers.  But as soon as you say, I can help you open the new location 30 days faster than the last and at half the cost, now they are listening.

The difficult aspect of gaining buy-in for this new approach is trust that we can deliver.  Budget must be allocated for the self-service project.  But how do we prove the worth without the dollars to begin?  Everyone will want to use it once it is built, but no one wants to take the risk of funding the implementation.  Below are my methods for kick-starting a rollout of a new concept.

1.    Find a “Friendly”

Enlist a business or application partner who will take a chance on you and a new technology. There will be at least one proponent of leaping into the future in the application side of the house.  Usually, you have helped this person in a previous project, and she sees the value you can bring to the table.

2.    Understand the Friendly’s Next Project

She is probably trying to achieve something new and great as well.  She might also be having a hard time getting the resources for the project or require technology that isn’t readily available in the current environment.  Show how your project will speed up the delivery of hers.  For example, if the Friendly believes that developing a new website or mobile application will unlock billions of dollars in revenue, connect the dots on how a self-service portal will allow her to develop on any platform she wants, instantly, without waiting the ten days for a request ticket to go through or swiping a credit card at AWS.

3.    Uncover “Keep the Lights On” Expenses

Another type of Friendly is the budget owner.  This one may even be more powerful than the first.  Statistics show that 70-80% of an IT budget is spent on keeping the lights on versus innovation.  If the new technology can reduce that spend by 5-10% that is a huge win.  Sit down with this VP to understand line item by line item how the budget is spent in maintenance activities.  This can be a support contract for solutions that can be removed due to redundancy, labor costs associated with break/fix, or replacing older technology on a one to one basis without improvement.  Create a matrix that shows how these numbers are reduced via specific technology capabilities.  Armed with this information it will be easy to show how the project can be self-funded and lower costs for the organization year over year.

4.    Agree to Success Criteria

Your Friendly is your best chance at proving the new way is the best way.  She might not understand what it is you will be able to provide and ask for things that seem blatantly unreasonable.  However, it isn’t malicious behavior on her part.  She is now a kid in a candy store and wants anything her imagination conjures.  It is important to clearly define the features of the new services.  Be very specific about what it can and cannot do in the proof of concept phase.  I do suggest painting the picture of the final vision, but keep her within the guardrails so she can be your advocate upon a successful completion.

5.    Request Good Press

After a job well done, it is imperative your Friendly sings your praises.  Create a project write-up and presentation to outline the specific and tangible benefits you both experienced as a result of this new technology and methodology.  Share this will leadership and request an audience with the CTO and CIO.  Be prepared with a business plan on how this will help the larger organization, not just the single use case from test.  Also, use informal lines of communication.  You want your Friendly to be the advocate for this new technology with her lines of business coworkers.   Illustrating how quickly she was able to begin coding in her brand new development environment, free of all the restrictions of “old IT” will be of large benefit to you.  Larger numbers requiring the same level of service are harder to ignore.

6.    Have a Plan for Production

In addition to the business plan mentioned above, it is important to identify the steps for a quick rollout of the new platform.  If the momentum is continued, there is a higher likely-hood of success.  If the technology is not implemented quickly, the folks who were on board will look for similar possibilities outside of IT.  Often this will be found in the ungoverned public cloud.  If they believe their needs are met, justification for implementation is once again difficult.  And the issue of rogue clouds are introduced or increased, which create management difficulties for IT in general.

The above steps illustrate the effectiveness of an application and business centric approach of IT.  We are now measured on innovation gained, not just cost avoidance.  However, without attention to dollars spent, we cannot gain the support of the business to move forward with the deployment of cloud technologies.

The Path of Cloud

Usually the title of this blog is referred to as the path to cloud.  Based on recent conversations, cloud is already leveraged in most organizations.  Thus, we are in the cloud, not moving to it.  But what is the best way to weave these pockets of cloud together?  I hear questions around the nature of Platform as a Service or public cloud versus private cloud versus hybrid cloud.   Sometimes discussions reflect on how to reign in rouge public cloud use or how to provide seamless access to multiple Software as a Service offerings.  The success of those efforts is the end state.  They should not be goals for the beginning of the journey.

As a former athlete and forever competitor, my experience in developing any skill is to begin with the basics.  All training is focused on ensuring form is correct before leveraging complex movements or thoughts.  This is one hundred percent true of cloud as well.  The purpose of enterprise cloud is for IT to become a service broker, or IT as a Service.   To achieve this, we need to automate everything IT has been trying to do for the last 30 years. 

These are my guidelines for a successful path of cloud:

1.    Solidify virtualization

If your organization is not comfortable with virtualization, this will be a large barrier to cloud.  Ensure the standard architecture for virtualization can support all workloads in performance and capacity.  Implement reference architectures across main data centers and remote sites.  Reduce the variables of physical hardware, software versions, patch levels, and virtual machine templates to a handful of approved types.  Remove the process of negotiating for resources with application owners and developers.  Leverage templates only.

2.    Implement cloud management and monitoring

Once virtualization is standardized across the enterprise, performance and capacity management is key to ensure internal and external customer satisfaction.  Older monitoring tools were not built for cloud agility and hypervisor capabilities.  Capacity management is also different for virtualized environments as the various workloads and peaks running on the same hardware is difficult to account for in manual methods.

This step does not mean all previous tools are discarded.  Instead, the ones providing unique data should be retained and rolled into the main analysis engine.  But, where features overlap, it is an opportunity to prune down the number of solutions in the environment to a manageable number.  This will also lower CapEx and OpEx costs for IT by not maintaining support on redundant software.

3.    Implement configuration management

The solution required for this is not unique to cloud environments.  The important requirements are that all hypervisors and operating systems are supported.  The configuration management system should tie into cloud management and monitoring for a complete root cause picture when diving into issues.

4.    Implement log aggregation and analysis

This toolset is also not unique to cloud environments but should support all standardized infrastructure and applications.  Integration with the management and monitoring tool is ideal for root cause analysis.  Understanding system and application logs is the only way to know the underpinnings of any environment.

5.    Implement cost analysis

The main purpose of cost analysis is not to “Chargeback” to the business units for their overall environment consumption.  Chargeback is a great feature, but more importantly IT must understand its spending.  Also, IT must strive to become an innovation center instead of the cost center that exists today.  Recognizing the least expensive location for workloads and reinforcing workload sizing is critical to success in later stages of cloud.

Once these five steps are complete, or at least well underway, then it is time to automate.  Cloud is automating the request and delivery of a service.  If an empty Operating System is deployed, this is Infrastructure as a Service.  If it is a development platform, a web server, a database server, etc, then it is Platform as a Service.  If the service is to a ready to use application, this is Software as a Service.

The reason I am providing these definitions is I have found, to my surprise, that they differ among the industry.  The delineation of these terms is not to require folks to adhere to them, but to instead show the complexity of each.  Each service builds on the former mentioned in the paragraph above.  Thus, when I am asked where to start with automation, I always suggest Infrastructure as a Service because this is the simplest to provide.  Simple is not to be confused with easy.  When integrating requests with approvals, ticketing systems, IP management systems, networks, security, global DNS, operating systems, backups, monitoring, change management databases, and other tools, automation is a big step.  Do not let this list of integrations deter you.  It is the end state.  The beginning is self-service access to basic virtual machines. 

Keep in mind that the rewards of cloud are vast.  The new measurement for an IT organization is not percent virtualized or time for deployment of a template.  It is not cost per GB of storage or amount of network ports used in a data center.  Innovation percentage versus “keeping the lights on” is the new measurement.  The early adopter organizations are reaching 50% to 75% on the innovation side of the equation.  This is only accomplished through the path of cloud.

Documentation. The Necessary Evil.

Documentation might not be evil, per se, but I have not heard the words, "Oh my gosh!  I get to document my implementation.  That is so exciting!"  Or "I love filling out tickets, change control forms, CMDBs, etc.!"  If someone who feels this way is out there, you are certainly a rare breed and could probably find a job anywhere, at any time.  For the rest of us, documentation is an often-dreaded afterthought, once the great fun of designing and deployment is over.

However, documentation is very important.  Without it, infrastructures, applications, and security function as a black box.  The environment is a mystery to anyone who did not implement the solution from the virtual ground up.  While some might consider this job security, it is not the best way to run an enterprise.  At this point, you might be thinking of the many times documentation would have been useful to you.  But, let me supply some examples nonetheless.

A prime example is the datacenter move.  Whether this move is due to a merger or acquisition or it is time to move to a better-suited location, knowing the ins and outs of the infrastructure and applications is critical to a successful migration.  So many times the project starts with the question, what is running over there?  Followed by, who is using it?  Once those questions have been answered satisfactorily, discovering the physical and logical layouts of said datacenter and applications must be tackled.  Generally, the project is launched into an investigative phase that lasts months or years.  The actual move usually does not take longer than a few scheduled weekends.  Of course, these scheduled outage windows do not imply success.  Surprises occur, and the IT organization spends thirty hours with all hands on deck delving into the system looking for the cause of the issue.  If the answer is not found, the application must be rolled back to the original datacenter.  Sometimes the entire migration project does not move past the discovery phase because the entire task is too daunting.  Enterprises are then stuck in costly datacenters on aging and insecure infrastructure.

Another important situation is the moment disaster strikes.  Maybe a single system or application failed.  Maybe a natural disaster or power event affects the entire datacenter.  Regardless, it is time to bring production workloads online in a short amount of time.  But how do we do this?  We might have backups to restore (fingers crossed that the restore actually works).  Or data could be replicated to a secondary or tertiary site, waiting to be powered on as systems.  If we are successful bringing the individual server online, this only sets our environment into a crash consistent state.   The application might not yet be functional in this scenario.  In addition to assessing the health of the environment, we must identify where IPs and host names are set, what the effect is if those values are changed, what inter-system communication occurs and how, what startup dependencies exist either internally or externally, and so on and so forth.  Stress is already high in a disaster, think how peaceful it would be to have the recovery work completely documented.

The above examples were extreme.  However, other very important needs for documentation arise, no matter how trivial they seem at first glance.  It might be time to promote an application into production that developers have tinkered with for the last six months in a test environment.  Go-live is in a week.  How is that accomplished without thorough documentation?  An application might already be in production, but it is time to integrate it with another system, enhance the security, or just upgrade the application.  Without proper documentation of the system configuration, well, the discovery work will far surpass the originally planned project.

I could provide statistics around how much money companies can save in datacenter migrations and consolidations alone - millions of dollars, often just in power savings.  Or we can consider the likely hood of a large enterprise sustaining a profitable business model after unsuccessfully recovering from a disaster – in the low 10s of percentages.  But it is more interesting to share a really irritating experience I had because I did not provide detailed and clear instructions on an upgrade I was leading.

Years ago when Active Directory integrated DNS was only four years old, I joined a company that still used BIND on their UNIX servers for internal DNS with  Active Directory Forest and Windows DHCP.  After convincing everyone AD integrated DNS was secure and reliable, I migrated the company to AD DNS during a Windows 2003 Forest-wide upgrade.  I performed the upgrades at the main datacenter and then provided instructions to my other team members to replace the Windows 2000 domain controllers in the field.  I basically said, “Check the box to install DNS and choose AD integrated.”

A few weeks after the transition, support calls started rolling in that e-mail wasn’t working.  We all know e-mail is the most important application in a company, regardless of where it is tiered on the official application inventory.  So, this was a big deal.  It was discovered that e-mail was malfunctioning because the mx record mysteriously disappeared in the forward lookup zone.  I quickly manually entered the record on the management console and e-mail was back online.  However, this kept happening every week or two.  My credibility was on the line, not to mention I was not getting enough sleep during my on-call rotation due to email being down at a global company.

I finally pieced together that this would happen after domain controllers were replaced in remote locations after reading through many, many AD log files.  In speaking with the person who touched the last DC, I discovered he created the Forward and Reverse Lookup zones manually after the promotion instead of letting them replicate on their own (as AD Integrated DNS is designed to function).  The a and ptr records would remain, but other useful records in AD DNS, like mx records, would not reappear.  You can imagine my fury at the revelation!  But, it could have all been avoided if I explicitly documented the replacement steps for the administrator instead of telling them in passing, “Install AD Integrated DNS.”

Why am I writing about documentation in the cloud era?  This is the age for 3rd generation applications, and mobile and social apps are the only types that we wish to develop.  Self-healing infrastructures are available, and environments automatically deploy more systems to support the user load as thresholds are reached.  This is all true, but it does not excuse us from documenting.  One cannot automate something until the process is defined.  And how is the process defined?  It is documented; otherwise it is ambiguous and open to interpretation.  This is true for any deployment to the hybrid cloud, as many options exist.  Explicit documentation ensures the system configuration is clear.

I recently visited with a few customers who confirmed my thoughts that new services such as self-service provisioning, automation, and network and security virtualization will causes issues without proper documentation.  Once the application is brought online and runs in the cloud, then the infrastructure and security teams must reverse engineer the implementation to ensure they understand the building block components, the configuration, and the overall support mechanisms required.  I wouldn’t go as far to say the cloud era brings more work to these teams, but it does leave them in a precarious position for day two operations and support.

During a presentation about cloud native applications, I recently asked a co-worker, how does developer and application promotion automation include documentation?  He thought for a moment and said, “That is a really good question that I don’t know the answer to.”  He is one of the brightest minds I know on this subject, so if he is not aware of a solution, it might not exist.  And we must resolve this soon.  Otherwise, we will continue to suffer uncharted applications and datacenters.  As folks retire or move on, the knowledge of how systems function will be lost.  Future workers will start from the beginning; they will reverse engineer the systems that generate millions, if not billions, of dollars in revenue.  They will not able to focus on innovation to neither help the company’s profits nor implement emerging technologies.  And this hurts everyone – the individual and the company alike.

If you are going to change your solution, why not change all of your solutions?

I was reading a blog by a coworker a month ago, and he said “If you aren’t going to change your process, why change your solution?”, or something to that effect.  It stuck with me, obviously.  It was a great blog on IT transformation from a people/process perspective.  I often say “technology is the easy part” when discussing IT transformation, but I’d like to retract that.  Implementing a new piece of technology is the easy part.  Displacing a set of existing solutions to build the new nirvana is not as easily achieved.  In fact, it is so hard, that most companies ignore it.  Forever.

So many sigh with relief when the words “green field” are used as the basis of an upcoming project.  And most everyone cringes at the idea of integrating with and upgrading the “brown field”.  There are business critical applications responsible for thousands of dollars in revenue daily that often function magically running in this black box. Unraveling the rat’s nest could mean dire consequences for the company and could require updating your resume if all is unsuccessful.  However, we cannot continue to look the other way and hope someone else addresses this issue.  Unlike the first days of Active Directory and Exchange, where it was easier to stand up a new system and migrate everything over, now we have thousands of servers and applications to address.

Regardless of the complexity or sheer magnitude of this undertaking, it is imperative that we solve this.  If not, we are only digging ourselves into a deeper hole.  But, the business and IT benefits that justify moving forward with our exciting projects will be canceled out by the costs of managing and maintaining the untouched environment if it remains.

I was recently working with a customer who has a great private cloud implementation, one that ranks in the top 5% of true private clouds in existence.  We were strategizing about the three year roadmap as it relates to business initiatives.  One of the metrics was mentioned was that the CIO wishes to reduce OpEx spend in IT by X amount.  Improving their private cloud would be one place to reduce OpEx by increasing automation, etc.  However, we discovered they still have thousands of physical x86 servers in locations other than the main, strategic datacenters.

We could easily meet the CIO’s reduction in OpEx goal by performing a large datacenter migration while virtualizing those systems rather than trying to squeeze out more efficiencies from their current private cloud build.  It won’t be new and fun, but it is the path with the greatest return.

You might be thinking, I am 95% virtualized, P2V is so 2009.  Great!  You can consider the next iteration in this process when most systems have been virtualized and the organization is moving down the IAAS and PAAS path.  It is likely the management and monitoring tools for these systems are actually from 2009.  Also, it stands to reason these tools do not understand nor function with cloud constructs.  It is very common that these ancient tools are only used for 5% of their, albeit questionable, abilities.

Rationalization of management and monitoring tools is a phenomenal way to reduce CapEx and OpEx spend.  The yearly spend on that software’s maintenance bills and footprint expansion will be avoided.   A rationalization project might not be met with enthusiasm like the deployment of automated day 2 operations.  However, it is the right thing to do.  And it will make the lives of everyone managing the cloud far easier.  This approach will ensure the success of the new platform because the new cloud management system will be able to understand the concepts and flexibility that is cloud.

I will summarize with the title, “If you are going to change your solution, why not change all of your solutions?”  If we only cherry pick the exciting, new technology projects and limit ourselves to deploying a portion of these architectures, we are not meeting the end goal of this new IT.  We want to provide choice, agility, faster time to market, and a reduced cost for everyone, not just those with brand new cloud-ready applications.  Welcome to operating in 2015!

Doing the Right Thing

We are faced with many visions, trends, and buzzwords daily.  It is difficult to decide where to focus.   In addition, so much needs our attention.  Where do we begin?  What deserves the most focus?  I am a firm believer in doing the right thing.  And, by targeting areas where we are most knowledgeable and passionate, we have the greatest impact.  Fortunately, I work in an industry and for a company where my knowledge and passion align.  It is my privilege to work with enterprises around the world and to determine the best path for their technology services teams.

The first step is to understand the business.  Now, don’t roll your eyes.  This isn’t a training class for selling or executive conversation.  This is how we do the right thing.  It is imperative to understand the business.  We cannot think in vague terms such as “reduce CapEx spend” or “become agile”.  We would only be playing boardroom bingo.  We need to roll up our sleeves: read annual reports and press releases, listen to quarterly earnings calls, talk to representatives from each line of business, read consumer feedback, read partner feedback, try it out ourselves, and live it as much as possible.

Are you working with a manufacturing company?  Walk a plant floor.  A hospital?  Follow around a doctor or nurse, carefully.  A financial institution?  Leverage a service from the website.  A retail store?  Go buy something.   A technology company?  Install a product and use it as anyone else would…  You can see where I am going with this.

All of this is hard work.  But, it is worth it!  With this newfound knowledge, we can do the right thing.   Instead of regurgitating sayings like “time to market” and “lower costs”, instead we can make meaningful conjectures about what will truly help the institution we are supporting.  This type of engagement is appropriate for anyone, at any level.  Are you a Cloud Architect wanting to create IT as a Service?  Are you a VP looking to achieve your MBOs?  Are you in sales trying to meet a quota?  By aligning ourselves to exactly what the business needs to be successful, we in turn, will excel at our jobs.  It is a win-win for us all.

If you are thinking, “What’s in it for me?” or “I’ve come this far without doing those things.,” that’s ok.  It is possible to continue in your career without proactive business alignment.  However, you will most likely be limiting yourself in advancement and opportunity.  If you map yourself, a project, or an initiative to a revenue stream, you will greatly increase your chance of success.  There are many things that motivate people, such as money, recognition, impactful contributions, self-satisfaction, et cetera.  I cannot think of a dimension of motivation that would not be fulfilled by aligning to what the business needs. 

By taking this path, you could begin a new program inside your company, create a new position for yourself, sell more products and services to your customers, or be a good corporate citizen.  It would be very difficult for anyone to say no to your proposal when it clearly meets the needs of the enterprise as a whole, not just an area of IT.  And if the company you work for or with vetoes your suggestions in favor of those that do not directly enable the business, then it is possible you are ready to expand your horizons to somewhere that is forward-thinking.

For those looking for more detail around questions you might ask yourself or others, these are a good start:

• What are the lines of business in the enterprise?
• How does each line of business generate revenue?
• What metrics does each line of business use to report success or failure?
• What type of  industry or consumer insight does each line of business need to be more successful?
• What types of new ventures is the company considering?
• Who or what is the target consumer for each line of business?
• How do the various business units interact today?
• How do the business units need to interact in the future?
• What types of merges and acquisitions occur and with what frequency?
• What new markets is the enterprise entering?
• What are the CEOs most important initiatives?
• What are the CIOs most important initiatives?
• How are the executives measured and what are their compensation plans?
• What are the company’s competitors doing?  Can the enterprise react quickly enough to or surpass these competitors with new offerings?
• What will attract new talent to the company?
• How does the enterprise measure employee satisfaction?
• How does communication occur to external partners and consumers?
• What is the biggest concern of the business three years out?

There are many more questions that you may think of after reading this list - please share them!  While the above thoughts are by no means exhaustive, it will begin a great journey of discovery.  Once we are armed with this information, we can create a roadmap of technology solutions and transformation activities that align to these needs and goals.  This roadmap will be the guide for all of us to do the right thing,.

Welcome!

Thank you for visiting Amanda Blevins.com! This blog is mainly a landing page for my thoughts and experiences from my professional life.  I often encounter organizations navigating technology buzzwords and shiney objects, and they wonder if their approach is correct.  Everyone wants to reach the other side, even if that only equates to the successful completion of a single project.  But what is success? And what are the long-term effects of short-term wins? I enjoy crafting the strategy that benefits everyone in the end: the C-suite's goals, shareholders' value, IT leadership's measurable business objectives, and the IT Administrator's enjoyment.  Snippets of these strategies will appear on this blog as inspiration strikes.  Thank you for your current and future visits.  I hope my ideas spark your own and together we shift the industry!