Beyond Keeping the Lights On

Perhaps the best way to understand the benefit of change control is to visualize what an out-of-control organization looks like: the staff is overworked, projects don’t get completed on time, security is compromised, audits fail, downtime is frequent, restoration of systems takes longer and longer, there are many middle-of-the-night emergency calls, the budget gets cut and company executives are researching the benefits of outsourcing IT.

Unplanned, unauthorized work - the “silent killer,” as the ITPI calls it - is most often the cause of an IT organization spiraling out of control. Unplanned work is changes made outside the change window, unapproved or untested changes, fixing a sudden outage - all work that does not follow defined processes and detracts from finishing business projects or other planned objectives. Gartner notes that 80 percent of unplanned downtime is caused by people and process issues, including poor change management practices.² Unplanned work not only undermines system availability, but also resources, compliance, security, automation procedures, the success of information technology infrastructure library (ITIL) and configuration management database (CMDB) initiatives, and the value of the IT organization to the business as a whole.

One of the most serious outcomes of unplanned work is the strain it puts on budgets. Analysts find that in many IT organizations, eight out of every ten dollars spent on IT is not contributing to growth of business or enhancing competitive advantage; it is to keep the lights on.”³ Why? Unplanned work can put organizations into constant firefighting mode. Resources are spent on keeping up with frequent emergencies or dealing with complicated troubleshooting during downtime. Unplanned work usually creates even more unplanned work. This means that planned work, which provides strategic value to the business, does not get done. 

Although some may argue that unplanned work is just the nature of IT, high-performing organizations have proven that nothing is further from the truth. The ITPI study showed that those companies with strong configuration and change control processes in place were able to add more services and applications, had very high rates of successful change, and experienced fewer outages/downtimes than those companies without controls in place. Change, not unplanned work, is the nature of IT, and it is up to IT to make sure change is managed and deliberate.

For those organizations that embrace configuration audit and control, there are broad benefits to reap. Change control ensures proof of compliance with detailed change documentation. It helps improve IT governance by minimizing distraction from strategic projects and reducing unplanned work and emergencies. It enhances efficiency with higher change success rates and less rework, and faster troubleshooting and remediation. Configuration control improves system availability by reducing outages caused by untested, unapproved changes that produce unexpected consequences. It accelerates CMDB/ITIL goals with better data integrity through less process circumvention and few out-of-band changes. And it helps organizations make better use of resources and budget.

Impact and Benefit of Configuration Audit and Control

Many of IT’s key projects benefit, and most often thrive, with configuration control in place. In many organizations, these projects fall into the general categories of compliance and security, availability (incident management), change and configuration automation and ITIL/CMDB initiatives. In each of these areas, configuration audit and control is a valuable ally.

Compliance and Security

Companies have adopted methods to meet various compliance regulations as they relate to change and configuration control, but many of those methods are still manually implemented. Not only is this not efficient or sustainable, since most regulations require a detailed audit trail, but manual systems are by nature a form of error and circumvention. Security is also an ongoing concern for organizations. While IT systems are fenced and fortified with passwords and firewalls to prevent external threats, security issues are also about internal events. Either accidental or intentional, internal threats can cause much harm. Automated detection and reconciliation is essential to the security of the system.

With a configuration audit and control solution, IT managers have oversight and visibility to all changes across the infrastructure, regardless of source, detecting unauthorized change and nonconforming configurations. Automated auditing reporting consolidates and documents all changes in a single verifiable audit trail. This capability automates the testing and reporting of critical IT process controls and provides support for all major regulations and industry standards, including Sarbanes-Oxley (SOX), Graham-Leach-Bliley Act (GLBA), Federal Information Security Management Act (FISMA) and more. With comprehensive change data and documentation, IT managers can investigate and remediate security issues, with the ability to locate and identify what changed, when it changed and who changed it. This helps managers make informed decisions about how to correct problems and enforce change protocols.

Configuration control helps IT organizations lessen risks and lower costs while managing compliance and security. It gives IT the tools to proactively discover and manage security and compliance exposures. It enables quick reaction and decision-making when compromises occur, minimizes manual audit testing efforts, and ensures continuous policy compliance. 

Availability

Perhaps the most visible and outward evidence that an IT department is performing as expected is how often systems are available to its internal and external users. Most outages are the result of good people making faulty changes - unplanned, unauthorized and undocumented changes. Without configuration audit and control in place, there is no easy way to determine which change caused the incident or outage.

Without proper controls, downtime can have a cascading or compounding negative effect on IT: it can create more firefighting, which increases the odds of more unplanned or unapproved changes. And many service level agreements (SLAs) include expensive fines if system availability slips below the contracted level.

Configuration control is essential to improving availability, reducing outages and speeding recovery. It helps by detecting unauthorized changes and configurations before they can impact availability. By automating the discovery of unplanned and unauthorized changes, troubleshooting is greatly improved; system changes are quickly identified and isolated. As a result, mean time to repair is shortened, so if there is an outage, problems can be quickly fixed and systems can be up and running in fast order.

Change/Configuration Automation

Change is a fact of life in IT, but how change is implemented and how unauthorized change is tolerated can affect everything from the integrity of data to the ability to deliver service, meet compliance and keep systems up and running. While tools are increasingly being used to automate changes and maintenance tasks, not all changes are made with these tools, and they can be easily circumvented with manual processes, workarounds and unauthorized changes. Automation tools aren’t designed with a system of checks and balances and are often unable to verify that bad changes from outside sources have happened. Configuration drift can soon follow, as well as failed changes, outages, noncompliance and unplanned work.

Configuration audit and control acts as an “overseer” (automated governance) to all change across the IT infrastructure, whether change is made to applications, databases, operating systems, directories or network devices. It also helps enforce change policies by detecting and reporting on every change made by any method, flagging circumvention and unauthorized changes, and discovering configuration errors early in order to minimize troubleshooting and firefighting.

This oversight gained from change control complements change and configuration automation by increasing the organization’s confidence that all systems across the infrastructure are operating in a known and trusted state. This in turn allows staff to trust the accuracy of configuration information when responding to incidents. Because configuration control brings order and integrity to the IT change process, it reduces unplanned work, and in doing so, becomes a valuable ally for an intelligent use of resources and budget.

CMDB

CMDBs are being adopted by many organizations to help fulfill ITIL and information technology service management (ITSM) initiatives. A CMDB acts as a repository of configuration information, providing an overall view of the interdependencies of IT assets and keeping a record of configuration items (CIs) across the enterprise. It is fed information from across a company’s technology domains. And that’s where the trouble begins.

Each of the technology domains may be feeding the CMDB a poor diet of unauthorized, inappropriate and undocumented changes, cooked up by weak control processes. These circumvented changes tend to happen most often “in the heat of battle” during incident response and service restoration. Regardless of intent, it undermines the accuracy of data that pours into the CMDB. Because a CMDB lacks the granularity of detailed change and the ability to reconcile changes to a change and configuration policy, it does not know of and cannot reflect those changes made outside of tools and processes. It reports only on what it has been told.

This creates configuration drift, which compromises the integrity of the system, the CMDB’s ability to provide an accurate service view. This is becoming a common problem: Gartner Research estimates that through 2008, 75 percent of all CMDB implementations will fail to achieve a comprehensive services view of all consolidated IT domains because of poor control processes.⁴ Fortunately, configuration audit and control helps companies accelerate and gain full value of an ITIL/CMDB investment.

Configuration audit and control ensures all changes are detected and reflected within the CMDB, and those changes comply and conform to policy. Configuration control strengthens change and configuration processes by reporting unauthorized change for further investigation. This helps ensure that information within each technology domain is reconciled before entering the CMDB. Configuration control also provides federated access (across the enterprise and all technology silos) to detailed continuous improvement (CI) change history through the CMDB. 

Configuration audit and control is key to keeping the CMDB valuable to a business. Accurate CMDB data is what enables and accelerates the return on investment of ITIL/ITSM processes, and helps organizations realize process improvement.

Controlling More than the Light Switch

What does “in control” look like? An organization in control understands that process matters and that following and enforcing process matters even more. There is total visibility to all change that takes place across the infrastructure. Unplanned work is greatly reduced as a result of having zero tolerance for unauthorized work. This has resulted in a professional staff that is no longer constantly firefighting or getting emergency calls in the middle of the night.

The organization’s CMDB data is accurate and comprehensive, providing full value on the investment. Systems are up and running in the “high nines.”  If outages do occur, they are infrequent, the cause is easily pinpointed and a repair is quickly and successfully made - the first time.  Compliance is met and audits are passed. Strong security measures in place and threats get immediate response.

The organization has implemented configuration audit and control and has change control policies in place that give it visibility across the enterprise. These controls help it mitigate risk, lower costs and operate efficiently.

High performing organizations learn there is more to IT than keeping the lights on. With configuration audit and control in place, they can be agile and quick acting, improve performance and service, and meet the strategic business goals. 

References:

1. IT Process Institute. “Reframing IT Audit and Control Resource Decisions.” IT Process Institute Executive Snapshot, 2006.
2. David Miller. “Hardware High-Availability Programs in Action. (Product Information).” ENT News, June 1999.
3. Daryl Plummer. “IT Must Think Differently, Act Differently and Be Different to Drive Business Growth.” Gartner Symposium/ITxpo, October, 2006.
4. Ronni Colville. “Do All Roads Lead to CMDB?” Gartner 25th Annual Data Center Conference, 2006.