AWS Outage & Customer Readiness

Reddit, Foursquare, EngineYard and Quora were among the many sites that went down recently due to a rather prolonged outage of Amazon's cloud services. On Thursday April 21, When Amazon Elastic Block Store (EBS) went offline, it took many of its Web and database servers depending on that storage down. With Amazon working aggressively to set this back right, on Sunday April 24, most of the services were restored back . As promised and as would be expected, Amazon has now come out with a detailed explanation describing what went wrong, and explaining why the failure was so widely felt and why it took that much time to restore back all the services. Some say that measured against Amazon’s promised availability, this lengthy outage would mean that Amazon may need to maintain full availability for more than a decade to adhere to their promised availability service level commitments.

Now, let’s examine what happened and how this happened. To start with some basics: Amazon has its facilities spread out around the world. Most users would know that its cloud computing data centers are in five different locations. Virginia, Northern California, Ireland, Singapore, and Tokyo. These centers are so architected that within each of these regions, the cloud services are further separated into what Amazon calls Availability Zone. The availability zones are within themselves self contained with physically and logically separate groups of computers setup therein. Amazon explains that such an arrangement helps customer choose the right level of redundancy as appropriate to their win needs. Such a design with a spectrum of options helps customers choose the right level of robustness also when they for a premium choose to host them in multiple regions. The logic here is that hosting in multiple availability zones within a same region must provide comparable robustness (as in hosting across multiple regions) but would come with a much better economics benefitting the customer.

Amazon offers several services as part of this arrangement. Amongst those services, Elastic Block Store(EBS) is an important service. With EBS, Amazon provides mountable disk volumes to virtual machines using the more well known Elastic Compute Cloud(EC2). This is quite attractive to customers, as Amazon with this service, provides the virtual machines with huge amount of reliable storage – typically this gets used for database hosting and the like. The powerfulness of this feature can be seen by the fact that while this can be used from EC2, another Amazon feature called Amazon Relational Database Service( RDS) also uses this as a data store. As an added feature for its services, Amazon has designed this feature for high availability purposes and replicates data through EBS between multiple systems. Given the volume and variety involved therein, this process is highly automated. In such an arrangement, if for some reason an EBS node loses connection form its replica, instantly an alternate storage within the same zone is made available to maintain connectivity.

As per Amazon, while doing routine maintenance operations in Virgnia operations on April 21, engineers were trying to make a change in network configuration to the zone. As part of the process, traffic to the routers affected apparently got moved into a low capacity network as against getting moved onto a backup. The low capacity network, is meant for handling inter node communication and not large scale replication/data transfer internally between the system and so the additional traffic made the network malfunction. With the primary network brought down for maintenance and the secondary network completely mal-functioning the EBS nodes lost their ability to replicate for want of nodes. This is where the unintended consequence of automation began to rear its ugly head. Every system in this network acted as if they are at risk and began to frenetically look for available nodes with free space for replication. While Amazon tried to restore the primary network, damage has been by then done, with all the available space within the cluster were already used, while some remaining nodes continued their search for nodes with free space available – while such nodes with free space were not available.

With a massive deadlock of nodes trying to find replicas, while there were not nodes with free space, impacted the control system’s performance. The control system performance issue severely impacted execution of new service requests like creating a new volume. A long back up began to get created for the slow control system to act upon and this with time reached catastrophic proportions, with some requests beginning to get returned with fail messages. Now, comes the second but the most crucial part of the outage – unlike other services, the control systems span across the region and not the individual availability zones. The net impact was therefore experienced across different availability zones. Remember the idea of Single Point Of Failure? That was proven here in its full might.

Slowly and deliberately, Amazon began the course correction – by beginning to tend to the control system and by adding more nodes to the cluster. Over time, the backlogs on the control system began to get cleared and this took painful efforts and a lot of time in the process. Outages of public cloud systems have made news in the past but clearly with time, the body of knowledge and maturity levels ought to improve things. Cloud service providers make high availability as the cornerstone of their offerings but this outage would in many ways, put such claims to question. Even while this outage happened with Amazon Virginia operations, there were many users of AWS, who managed to maintain availability of their system. A majority of those installations had fall back in terms of multiple regions, multiple zone coverage. Such moves necessarily bring cost, complexity equation into consideration.

It’s a little odd to see that when the problem of non availability of nodes happened, Amazon almost began to get into a denial –of-service attacks within their environment . Amazon now claims that this aspect of crisis related actions have been set right but one may have to wait till next outage to see what else could give way It may be noted that Amazon cloud services suffered a major outage in 2008 – the failure pattern looks somewhat similar upon diagnosis.Clearly, the systems need to operate differently under different circumstances – while it’s normal for nodes to keep replicating on storage/access concerns, the system ought to exhibit different behavior with a different nature of crisis. With the increasing adoption of public cloud services, certainly the volume, complexity and range of workloads would increase and the systems would get tested under varying circumstances for availability and reliability. All business and IT users would seek answers to such questions as they consider moving their workloads onto the cloud

It is interesting to see how Netflix, a poster user of Amazon cloud services managed to survive this outage. Netflix says,” When we re-designed for the cloud this Amazon failure was exactly the sort of issue that we wanted to be resilient to. Our architecture avoids using EBS as our main data storage service, and the SimpleDB, S3 and Cassandra services that we do depend upon were not affected by the outage”. Netflix admits that their service ran without intervention but with a higher than usual error rate and higher latency than normal through the morning, which is the low traffic time of day for Netflix streaming. Amongst the major engineering decisions that they implemented to avoid such outages includes designing things as stateless applications and maintain multiple redundant hot copies of the data spread across zones. Netflix calls their solution –“ Cloud Solutions for the Cloud” as the claim here is that instead of fork-lifting the existing applications from their data centers to Amazon's and simply using EC2, with their approach they believe that they have fully embraced the cloud paradigm. Essentially, Netflix has automated its zone fail-over and recovery process, hosted its services in multiple regions while reducing its dependence on EBS.

Clearly there are ways to get the best of cloud – except that some of these may have different economics and would call for greater ability to engineer and manage the operations. Amazon may have to increase the level of transparency in terms of their design and the operational metrics need to cover many more areas of operations as against the narrow set of metrics that users get to see now. To sum up , I would hesitate to call AWS as failure of the cloud but this journey into the cloud would call for more preparation and better thought out design to be in place from user’s side.

The Cloud & Enterprises

This is in continuation of the previous note:

Thomas Friedman makes the case that Value creation is becoming so complex that no single firm can master it without closely collaborating with a wide set of partners. John Hagel brings this up :” We are shifting from a world where the key source of strategic advantage was in protecting and extracting value from a given set of knowledge stocks — the sum total of what we know at any point in time, which is now depreciating at an accelerating pace — into a world in which the focus of value creation is effective participation in knowledge flows, which are constantly being renewed”. All these thoughts presuppose or recognize the role that information technology plays in making this shift happen. Extending the thought, once can see that from an infrastructural perspective, externalization of data and processes, for example through cloud computing, can create a secure foundation for collaboration that will eventually be indispensable. This flow and collaboration – critical components in the shift becomes so important that it is worth dwelling a little more into this theme.

With the global competitive forces getting more and more powerful one case that business around the world are keen to get more agile and more lean. With dependence in IT increasing with time, solutions centered on IT get more significant. With cloud as an enabler to such a change, one can see many things are coming together to make benefits get realized. As business tends to focus on getting the easy to do business with tag, the ease of provisioning extranets makes the organization more agile as it establishes lightweight, short term partnerships and outsources granular services to external providers. When information and goods flow across borders and enterprises, the concern of rising transactional cost is bound to arise. With well designed cloud solutions , transaction costs can be actually managed better. And by reducing the transaction costs of contractual collaboration the company can effectively leverage external resources without engaging in full scale mergers and acquisitions or setting up joint ventures.

How to engineer a seamless and reliable experience that can not only absorb changes in the external environment but also function as a critical enabler of such change? Look carefully and we can see that at the operational level , an increasing number of data sources are becoming available in the form of web services, truly interoperable and are easy to integrate. Enterprises move really aggressively to make gains on this count – some of them are able to leverage these effectively have an advantage over their competition. The real advantage comes by being able to extract context sensitive, pattern based business intelligence by combining the data sources with their internal information and that of their partners.

This in essence sets the stage of preparing to not only take advantage of emerging technology to stay competitive but also potentially help a set of enterprises to create new standards of competing and thereby create competitive advantage through differentiation. Talking of differentiation, form the perspective of business enabled through the cloud it can be seen that the increased service orientation of cloud squarely uplifts the importance of identifying and analyzing competitive differentiation. Once core functions are established inside enterprises likely centered around core competencies the next question to seek is : determining whether they lead to a business benefit and the larger question therein is whether they are indeed unique and whether the uniqueness is sustainable in the fast changing world?

In such critical turns and decisions, enterprises need to take a far more involvement in activities that may look too mundane and operational. For example, it is a perfectly valid question to ask and keep asking at regular intervals as to how much of IT should be delivered by internal sources. As the technology and technology enabled markets and business services mature, many viable and economical solutions become available for enterprises to consider. And if standardized services (preferably configurable) are available on the market on a more economical footing, then it is obligatory on the part of enterprises to investigate whether it would be possible to leverage them. There may also be alternate forms of delivering the services. Let’s see from an IT perspective - in such a scenario, very effective solutions delivered over the cloud are becoming more and more commonplace. For example, self service portals can reduce human involvement overhead and can thereby lower the costs of basic services. Add ability to configure and integrate – the potential multiplies. Such decisions help enterprises move resources to focus on efforts inside the enterprise that could yield far better returns and may help enterprises become more lean and efficient and in some cases can make them more innovative as well.

Then where does it leave genuinely core processes that are supposed to provide differentiation by design? Where these processes begin to get intertwined with undifferentiated tasks, the effectiveness definitely goes down. Many of the generic IT solutions with customized overlays clearly fall into this category. Such a scenarios also provides enterprises to examine objectively if it would be possible to isolate the generic functions and have them sourced from the most effective and efficient source. Obviously there may not be standard answers for every conceivable scenario but enterprises can think through and decide on embracing appropriate choices.

Now comes the question of horizontal scalability – can the core competencies be looked as a platform to provide a base for a broad range of solutions? Can most of the solutions be plausibly monetized? Too often we see that the competitive advantage can begin to help in gaining business in related areas as well – there cloud solutions can provide can help in providing quick entry and act as a simulation media before eventually becoming a core infrastructure for leverage in steady state on scale up. Similarly IP that can be enabled through cloud can facilitate embracing new business models for cross domain/ cross enterprise usage. Obviously these things don’t happen just by chance – every such possibility needs to be thought through and details worked out in a rigorous manner. When competencies get stretched to serve a more broad base of services, it would invariably call for a realignment of resources and focus inside enterprises. Enterprises then get sucked into taking decisions on designing organization structures ranging from divisions to horizontals.

With market shifts happening more frequently – the dynamism with which enterprises monitor and prepare for them increase more rapidly. Too often, today we see that corporate strategies are reflecting upon changes across all stakeholders – competition, suppliers, customers besides geographies and market segments. This is a more complex game but technology and cloud by extension can provide more strategic enabling support. Such changes can foist huge demands on enterprises –some of them could be very direct and some of them could bring in an indirect but overbearing expectations on the business. The utility model is not just limited to computing CPU cycles and counts the saving. Its actually about making a range of services available on demand – information consulting, data streams, business processes, real time collaboration etc. The reality is that almost all the industries would have a need to consume such services as they begin to navigate the effects of changes that are happening in their industries and in some case extend such services when they act the role as providers. The lesser recognized part of the equation viz. the indirect impact : this can be more powerful and with a larger reach. How? In this complex web of business, enterprises which don’t provide such services may have to engage in transactions with others that do provide such services. Now one will have run as fast as the ecosystem to at least hold on to the current competitive position ( in some cases –in fast changing industries, one will have to run faster to hold on to the position). So the moral here is : no enterprise is likely to be immune from this sort of change and this is going to create a series of cascading changes across the business landscape.

The fact remains cloud provides a very huge canvas. By its huge capabilities and reach , the cloud can effectively change the business dynamics along with the progressions that it creates and this can simply dominate careful setups laid inside enterprises. By attacking the cost structure of IT operations and being seen as business friendly, it can find more support in its absorption. And, the truly disruptive phenomenon that cloud is - shall influence this business ecosystem more rapidly and with greater reach : net result – cloud could become the harbinger of change that will accelerate the changes in the partner landscape in this interconnected world.

Cloud Powered Outsourcing

I wrote an Op-Ed piece for Sandhill.com on “Cloud Powered Outsourcing” available here. The focus is to highlight how services firms can leverage the disruptive nature of cloud computing to deliver new value to their enterprise clients. Thought I shall leave here a quick synopsis.

In the fast changing world of technology and outsourcing, many forces are at play. And for the outsourcing industry, any development – be it hardware, software, service delivery or something new – will change the business significantly. Cloud computing is rapidly changing the climate of enterprise IT – and the outsourcing business along with it. The disruptive nature of the cloud model will mean outsourcers will have to adapt for success in the next era.

Two significant factors are currently creating a new wave of change in the outsourcing industry. First, enterprise software is increasingly being delivered over the cloud. This is impacting the hardware commitments that a business makes. 

Second, we are now seeing a situation where high quality IT infrastructure capacity is quickly becoming a commodity. With delivery models being disrupted, IT service providers should get into a mode of investing massively in creating/provisioning an infrastructure that their outsourcing customers can leverage sooner rather than later. This translates to an investment cycle from the service provider side as IT service providers invest heavily in capacity, virtualization, globalization and automation. With these investments, service providers begin to get positioned as players that are able to offer capabilities that can transform the way organizations access and use IT services. 

The advantages of employing a cloud-based model of service delivery are well established now. In certain types of computing environments, embracing clouds can provide significantly lower costs, higher reliability, assured uptime, greatly enhanced flexibility , robust availability and up-and-down scalability configurable in real-time. The contention here is that outsourcing contracts can block move to the cloud. The separation of service lines in outsourcing that forces distinct blocks of services outsourced to different service providers creates significant challenges to embracing a cloud-based service model because both application and infrastructure management services typically come together in the cloud and normally would get supported by a single service provider. Here is a case where the incumbent service provider’s interests conflict with a client trying to adopt a cloud model. Many enterprises have a range of services contractually assigned to different service providers. Today, many buyers have a range of IT services contracts spread across multiple service providers. Under such arrangements, by default, we see that contracts get optimized at the tower level and by extension, the cost and value get sub-optimized at the tower level. Conflicts between the designated service providers managing the different towers get managed by differing contractual terms and it is quite common to see such partners in a conflict.
This is a “first order challenge” – a conflict rooted in the existing models of engagement and so typically calls for a fundamental rethink on the outsourcing model per se!

Moving further, adoption issues could come in the way . However attractive the benefits, the determination as to what, how, where and when to launch a new model of cloud service is always an issue that businesses have to grapple with – like any other operational decision. Enterprise outsourcers also find the need to find a solution to the dilemma: Do they continue, modify or disband their existing outsourcing arrangements to embrace the new model and reap the benefits in terms of savings, ease of use and performance.

In a nutshell, a revised cloud/SaaS model of outsourcing through service providers can help buyers in new ways resulting in lowered costs and improved operational performance - never ending requirements in today’s business world. This calls for a majority of businesses with mature IT environments and outsourcing arrangements in place to look at recasting their existing contracts and embrace a new model of outsourcing governance. This transition won’t happen by the flip of a switch. Moving into a cloud environment for consuming IT services requires a fundamental change in the delivery mechanisms that would impact almost all the stakeholders inside business. Read the full article here.

Analog Dollars Onto Digital Dimes : Challenging Transitions

I was in a meeting with a CIO in New York on Friday where he outlined the onslaught of disruptive innovation in his industry and the broad set of challenges faced therein. The magnitude of the change that his organization needed to face was mindboggling! Just before, I entered to join the meet in the lounge saw this very interesting interview over the television. Jeff Zucker, President and CEO of NBC Universal, has succinctly captured what is going on in the media industry, with his widely quoted comment that the new digital business models are turning media revenues from analog dollars into digital dimes. As he puts it, the key problem for established media companies is thus to come up with new, innovative business models that will enable them to make money and survive as their industry accelerates the transition from a business model based mostly on analog dollars into one based mostly on digital dimes. The ongoing tussle on online advertising is yet another example of the pain transitions bring on.

What a difficult/interesting situation to be in : Lets see how typically leading enterprises face issues centered around disruptive models where the effects –positive if handled well can be far reaching and can turn disastrous when affected negatively. How do well run business handle the shift ? In my view, while it is tempting to look at acting all things material in a startup mode to tide over the situation – reality shows that this would be next to impossible to make it happen. Would-be innovators know that one of their biggest challenges is systematically identifying the innovations with the greatest likelihood of creating disruptive growth. Pick the wrong one, and squander a year or more of focus and investment. It is not by luck based on the draw anymore. By conducting a series of diagnostics, companies in any industry can quickly identify the most promising opportunities. By conducting customer, portfolio, and competitor diagnostics to pinpoint the highest-potential opportunities and the best business models for bringing them to market. Successful approach to make a positive spin around the disruption could be centered around figuring out how to leverage the new innovations to evolve its organization, culture and business models into the future. Too often solution points revolve around assessment and leverage of skills and capabilities of the enterprise in adapting to new changed realities. The challenge would be in figuring out how to leverage existing relationships and expand the adoption quotient across the enterprise ecosystem. A judicious assessment of what part of existing organization could be useful and what needs to ripped, replaced or retired need to be carried out in the most objective manner. Preparing leadership internally to get ready to manage the disruption is a prerequisite to succeed,

In reality, these seemingly ordinary questions/concerns may become the most tough one to crack, but no point in trying to avoid facing reality. In many cases, matter of fact cultural impediments may overshadow other set of concerns in managing the transition. There is no other way forward – except to properly manage the delicate balance between winning, excelling in its current operations, and quickly embracing disruptive innovations and lay a solid plan for growth.