The New Face Of The CIO

The responsibilities of the CIO span a spectrum of managerial tasks, with one end of the spectrum as "supply" - the delivery of IT resources and services to support business functions - and the other end of the spectrum as "demand" -the task of helping the business innovate through its use of technology. Many CIOs admit that balancing both demand and supply is a difficult task. Fortunately, the CIO has a range of new opportunities and tools to help him manage and order these competing priorities. The process starts with an understanding of how new sourcing models can liberate internal resources and funding for strategic business enablement and innovation.

While every CIO plans aligning IT and business strategy, the irony is that they don't have enough time for effective strategic planning. Usually they blame it on demand-side pressures.Look at the challenges confronting the CIO:
The business side complains that their CIOs aren't up to speed on issues confronting the business and can't think through the implications of systems trade-offs, on a business-unit level, for planned implementations or proposed IT investments. At the same time, the business side usually gets confused in making assessments of the relevance of new technologies to safeguard their business competitiveness. More often than not, business leaders say that their CIOs are not proactively bringing them new ideas about how technology can help them compete more effectively.

Part of the problem stems from the inherent conflict of managing supply and shaping demand. CIOs often must meet requirements to reduce total IT spending, for instance, while making investments to support future scenarios-even though these upgrades will increase IT operating costs. It's indeed a tough job - trying to be both a cost cutter and an innovator - and the CIO sometimes compromises one role. Structural issues whereby parts of the organization are under the control of other executives also complicate the job. Business-unit leaders want more IT leadership, but they are wary of CIOs who don't tread carefully along business leaders' boundaries. Strategic IT management calls for making improvements on the demand side. Managing the demand side of the equation broadly covers:
- The financial understanding of costs and benefits,
- Business accountability for IT and
- Clear framework for investments in technologies.
CIOs shift their attention to different aspects of these three core components. As part of the evolution the CIOs shift focus: once operations are stabilized and business credibility has been achieved, emphasis shifts toward working more closely with business leading to opportunities to contribute to strategic initiatives and direction.

In practice, it can be seen that CIOs who meet and exceed business expectations get rewarded with greater participation in their enterprise's business strategy, higher budgets and become favorites with the business side. In most cases, these CIOs tend to have the ear of the CEO through a direct reporting relationship. CIOs need to know not only what the differences are but also how to time the shift; move too soon or too late and credibility with business leaders will suffer.


This month IBM released its findings from the new global study of more than 2,500 chief information officers (CIOs), covering 19 industruesindustries and spread across 78 countries. The study confirms the strategic role played by CIO’s in making their business become visionary leaders of innovation and financial growth. Many CIO’s are getting much more actively engaged in setting strategy, enabling flexibility and change, and solving business problems, not just IT problems

The report replete with innumerable insights is an excellent collection and I started by looking at understanding some themes and associated metrics that preoccupy the CIO’s the most . I was startled to find that more and more CIO’s appear to be genuinely focusing on getting the growth lever of IT and business fire by rightly turning their attention in increased measures towards innovation. Someone quips overtime the role of the CIO is less and less about technology and more and more about strategy. Really hitting the nail on the head. As the role of the CIO itself transforms so do the types of projects they lead across their enterprises, which will allow CIOs to focus less time and resources on running internal infrastructure, and more time on transformation to help their companies grow revenue. CIOs are transforming their infrastructure to focus more on innovation and business value, rather than simply running IT. The report finds that today’s CIOs spend an impressive 55 percent of their time on activities that spur innovation. These activities include generating buy-in for innovative plans, implementing new technologies and managing non-technology business issues. The remaining 45 percent is spent on essential, more traditional CIO tasks related to managing the ongoing technology environment. This includes reducing IT costs, mitigating enterprise risks and leveraging automation to lower costs elsewhere in the business. Obviously not every CIO would make the cut. It’s reported that High-growth CIOs actively integrate business and IT across the organization 94 percent more often than Low-growth CIOs. The study notes that CIOs spend about 20 percent of their time creating and generating buy-in for innovative plans. But High-growth CIOs do certain things more often than Low-growth CIOs: they co-create innovation with the business, proactively suggest better ways to use data and encourage innovation through awards and recognition. 56 percent of High-growth CIOs use third-party business or IT services, versus 46 percent of Low-growth CIOs. The study also found that High-growth CIOs actively use collaboration and partnering technology within the IT organization 60 percent more often than Low-growth CIOs. Even more impressive, High-growth CIOs used such technology for the entire organization 86 percent more often than Low-growth CIOs
Successful CIO’s , the report notes actually blend three pairs of roles. At any given time, a CIO is:
• An Insightful Visionary and an Able Pragmatist
• A Savvy Value Creator and a Relentless Cost Cutter
• A Collaborative Business Leader and an Inspiring IT Manager


Adjusting the mix one pair at a time, the study reports make the CIO’s perform tasks that make innovation real, raise the ROI of IT and expand the business impact.

Other key findings of the survey:
• CIOs are continuing on the path to dramatically lower energy costs, with 78 percent undergoing or planning virtualization projects
• 76 percent of CIOs anticipate building a strongly centralized infrastructure in the next five years.
IBM's CIO Pat Toole has this to say about the findings. In addition to the detailed personal feedback, IBM also incorporated financial metrics and detailed statistical analysis into the findings.The report also highlights a number of recommendations from strategic business actions and use of key technologies that IBM has identified that CIOs can implement, based on CIO feedback from the study.

(Picture Courtesy :IBM)

The Product Maintenance Paradox!

There was a mild tremor in the IT market few weeks back when Siemens threatened to terminate its annual maintenance contract with SAP owing to high price being charged by SAP. I have noted four years back that around half of Oracle's total revenue of $11.8 billion(in 2005)came from maintenance. As a matter-of-fact oracle sometimes used to justify its roll-up strategy based on attractive maintenance revenue streams of acquired products! The current maintenance stream revenue for Oracle continues to be very high. These open the questions :Why do customers pay vendors annual maintenance? What do they get in return to justify this substantial outgo year after year? For end customers, switching software providers is not easy. Most experts and customers agree that replacing big applications from one company with those from another is far too costly (millions at minimum) and time consuming to be worth the bother.
Now Wall street seems to be waking up to the fragile nature of the solid natured maintenance revenue stream of enterprise software vendors.

Courtesy Vinnie saw this CGCowen report written by Peter Goldmacher, Joe del Callar dated Sep 25 - focusing on the license revenue stream of enterprise software vendors.The report highlights

“As we interpret this data, we can't tell if vendors aren’t investing in R&D because customers aren.t buying products, or customers aren’t buying products because vendors aren.t investing in R&D. Regardless of the cause and effect, the trend is clear. Absolute investments in ERP are down dramatically, way beyond the synergies created by scale via M&A. The net result is that customers no longer look at material ERP upgrades as a competitive advantage and therefore vendors are unwilling to increase investments. Customers are watching their ERP vendors generate expanding margins without plowing that profit back into the product and we believe customers are getting resentful”

I certainly believe that the time is now ripe for the Tier I enterprise software market to be disrupted by third-party maintenance providers. With SAP and Oracle said to be realizing gross margins in the neighborhood of 90% on their maintenance business, the economics are simply too strong for third party maintenance providers not to rise up. Some regulatory interventions like antitrust suits may help accelerate the shift,this would embolden service providers to look at the maintenance market form a fresh value perspective.

I must also agree that with the moderate success of some pure play third party service providers, the landscape is changing and customer expectations are increasing and we shall begin to see third party supports coming in and several customers are also demanding annual lease contracts in place of perpetual license – at least in emerging economies. The era of enterprise software as we know may be over. The straightforward calculations about existing customers continuing to pay very high maintenance revenue year after year may prove to be wrong moving forward and in fact may choose to converge into one homogeneous platform and look at saving lot more costs. The real test of one's ability to hang onto customers will not come when maintenance contracts expire but when the major software companies, transition to so-called "service oriented architectures," a fundamental change in the way applications are deployed, integrated and accessed which should accentuate the ability of different vendors to provide ongoing maintenance support and create a new vibrant win-win business therein.

The Coming Video Revolution & The Internet Buildup

Nielsen reported few weeks back that on an average Americans watch 153 hours of TV every month and 3 hours of online video every month. Video ads are beginning to appear in newspapers! Video has a powerful impact, it is easy to share on social platforms, it is much less expensive to produce that in the past, and more and more people have access to it as technology and reach are improving.

Cisco’s recent forecast of IP traffic for the coming years, makes an interesting read. The company expects IP traffic to increase fivefold compared to today, reaching two-thirds of a zettabyte by 2013. That's two thirds of a trillion gigabytes with a compound annual growth rate of 40 percent. In 2013, the Internet will be nearly four times larger than it is in 2009. By year-end 2013, the equivalent of 10 billion DVDs will cross the Internet each month.

An important highlight of the Cisco report : A significant part of that traffic will be online videos, which will make up 90 percent of all consumer IP traffic by 2013, reaching over 18 exabytes per month. Internet video is now approximately one-third of all consumer Internet traffic, not including the amount of video exchanged through P2P file sharing. The sum of all forms of video (TV, video on demand, Internet, and P2P) will account for over 91 percent of global consumer traffic by 2013. Internet video alone will account for over 60 percent of all consumer Internettraffic in 2013.Video communications will be a formidable block in this traffic – Cisco anticipates an increase by around ten times by 2013 (in the next four years). As the global mobile explosion continues, it reflects in mobile video volume as well. Mobile online video watching is also expected to rise spectacularly, reaching 64 percent of the total mobile IP traffic in 2013, growing from 33 petabytes in a month in 2008 to an estimated 2,184 petabytes, this is 2 exabytes, per month in 2013. The increase represents a 131 percent annual growth rate. However mobile IP traffic will still be only 4 percent of the total IP traffic.

The volume growth also factors in expected growth in Peer-to-peer traffic, which is also expected to increase but its percentage of the overall IP traffic will decline by 2013. P2P networks currently use 3.3 exabytes per month and the numbers are expected to grow at a compound annual growth rate of 18 percent. However, overall P2P traffic will decline to only 20 percent of the total consumer IP traffic coming from 50 percent in 2008. Cisco expects file hosting services to grow instead at a much bigger rate of 58 percent per year reaching 3.2 exabytes per month in 2013.

The internet continues to grow. With more and more news of a global economic turnaround, the growth can only accelarate. The support Infrastructure needs to grow faster to keep pace and this seems to be happening. Since 2007, the annual growth rate of international Internet capacity has exceeded 60%. In 2009, international internet bandwidth increased 64%. In 2009, network operators added 9.4Tbps of new capacity—exceeding the 8.7Tbps in existence just two years earlier.No wonder the internet buildout phenomenon is indeed an amazing one

(Picture courtesy : Cisco)

Consumerization Inside Enterprises Reinforces The Idea : IT is Business

When more and more focus is put on innovation, its evolution, growth and in managing innovation while looking through what conventional collaborative mechanism in fusion with powerful mechanisms like internet enabled collaboration could help achieve –all these point to a world of immense possibilities. With a dominant number of internet users poised to take a dip in the virtual world, the virtual world could become more and more real!! Apple and the high tech semicon industry can vouch for the pull from the consumer segment – for both of them, consumer segment happens to be the largest consuming class!

The interesting part is that the consumerization of IT is creating a whole new world, all managed by a new set of rules. The impact of consumerization on enterprise and opportunities to leverage such advances are all groomed in the consumer space itself. The transition of such things into enterprise IT thereby happens automatically – in a way, advances in consumer space dictates the corresponding fallout in the enterprise space. Many of the digital collaboration mechanisms are made available at throwaway prices today. This creates so much pressure inside enterprises such that the IT departments are forced to give corporate users access to the scale and innovation of the consumer market. True, but difficult to believe – right? An analysis of the past shows that in a significant number of cases the technologies that were originally focused on consumer space have made deep impact over time on the enterprise space – Personal computers, search, IM all are shining examples of this powerful trend. Native web companies keep coming out with a lot of full blown but trial offerings that entices lot many more consumers and many a times a revenue and utilization value evolves out of more and more usage of such offerings. In the process, the consumer space gets more and richer forcing successful offering(s) to be pushed into the enterprise –in larger numbers and faster pace.

Consumerization opens up the organization to consumer-grade services that innovate at a much faster pace than the organization can. –providing in the process, an unmatchable potential for handsome returns to business. Productivity could rise as workers become less tied to the office. consumerisation is also forcing massive changes in resource consumption - consumerization offers a path to reducing a company’s carbon footprint by encouraging telecommuting and Internet-based applications run by mega-scale server farms, which are in many cases powered by greener energy sources and are more energy efficient than hardware in corporate data centers. Large corporates are beginning to adopt such technologies aggressively. With an impending explosive growth of communication and broadband capabilities, the medium of virtual reality/world is sure to take a central seat. Clearly the virtual reality movement does not appear to be a fad per se but can help business create and define new frontiers in its growth path. Implementing IT consumerization is not a major technical challenge, but it does need effective organizational change management discipline . What should the CXO’s do in such contexts: Beat the status quo. Break any resistance that comes from outsourcing partners who come in the way of faster adoption of consumer technologies inside the enterprise. By definition, consumerization is at odds with the notion of paying such high or fixed costs and, therefore, is perceived to be against outsourcers’ interests. .Embrace such technologies faster and in innovative ways align them to their business growth plans. Consumer technologies are not a taboo to be shunned - these need to be constantly assessed for their potential for innovative leverage in growing business. This could end up forcing a larger role for IT in Business further reinforcing the idea that IT is Business.

Dell To Buy Perot Systems : Too Little Too Late!

Dell plans to acquire Perot systems. The momentum picks up! Dell expects this deal to position itself as a more formidable player parading both its hardware and services expertise given the fast changing nature of the business and the faster adoption of cloud computing. Dell was as always seen as a laggard when it to comes to services and given that its principal competitors – HP & IBM are now very big in services, Dell had to anyway bite the bullet of acquiring some big service player. Perot’s strengths are mostly in Banking, Financial Services, Healthcare & Government and would to a limited extent help Dell directly with its footprint. The capabilities of Perot system may be more useful to Dell compared to its current customer base. Perot systems customers would have to factor in the new reality of Perot systems ownership changing to Dell, though the existing CEO would continue to run the business. Two things struck me:

A. Dell must have acquired a services company at least two - three years back when it confronted serious growth troubles – At around the same time, HP muscled in to acquire EDS. I predicted that HP may buy EDS in years before ithappened.

B. Perot has limited scale compared to the other global service players and India headquartered service players. Perot’s offshore capability also is generally seen to be quite limited compared to other traditional global players. Valuation looks interesting here: 2.8 billion USD revenue gets a valuation of 3.9USD billion after providing a substantial premium to last quoted trade.

I am not too sure if this move by Dell would perturb IBM or HP given the lack of scale of Perot's operations, while this may give some limited upside to Dell. The corporate integration may get accomplished easily given that both companies are headquartered in Texas. I was actually expecting Dell to make a serious move to get into smartphone market - say by acquiring Palm. It may happen in the future - we will have to wait and see. It would be interesting to see how Oracle (which recently acquired Sun) looks at this development. I am very keen to watch what Cisco does now – it has entered into the more competitive server business (big competition to Dell , IBM, HP) and has more ambitions in unified computing. Cisco cash position and appetite for acquisition is well known and in the recent past there had been rumors of Cisco looking at acquiring Accenture.While am not clear about how this acquisition may decisively benefit Dell, I do believe that Dell’s move may trigger a new momentum in Cisco’s next acquisition move as well!

Update : See Phil's and Vinnie's perspective as well.

The Changing Enterprise Software Momentum

Changing economic sentiments, and diminished IPO market create the perfect storm for the Big 4 MISO - (Microsoft, IBM, SAP, and Oracle)as they see the medium/small sized vendors are beginning to make M&A moves. See this:

-> Adobe acquires Omniture

-> CA buys NetQoS

-> VMware buying SpringSource

-> PE players scooping Skype

-> Intuit buying startup Mint

- >Avaya buys parts of Nortel

Clearly momentum is slowly beginning to hit the M&A circuit - the backdrop has been that this year has seen very few deals compared to last few years averages. Some trends at work that shape the thinking on why the immediate future could see more and more of continued consolidation. Sramana Mitra assesses the prospects of some players. One may ask - why think of enterprise software when consumer technologies and internet infra/app players are getting more attention. As I wrote here, the consumerization of enterprise technology has the potential to open up new powerful combinations. The possibilities of such fusion of different worlds may open up good chances for disruptive innovation - this provides a platform for such an ideal fertile ground that can lead up to a potential business model innovation – so enterprises need to be well prepared to capitalize on such possibilities. What should the CXO’s do in such contexts: Embrace such technologies faster and in innovative ways align them to their business growth plans. Consumer technologies are not a taboo to be shunned - these need to be constantly assessed for their potential for innovative leverage in growing business.

• Strategic acquisitions target vendors with new product presence/ strong recurring revenue streams in well established areas. The cognoscenti keep whispering that large maintenance revenues as an area for potential targets. Look at Oracle’s reported numbers for this quarter: GAAP new software license revenues were down 17%; software license updates and product support was up 6%.Nurturing a profitable and recurring revenue stream will allow many vendors to share overall development and support costs as they weather the next storm. The hunt is on for vendors who fit this bill as megavendors and private equity actively chase after these assets.

• Weaker companies would see much lowered publicly traded vendor valuations. For companies on the prowl for acquisition with a target class, its never been cheaper and easier to acquire a competitor. Most P/E ratio have become quite attractive and fall below the standard 2X to 3X revenue price target.

• International market expansion. the larger vendors express tremendous interest in acquiring new distribution channels, micro industry verticals, and new geographical coverage. Many in the MISO ecosystem see a lot of turbulence and rumors of M&A run fierce as the partners consolidate to gain scale for regional and global delivery.

• Most privately held vendor exit strategies revolve around acquisition not IPO. Many firms with IPO plans have been told by their boards to refocus on revenue growth and partnerships. The intention - use partnership success to both drive revenue growth and attract acquisition by a larger vendor. Many see acquisition by the Big 4 as the best exit strategy at this point in time.

• Newer delivery models get more and more acceptance : If we analyze the standalone new sales numbers we may get to see this trend clearly. MISO on-premise license revenues may keep dropping/stagnating and in some cases record very moderate growth (in specialized areas). The ERP refresh rates may slowly begin to show a downward trend! SaaS and cloud players need to and i would believe will expand their presence beyond FAS,CRM & HCM spaces - this may also include on premise players getting to offer SaaS solutions in niche areas like procurement, PLM etc. Cloud computing models would over time get to become more popular with ease of use, quick implementation times, pay as you go, no infrastructure model a la google or salesforce and are in fact seeing more faster adoptions.

• Net-Net : Despite the expectations of a slow moving economy, end users should assume that the biggest vendors/ faster moving niche vendors will continue their torrid pace of acquisitions. As these acquisitions factor into long term apps strategies and planning for 2010 purchases, users must assume that truly specialized solutions with significant industry footprint will be acquired. Many customers recommend niche vendors that they work with to the megavendors to acquire them so that their investments are deemed to be safe.

Tungsten Replicator 1.0.3 Release

Tungsten Replicator version 1.0.3 is now released and available as a download from Source Forge. Tungsten Replicator provides advanced, platform-independent replication for MySQL 5.0/5.1 with global transaction IDs, crash-safe slaves, flexible filtering, and built-in consistency checking. The 1.0.3 release adds backup and restore, which I described in a previous blog article.

In addition, there are numerous small feature editions and some great bug fixes that raise performance and stability for large-scale deployment. For example, the replicator now goes online in seconds even when there are millions of rows in the history table. This fixes our previous go-online performance which was, er, pretty slow. Thanks to our users in the Continuent forums for helping us to track down this problem as well as several others.

As of the 1.0.3 release we are also starting to offer the enterprise documentation for the open source replicator. I think this provides better documentation all around, not least of all because we can do a better job of maintaining a single copy. Get current replicator documentation here.

The Future of Database Clustering

Baron Schwartz started a good discussion about MMM use cases that quickly veered into an argument about clustering in general. As Florian Haas put it on his blog, this is not just an issue of DRBD vs. MySQL Replication. Is a database cluster something you cobble together through bits and pieces like MMM? Or is it something integrated that we can really call a cluster? This is the core question that will determine the future of clustering for open source databases.

I have a strong personal interest in this question, because Tungsten clustering, which I designed, is betting that the answer is changing in two fundamental ways. First, the problems that clustering solves are evolving, which will in turn will lead to significant changes in off-the-shelf clusters. Second, for most users the new clusters will be far better than solutions built from a bunch of individual pieces.

To see why, let's start with some history of the people who use open source databases and why they have been interested in clustering over the last decade or so. Open source databases have a wide range of users, but there are a couple of particularly significant groups. Small- to medium-sized business applications like content management systems are a very large segment. Large web facing applications like Facebook or GameSpot are another. Then there are a lot of custom applications that are somewhere in between--too big to fit on a single database dual- or quad-core server but completely satisfied with the processing power of 2 to 4 servers.

For a long time all of these groups of users introduced clusters for two main reasons: ensuring availability and raising performance. Spreading processing across a cluster of smaller commodity machines was a good solution to both requirements and explains the enormous popularity of MySQL Replication as well as many less-than-successful attempts to implement multi-master clustering. However the state of the art has evolved in a big way in the last couple of years.

The reason for change is simple: hardware. Multi-core architectures, cheap DRAM, and flash memory are changing not just the cost of databases but the fundamental assumptions of database computing. Pull out your dog-eared copy of Transaction Processing by Gray and Reuter, and have a look at the 1991 price/performance trade-offs for memory inside the front cover. Then look at any recent graph of DRAM and flash memory prices (like this one). For example, within a couple of years it will be practical to have even relatively large databases on SSDs. Assuming reasonable software support random reads and writes to "disk" will approach main memory speeds. Dirt-cheap disk archives are already spread across the Internet. The old graph of costs down to off-line tape has collapsed.

Moreover, open source databases are also starting to catch up with the hardware. In the MySQL community both MySQL 5.4 and Drizzle are focused on multi-core scaling. PostgreSQL has been working on this problem for years as well. Commercial vendors like Schooner are pushing the boundaries with custom appliances that integrate new hardware better than most users can do it themselves and add substantial database performance improvements to boot.

With better multi-core utilization plus cheap memory and SSDs, the vast majority of users will be able to run applications with adequate performance on a single database host rather than the 2 to 4 nodes of yore. In other words, performance scaling is rapidly becoming a non-issue for a larger and larger group of users. These user don't need infinite performance any more than they need infinite features in a word processing program. What's already there is enough, or will be within the next year or two.

Performance is therefore receding as a motivation for clustering. Meanwhile, here are three needs that will drive database clustering of open source SQL databases over the next few years.

1. Availability. Keeping databases alive has always been the number one concern for open source database users, even back in the days when hosts and databases were less capable. This is not a guess. I have talked to hundreds of them since early 2006. Moreover most users just don't have the time to cover all the corner cases themselves and want something that just works without a lot of integration and configuration.
   
2. Data protection. Losing data is really bad. For most users nirvana is verified, up-to-the-minute copies of data without having to worry a great deal about how it happens. Off-site protection is pretty big too. Talk to any DBA if you don't believe how important this problem is.
   
3. Hardware utilization. With the dropping cost of hardware, concerns about up-front hardware investment are becoming somewhat outdated. Operational costs are a different matter. Let's look at power consumption and assume a dual CPU host drawing 250W, which we double to allow for cooling and other overhead. Using recent industrial electricity rates of 13.51 cents per kilowatt/hour in California you get an electric bill of around $600 per year. Electricity is just one part of operational expenses, which add up very quickly. (Thanks to an alert reader for correcting my math in the original post.)
   

We will continue to see database clusters in the future: in fact lots of them. But off-the-shelf clusters that meet the newer requirements in an efficient and cost-effective way for open source databases are going to look quite different from tightly coupled master/master or shared disk clusters like Postgres-R and RAC. Instead, we will see clusters based for the most part on far more scalable master/slave replication and with features that give them many of the same cluster benefits but cover a wider range of needs. To the extent that other approaches remain viable in the mass market, they will need to cover these needs as well.

• Simple management and monitoring - The biggest complaint about clustering is that it's complicated. That's a solvable problem or should be once you can work with master/slave methods instead of more complex approaches. You can use group communications to auto-discover and auto-provision databases. You can control failover using simple, configurable policies based on business rules. You can schedule recurring tasks like backups using job management queues. You can have installations that pop up and just work.
  
• Fast, flexible replication - Big servers create big update loads and overwhelm single-threaded slaves. We either need parallel database replication or disk-level approaches like the proposed PostgreSQL 8.5 log-streaming/hot standby or DRBD. Synchronous replication is a requirement for many users. Cross-site replication is increasingly common as well. Finally, replication methods will need to be pluggable, because different replication methods have different strengths; replication itself is just one part of the clustering solution, which for the most part is the same regardless of the replication type.
  
• Top-to-bottom data protection - Simple backup integration is a good start, but the list of needs is far longer: off-site data storage, automatic data consistency checks, and data repair are on the short list of necessary features. Most clustering and replication frameworks offer little or nothing in this area even though replica provisioning is often closely tied to backups. Yet for many users integrated data protection will be the single biggest benefit of the new clustering approach.
  
• Partition management - In the near future most applications will fit on a single database server, but most organizations have multiple applications while ISPs run many thousands of them. There need to be ways to assign specific databases to partitions and then allow applications to locate them transparently. This type of large-scale sharding is the problem that remains when single application databases can run on a single host.
  
• Cloud and virtualized operation - In the long run virtualization is the simplest cure for hardware utilization problems--far easier and more transparent than other approaches. A large number of applications now run on virtual machines at ISPs or in cloud environments like Amazon for this reason. To operate in virtual environments, database clusters must be software only, have simple installation, and make very minimal assumptions about resources. Also, they need to support seamless database provisioning to as capacity needs rise and fall, for example adding new VMs or provisioning an existing 4 core VM to a larger 8-core VM with more memory as demand shifts.
  
• Transparent application access - Applications need to be able to connect to clusters seamlessly using accustomed APIs and without SQL changes. This is actually easier to do on databases that use simple master/slave or disk block methods rather than more complex clustering implementations. (Case in point: porting existing applications to MySQL Cluster.) Also, the application access needs to be able to handle simple performance-based routing, such as directing reports or backups to a replica database. The performance scaling that most users now need is just not that complicated.
  
• Open source - For a variety of reasons closed approaches to clustering are doomed to insignificance in the open source database markets. The base clustering components have to be open source as some of them will depend on extensions of existing open source technology down to the level of storage and database log changes. You also need the feedback loops and distribution that open source provides to create mass-market solutions.
  

What I have just described is exactly what we are building with Tungsten. Tungsten is aimed at the increasingly large number of applications that can run on a single database. We can help with database performance too, of course, but we recognize that over time other issues will loom larger for most users. The technical properties described above are tractable to implement and we have a number of them already with more on the way in the near future. Master/slave clustering is not just feasible--it works, and works well for a wide range of users.

Still, I don't want anyone to mistake my point. There are many applications for which performance is a very serious problem or whose other needs cannot possibly be met by off-the-shelf software. Facebook and other large sites will continue to use massive, custom-built MySQL clusters as well as non-SQL approaches that push the state of the art for scaling and availability. Analytics and reporting will continue to require ever larger databases with parallel query and automatic partitioning of data as Aster and GreenPlum do. There are specialized applications like Telco provisioning that really do require a tightly coupled cluster and where it's worth the effort to rewrite the application so it works well in such an environment. These are all special cases at the high end of the market.

Mainstream users need something that's a lot simpler and frankly more practical to deliver as an off-the-shelf cluster. Given the choice between combining a number of technologies like MMM, backups of various flavors, cron jobs, Maatkit, etc., a lot of people are just going to choose something that pops up and works. The hardware capability shift and corresponding database improvements are tilting the field to clustering solutions like Tungsten that are practical to implement, cover the real needs of users, and are fully integrated. I'm betting that for a sizable number of users this is the future of database clustering.

p.s., We have had a long summer of work on Tungsten, which is why this blog has not been as active as in some previous months. We are working on getting a full clustering solution out in open source during the week of September 7th. For more information check out full documentation of open source and commercial products here.