Slouching towards Multi-Master Conflict Resolution

This title is not nearly as snappy as Yeats' line from "The Second Coming," but it will have to do.  Conflict resolution has been a holy grail for us on Tungsten--we had people asking for it when MySQL was still MySQL AB, lo these many years ago.   Well, it's finally starting to happen after about a year of work.  

Let's start with a simple multi-master configuration.  To replicate bi-directionally between two masters, you typically run Tungsten replicator on each master with two replication services on each master.  The first local service reads the master log.  The second remote service is a slave of the other master.  It looks like the following picture:

One of the big problems in multi-master replication is to avoid loops.  Let's say you update something in the SJC master.  The transaction replicates to NYC and appears in the log.  Then it wants to replicate back to SJC and appear in the log there.  If we don't do anything, the poor transaction will loop forever.

Tungsten solves this problem using a filter named BidiRemoteSlaveFilter that runs on slave pipelines  It has a simple job, which is to identify where transactions originated and keep local transactions that are returning from the other master from being applied again.  We use a variety of tricks to "tag" SQL in a way that allows us to deduce the origin--the most common are comments added to SQL statements or specially formatted row updates that we add to user transactions.  As long as you set things up properly and don't break some simple rules you can now replicate bi-directionally between 2 or more masters.

This brings us to conflict resolution.  Conflict resolution prevents incompatible transactions from colliding with each other.  The BidiRemoteSlaveFilter does a simple form of conflict resolution by preventing transactions from the local service from looping back and being applied again.  However, it's a very short hop to filters that that address application conflicts.

Here's a simple example that is next on my list to implement.  It is not unusual to split customers in multi-tenant applications across sites so that they have active updates on only one site and backup copies on the others.  You could imagine a filter that works off a simple file like the following:

[sjc]
   sharks
   athletics
[nyc]
   mets
   yankees

This tells the filter that transactions with shard ID sharks and athletics are allowed on the sjc master from local application.  However, if the nyc master updates these, we will reject the updates when the nyc remote service tries to apply them and generate an error message in the log, or perhaps put them in a special log file for later inspection.  You could even generate a dummy update on the local master that would result in the sjc data being sent back over to correct the nyc DBMS information via replication.

What we have just done is implemented conflict resolution for a system-of-record approach to multi-site data management.   There are many types of conflicts as well as ways to manage them.  Tungsten Replicator filters have a lot of potential to implement other schemes as well.  Filters are pluggable, so there is a convenient escape hatch if you need to do specialized rules of your own.  Meanwhile, there is plenty of scope for Tungsten development to provide useful conflict resolution mechanisms.

The Yeats poem I referred to at the beginning of the article is one of my all-time favorites.  Here are the last two lines:

And what rough beast, its hour come round at last,
Slouches towards Bethlehem to be born?

This could certainly describe a lot of software projects.   However, Tungsten is not like that at all.  We Tungsten engineers wear white lab jackets with pocket protectors and have shiny clipboards to take notes on our breakthroughs.  We rarely slouch.

P.s. Speaking of Tungsten engineers my colleague Giuseppe Maxia and I will be doing a webinar on multi-master replication on Thursday March 31st.  It will be mostly technical with only a small amount of marketing fluff.  As usual Giuseppe has cooked up a cool demo.  Sign up at www.continuent.com if you would like to find out more.

Mobile and Social Innovations

I woke up this morning to see significant developments.

MOSOLOCO – Mobile, Social, Local, Commerce, a term getting popularized by Jason Maynard is really brimming with lot of activities now. We are now seeing that four proximate but distinctive trends are converging to change the way consumers interact, explore, transact and putting pressure on businesses to keep pace and in some cases create entirely new business opportunities.

Amazon is now fully charged to muscle into the Android Ecosystem. Am not just talking about the Amazon Appstore. With the launch of the new Cloud Drive and Cloud Player joining the Amazon Appstore , the strategy is clearly to parade as a full scale service shopping destination for Android device users. With the new Cloud Drive, Amazon in essence has got an update to the Amazon MP3 app for Android, bringing cloud storage into the music buying scheme, and further adds the Cloud Player to Amazon MP3 for streaming the user’s music to any Android device or web computer. This multi device rendering is a key tenet of success for the Kindle platform and Amazon is clearly focused on building on this strength. I know that one other formidable cloud giant has all got everything ready to roll out – may be they were waiting to launch it in the developer’s conference in May. Now Look at what all Amazon can do with their superior personalization and ecommerce capabilities – the value of the platform is beginning to unfold here. Amazon is essentially taking a step forward in the consumerization of Amazon web services.

Tracking another development, CapLinked, a social platform for private investing, announces exponential growth of its platform with over $1 billion now available in potential investment activity. The company claims to have seen an exponential increase in user base sincee Jan 1 2011, with a total of 5500 dealmakers, investors and entrepreneurs now on the platform. This news comes on the heels of CapLinked’s angel funding totaling $900k, led by PayPal alums Peter Thiel, Dave McClure, Joe Lonsdale and Aman Verjee as well as David Anderson of 7th Rig.This rapid growth shows confirms the power of platform and communities - essentially in this case investors and businesses need a common platform to connect – and it is happening now on CapLinked, the LinkedIn-meets-Salesforce for private investing,” says Eric Jackson, CEO of CapLinked and former PayPal Vice President. He adds that In addition, the new CRM tools will help identify potential deals, share information and get the right deals done faster.

Caplinked sees the trillion dollar a year private investing still using old, inefficient technology and points out to the fact most of these enterprises have been stuck in the email-and-spreadsheet mindset for too long. This gives them an opportunity to focus on bringing innovation in the form of social, cloud based, secure software to the world of private investment. The objective is to get investors, advisors, and companies the tools they need to better manage making connections and handling deal flow, due diligence, closings, and company reporting.
Both the entrepreneur and investor are brought together in the same platform. For entrepreneurs raising capital, the tool makes it easier to be transparent and share information. That makes it easier to get money in the door, and keep investors happy for follow on investments. For investors, this helps track their investments in one place, and have an idea of what is going on with their companies. Added to that, this is not a pay-to-play service, the claim here is that by letting entrepreneurs access the social graph, the value increases and so to get more strong social signals, the service strives to keep this platform free for entrepreneurs to raise capital.

We will actually see more and more innovation and success stories in the MOSOLOCO world.

Reinvention, Week 2: Identity Shift

Last week I discussed "reinvention"--a popular buzzword and new imperative for ongoing success in business and life in a culture of constant change--and decided it's time for me to take it on myself. What will it really take for me to reinvent myself? I ask this question seriously and with an actual lump in my throat.

Lately my ongoing attempts to live a creative life and spark creativity in others have not led to enough satisfaction or sense of purpose for me. I seek sustained changes and new results that only full reinvention can spark. My typical back door methods--perhaps a splash of charm or a well-placed creative quip--are no longer cutting it.

It's time for me to jump into the cold water of reinvention and learn how to swim, even if my flesh has to wrinkle and age and prune and peel.

Okay, so let me repeat from last week: Reinvention cannot happen unless there is a fundamental change in belief about myself. I can't just wear purple clothes and say I have been reinvented. I have to change something within--something that shifts my guiding mindset--and let it work its way out. Maybe it's possible to do that instantly but I've never discovered how. Coming up with new ideas is not my problem. But they lead only to occasional reinvigoration. Not real reinvention.

So I've decided to change a fundamental belief--the belief about my identity itself. I've been stuck in whoever I thought I was and not able to bring about changes I wanted in my life. So forget whatever my identity was two weeks ago. As of last week, I took on another, may the flags be unfurled and words now capitalized.

I am a Warrior of Aliveness.

Yes, you heard me. Warrior. Of Aliveness. That's the new identity I have been working from in order to realize reinvention that sticks. Warrior of Aliveness. WOA.

If you know me, you know that I am not a particularly warmongering man. Yet there is something about the uncompromising mission of a warrior that appeals to me right now. And generally I believe in language grounded in specific, concrete reality. "Aliveness" doesn't really do that. But it works for me. I will explain more in the future.

For now, as I've been settling into this new identity, I've realized that I need an ongoing power source to survive in this world. It's not easy to be a Warrior of Aliveness in the thought-mind I currently own or in the 21st century Chicago culture I am surrounded by. So I must find certain kinds of fuel that enable my warrior-fire to burn within. I now go off to seek it.

Parallel Replication Using Shards Is the Only Workable Approach for SQL

There have been a couple of recent blog articles (here and here) asking for parallel replication based on something other than schemas.  These articles both focus on the problem of parallelizing updates within a single MySQL schema.  I read these with great interest, not least because they both mentioned Tungsten (thanks!) and also found that our schema-based parallelization approach is too limited.  It is therefore worth a short article explaining exactly what the Tungsten approach is and why we chose it.

First of all, Tungsten does not exactly use schema-based parallel replication.  Tungsten is actually based on what I call the serialized shard model of replication.  We assign global transaction IDs to all transactions, which means that for any particular set of transactions we can always figure out the correct serialization and apply in the right order.  This is true even if the transactions travel across independent replication paths or if we have master failover.

Second, we assign a shard ID to all transactions.  Shards are independent streams of transactions that execute correctly when applied by themselves in serial order.  Shards are typically independent, which means transactions in different shards can execute in parallel without deadlocking or corrupting data.  This is the case when each shard contains data for a single customer in a multi-tenant application.  We also have a notion of "critical shards," which are shards that contain global data, such as shared currency rate tables.  Updates in critical shards cause full serialization across all shards.  

You can define shards in a variety of ways, but as a practical matter identifying durable shards inside individual MySQL schemas is hard for most applications, especially if there are constraints between tables or you have large transactions.   Many SQL applications tend to make most of their updates to a small number of very large tables, which makes finding stable dividing lines even more difficult.  Schemas are therefore a natural unit of sharding, and Tungsten uses these by default.

Schema-based sharding seems pretty limiting, but for current SQL databases it is really the only approach that works.  Here are some important reasons that give you a flavor of the issues.

* Restart.  To handle failures you need to mark the exact restart point on each replication apply thread or you will either repeat or miss transactions.  This requires precise and repeatable serialization on each thread, which you get with the serialized shard model.

* Deadlocks.  If there are conflicts between updates you will quickly hit deadlocks.  This is especially true because one of the biggest single thread replication optimizations is block commit, where you commit dozens of success transactions at once--it can raise throughput by 100% in some cases.  Deadlocks on the other hand can reduce effective throughput to zero in pathological cases.   Shard-based execution avoids deadlocks.

* Ordering.  SQL gives you a lot of ways to shoot yourself in the foot through bad transaction ordering.  You can't write to a table before creating it.  You can't delete a row before it is inserted.  Violating these rules does not just lead to invalid data but also causes errors that stop replication.  The workarounds are either unreliable and slow (conflict resolution) or impractical for most applications (make everything an insert).  To avoid this you need to observe serialization very carefully.

* Throughput.  SQL transactions in real systems vary tremendously in duration, which tends to result in individual long transactions blocking simpler parallelization schemes that use in-memory distribution of updates.  In the Tungsten model we can solve this by letting shard progress vary (by hours potentially), something that is only possible with a well-defined serialization model that deals with dependencies between parallel update streams.  I don't know of another approach that deals with this problem.

If you mess up the solution to any of the foregoing problems, chances are good you will irreparably corrupt data, which leads to replication going completely off the rails.  Then you reprovision your slave(s).  The databases that most need parallel replication are very large, so this is a multi-hour or even multi-day process.  It makes for unpleasant calls with customers when you tell them they need to do this.

I don't spend a lot of time worrying that Tungsten parallel replication is not well suited to the single big schema problem.  So far, the only ways I can think of making it work scalably require major changes to the DBMS or the applications that use it.  In many cases your least costly alternative may be to use SSDs to boost slave I/O performance.

My concerns about Tungsten's model lie in a different area.  The serialized shard model is theoretically sound--it has essentially the same semantics as causally dependent messaging in distributed systems.  However, if we fail to identify shards correctly (and don't know we failed) we will have crashes and corrupt data.  I want Tungsten either to work properly or tell users it won't work and degrade gracefully to full serialization.  If we can't do one of these two for every conceivable sequence of transactions that's a serious problem.

So, to get back to my original point, serialized shards are the best model for parallel replication in SQL databases as we find them today.  I suspect if you look at some of the other incipient designs for parallel replication on MySQL you will find that they follow this model in the end if not at first.  I would think in fact that the next step is to add MySQL features that make sharded replication more effective.  The drizzle team seems to be thinking along these lines already.

Small Business Social Network

"Experience the new OGoing!"

Create your own small business profile on OGoing, and begin making connections, promoting your business, building relationships, and creating new opportunities today!

OGoing is your new Small Business Social Network!

Get started on the all new, most innovative Small Business Social Network: http://ogoing.com

Share your SMB business news, events, services, deals, ideas and needs in real-time, live on the OGoing Small Business Community now... And it's FREE to join!

7 reasons why you must join OGoing today:

• VERY easy to use, and FREE to join!
• FOR your business! Find and Connect with local small businesses instantly
• MARKET your business! Promote your products and services to your local markets
• DO more! Find new referrals for your business, and post your business needs now
• TALK more! Share real-time updates about your business deals, news and events
• POWER your brand! Promote your brand on Twitter and Facebook, and cell phones
• GROW your business! Expand your local contacts, brand and sales

DRIVE your small business forward with OGoing! Go BIG with OGoing!

Who should join?

Small Businesses, Entrepreneurs, Startups, Professionals, Consultants, Sellers, Buyers, Procurement

Get started on the all new, most innovative Small Business Social Network: http://ogoing.com

OGoing - Drive Your Business Forward!

If you need private OGoing to create your exclusive business community, get started here:
http://ogoing.com/enterprise

Sincere regards,

Sanjay Dalal
CEO & Founder
OGoing Inc.
Follow me:
http://ogoing.com/sanjaydalal

Reinvention now!

"In today’s environment of superacceleration, catch-up is a fool’s game. There is no advantage in keeping up. Forget about trying to compete. Instead, leapfrog the competition by redefining anything and everything about your business. Look at what the competition is doing—and do something entirely different." ~Daniel Burris and John David Mann, "The Reinvention Imperative"

Whether you know it or want it, you are in the process of reinventing yourself. The idea of reinvention--not just once but continuously, not just personally but organizationally--is now part of the 21st century conversation on success. It goes like this: we have to keep changing, keep learning and keep innovating in order to be relevant and in demand in the marketplace.

I mostly agree, and as a passionate advocate for more creativity in our lives and our culture, I enjoy taking part in this forward-thinking conversation from upstarts such as Change This, an online content source whose mission is to spread new ideas from original thinkers. Its latest issue includes "The Reinvention Imperative" by Daniel Burrus & John David Mann, which compellingly makes the case that it is now an imperative to reinvent ourselves and our businesses in order to keep up in a world where "change itself has changed":

There are two kinds of change: change from the outside in, and change from the inside out. The first happens to you. The second is an initiative that you take through conscious intention. Today there is an urgent need to anticipate and take the initiative to change from the inside out, even as all these transformations are coming at us from the outside in.

Burrus and Mann make the point that we have to be both proactive and extraordinary, and that real reinvention means getting closer to our own core, our own unique gifts. "The reinvention imperative," they write, "puts each of us on a quest to be the best me we can be."

Okay. But let's stop here. Talk of reinvention sounds cool. And it fits with my blog articles of these past few years and with my creativity competency principles:

>Fluency: Consider all of my possible options and identities

>Flexibility: Shift some things around and see things with fresh eyes

>Originality: Truly embrace my best and most unique self and offer it to the world.

But how do we really reinvent ourselves so that change is not just temporary and the results are real and different? If I'm going to reinvent myself--which I happen to be in the process of doing quite actively--what does that really mean? Does it mean that I have to change my insides first (umm, how?) or that I have to just embrace the core me and put it out in the world more effectively (oh, is that all)?

I think for personal reinvention to occur there must be a powerful mindset shift, a real change in belief about ourselves. Reinvention must be powered by a belief shift and real differences in behavior that comes from an alternative belief frame.

The truth is, this blogger needs a reinvention, a real one, not just one written on paper or that sounds good. It's time for me to put the creativity principles and rabble-rousing I've recorded here--yes, this blog holds more than 130 articles that can keep you busy reading and linking and viewing for days--to action.

Time to take the advice of the "Reinvention" authors: to stop trying to keep up and do the same as others and instead do something entirely different.

Time to embrace a belief and identity that forces reinvention and risk-taking.

Time to play by different rules.

Time for a new identity and new belief system that befits a warrior of aliveness.

Innovation in Networking

May 11th - Los Angeles, CA

The Global Leaders brings together today's leaders in business, government, education and philanthropy to network with purpose.  On Wednesday, May 11th, TGL brings their exclusive networking activities to the Los Angeles Athletic Club from 6p to 9p.

Mingle, relax and enjoy an evening of cocktails and hors d'oeuvres as you meet your peers and colleagues.  Special key note speaker for the evening will be Kim Shepherd, CEO of Decision Toolbox and nationally recognized Recruitment Executive, Ex-Broadcast Journalist, Author and Thought Leader.  Sponsors for this event include the Los Angeles Consulting Group (LACG).  Register now

Is Apple Good for Innovation?

Just about everyone on the planet agrees that Apple products are the soul of innovative design.  But are they good for innovators?  For me the answer is "not so much."

I have been using Apple laptops and iPhones for years.  As a software developer, I have a list of annoyances with Mac OS X starting with Apple's incomprehensible management of Java.  However, Mac OS X is far more productive than MS Windows, with its viruses, crummy OS releases, and bloatware.  iPhones are close to worthless as telephones in the area where I live in large part due to ATT's network.  But you can now switch to Verizon, so that's not such a problem either.

The real problem with Apple is that their products are closed.  Want to install a new file system?  Not here.  Want to pick a different motherboard to play around with power utilization?  Try somewhere else.  Want to know what the OS is really doing under the covers or (gasp) inspect the source code?  Dieu forfend!

Innovation in my chosen field of databases is increasingly based on breaking down the dividing lines between hardware and software to manage massive quantities of data economically and quickly.  The more I learn about hardware, the less I want fully integrated products.  I want devices I can interact with and learn from.  I want visibility into internals.  I want works-in-progress, not ready-made perfection.  In short, I want open platforms that give me the parts but do not tell me what to build with them.

A few weeks ago my iPhone dropped on the floor and shattered.  The replacement is a Droid 2 Global running Android.  The user interface is clumsy.  You have to watch out for viruses again.  But the hardware is lightning fast.  There is a free-for-all of people inventing new Android applications.  The source code for Android itself is available on code.google.com.  The open nature of Android is rapidly making it the locus of innovation for mobile devices.  I feel at home already.

Tuning Tungsten Parallel Replication Performance

Last month my colleague Giuseppe Maxia described how to operate Tungsten parallel replication. Since then we have been doing a good bit of benchmarking on both synthetic as well as real production loads. In this article I would like to follow up with some tips about how you can goose up parallel replication performance.  These apply to Tungsten Replicator 2.0.1, which you can find here. 

The first way to get good performance with Tungsten is to have the right workload. As explained in an earlier article on this blog, Tungsten parallel replication works by replicating independent databases (aka shards) in parallel.  Here is a picture that summarizes what is going on.

If you have a lot of schemas, if the updates are distributed evenly across schemas, and if you don't have many dependencies between schemas that require full serialization, parallel replication can speed things up significantly for I/O-bound workloads.  For example, Tungsten runs three times faster than MySQL native replication on large datasets when the slave is catching up to the master following mysqld restart. 

Catch-up is a famous slave lag case and one where Tungsten can be quite helpful.  (I think we will be faster in the future, but this is a good start.)  Nevertheless, there's a chance you'll need to do a bit of tuning to see such benefits yourself.

Tungsten currently uses a structure called a parallel queue to enable parallelization.  The parallel queue typically sits at the end of a replicator pipeline in front of the parallel apply threads, as shown in the following handy diagram.

One key to getting decent parallel replication performance is to watch the parallel queue in operation.  In Tungsten Replicator 2.0.1 we introduced a new status command trepctl status -name stores that goes a long way to help diagnose how well parallel replication is performing.   Here's a typical example using a 6 channel queue store.

$ trepctl status -name stores
Processing status command (stores)...
NAME                VALUE
----                -----
criticalPartition : -1
discardCount      : 0
eventCount        : 3217
maxSize           : 1000
name              : parallel-queue
queues            : 6
serializationCount: 1
serialized        : false
stopRequested     : false
store.queueSize.0 : 0
store.queueSize.1 : 480
store.queueSize.2 : 310
store.queueSize.3 : 1000
store.queueSize.4 : 739
store.queueSize.5 : 407
storeClass        : com.continuent.tungsten.enterprise.replicator.store.ParallelQueueStore
storeSize         : 2936
syncEnabled       : true
syncInterval      : 100
Finished status command (stores)...

The two most important things to look at are distribution of transactions across queues and serialization.  Let's start with transaction distribution.  In this particular example we were running a parallel queue with 6 channels but only 5 databases.  The distribution therefore looks pretty good.  One queue is empty but the other have a fairly even distribution of transactions.

Notice that one queue has exactly 1000 transactions.  In Tungsten Replicator 2.0.1 the parallel queue has a maximum size parameter (maxSize), which is set to 1000 for this example run.  Once an individual queue hits the maxSize limit, the entire parallel queue blocks.  It is not uncommon to see one queue blocking in this way if the replicator is catching up, which is exactly what is happening here.  In fact, if the queues are all empty it is possible Tungsten is somehow not supplying transactions to the queue fast enough.  That is not a problem here.

Bad workloads on the other hand tend to have a lot of transactions in one or two queues and few or none in all the rest. The following is an example of a possibly bad distribution.

$ trepctl status -name stores
Processing status command (stores)...
NAME                VALUE
----                -----
...
store.queueSize.0 : 0
store.queueSize.1 : 4
store.queueSize.2 : 3
store.queueSize.3 : 972
store.queueSize.4 : 0
store.queueSize.5 : 15
...
Finished status command (stores)...

If you see such skewed distributions persistently, you may want to try to adjust the queue partitioning using the shard.list file. The default parallel queue partitioning algorithm hashes shards into channels. This does not always gives optimal performance if your shards mostly happen to hash into the same channel.  The other possibility is that the workload is just badly distributed across databases.

You can decide whether the workload or partitioning is at fault using the trepctl status -name shards command.  Here's an example.

$ ./trepctl status -name shards
Processing status command (shards)...
NAME                VALUE
----                -----
appliedLastEventId: 000007:0000000000000384;20
appliedLastSeqno  : 1471201
appliedLatency    : 0.0
eventCount        : 6
shardId           : #UNKNOWN
stage             : d-pq-to-dbms
NAME                VALUE
----                -----
appliedLastEventId: 000005:0000000326365895;41
appliedLastSeqno  : 1470999
appliedLatency    : 0.0
eventCount        : 311605
shardId           : db1
stage             : d-pq-to-dbms
NAME                VALUE
----                -----
appliedLastEventId: 000005:0000000326512277;95
appliedLastSeqno  : 1471200
appliedLatency    : 0.0
eventCount        : 298522
shardId           : db2
stage             : d-pq-to-dbms
...

This shows that the distribution of transactions between the db1 and db2 databases is pretty even.  If you have many databases with roughly even values in the eventCount parameter, the workload is well suited for parallelization.  In that case you may want to assign shards explicitly in the shard.list file if you don't like the distribution in the parallel queue.

Meanwhile, the previous example shows an example of another potential problem.  We also see counts for #UNKNOWN, which is a special shard ID that means "I could not tell what schema this is." #UNKOWN transactions can occur if Tungsten cannot parse a SQL statement properly or there is a transaction that updates multiple schemas.  In either case, Tungsten serializes the parallel queue.

However it occurs, serialization is a performance killer because it means we have to block the parallel queue until all parallel transactions complete, execute one or more transactions serially, and then reopen the parallel queue.  You can see how often this is happening from the serializationCount value on trepctl status -name stores.  For many workloads a serializationCount value that is more than a few percent of the number in eventCount means the entire transaction stream is effectively serialized.

If the serialization is occurring due to #UNKNOWN shards, you may be able to improve things using a new value in the replicator service properties file that was added in version 2.0.1. It controls whether we assign the shard ID using the default schema even if Tungsten cannot tell from the SQL command what you are doing.

# Policy for shard assignment based on default database.  If 'stringent', use
# default database only if SQL is recognized.  For 'relaxed' always use the
# default database if it is available.
replicator.shard.default.db=relaxed

Setting the parameter to relaxed can help quite a bit if the problem is due to unusual SQL that confuses the parser.  On one workload we were able to reduce the serializationCount from about 10% of transactions to 0% in this way.  We then saw the expected speed-up from parallel replication.

Cloud & Outsourcing Service Providers

Far too often, am asked the question – Won’t cloud disrupt the outsourcing vendors in a big way. My answer is Yes & No. Yes. The cloud will definitely impact those outsourcing vendors who just wait and watch or do superficial adjustments to the emergence of cloud. For those outsourcing vendors (just like in the case of all the players in the software ecosystem), trying to understand the impact of the cloud and proactively embracing the cloud, the impact is going to positively felt. Let’s examines this further. It’s clear: The cloud’s impact on outsourcing over the next five years will be profound because “it will significantly influence the demand made on the nature of services and the type of expertise built by service players as seen in the industry today. On an As-Is scenario, one of the key changes would be that the demand on labor will go down – as cloud could cut down need for support processes while enabling near real-time data processing across the value chain for business”. At the same time, the disruptive cloud technology can enable whole host of new business opportunities centered on new business and technology and support execution models hitherto just dreamt of.

Zoom forward: The beauty of the cloud technologies is such that overtime, it would be seen that cloud technologies will enable IT service providers to deliver end-to-end services regardless of the various platforms, applications, and technologies involved. Extend the thought – one can easily see that within enterprises the classification of core/non-core would begin to diminish overtime as cloud enables Enterprise IT to act as a hybrid environment of on-premise, private cloud, and public cloud services.
The cloud-based services will change outsourcing contract methodologies. “Buyers will move away from long-term contracts where the return on investment depended on continuous improvement, and move to shorter-term contracts with more flexibility to quickly buy new services. In respect of some business functions, the complexities involved in structuring a deal centered around cloud would encompass requirements like providers promising to take over customer infrastructure and run it from their shared centers to minimize multiple cloud vendor management for customers”. Cloud-based services will also cause an evolution and huge change in the way outsourcing providers price their services.

There are various types of cloud-related services that outsourcers will typically provide:
• Consulting around integrating enterprise IT with private and public clouds to create a hybrid environment
• Organization change management & risk management
• Implementing and managing private clouds to consolidate and optimize infrastructure
• Developing custom applications for the emerging cloud software platforms
• Developing new applications that integrate collaboration, communication, and cloud platforms
• Migrating enterprise applications to the cloud and the related testing, certification, and governance for risk and compliance
• Offering many commonly used functions as –a –service
• Governance mechanisms, regulation compliance
• Making Business –IT alignment realized over time.

In its fully evolved state, service providers will take on the role of a trusted partner to integrate cloud services of multiple service providers with Enterprise IT. Outsourcing providers are going to move up the value chain, offering consulting and information management services — not in the actual delivery of IT but in how buyers should provision and organize their systems and business process workflows.

Services providers that focus on providing services at the lower levels of the infrastructure and platform stack will need to become more agile and nimble to new technologies and faster technology cycles. They will have to provision new services as quickly as their cloud computing alternatives. The ease with which users can access social technologies, mobile devices and SaaS technologies will mean that IT and outsourcing partners will be bypassed, leaving a myriad of contract support challenges. Inevitably, outsourcers will have to support new technologies and do so in a far more rapid manner. This will lead to tactical responses by outsourcing providers for example leveraging service catalogs to provide a range of computing alternatives and service options to business users (including dedicated, shared and cloud services at different prices and service alternatives). But the real change is one of drastic business model change from hierarchical command structures to a modularized, configurable set of services that can be provisioned to clients in a rapid fashion.

Cloud Computing challenges services providers to sell beyond IT - . Service providers that own the IT budget do not have the relationships to effectively sell many of the cloud enabled business services that are emerging, as they require IT, business and executive relationships – and of course process knowledge. “Upwardly mobile” services providers capable of selling at the process layer will become aggregators of on-premise and Cloud IT technologies within hybrid environments.

If a service provider’s revenue is dependent on increased resource consumption as defined by resource units (e.g., managing more servers) any move to the Cloud most likely will result in a net reduction in resource consumption as infrastructure is consolidated, automated and virtualized.
Overall, I think Cloud Computing is about to bring a lot of changes to the traditional Outsourcing world with a few challenges such as security which is slated to have crossed the tipping point and is poised to put customers at greater risk at the cost of low-cost cloud alternatives that today present themselves as the new-world outsourcing parties.
Nevertheless it will accelerate offshoring / outsourcing, in my opinion because:
- It will drive the traditional outsourcing parties to adopt faster nimble methods

- Contract cycles and agreements will have to be revisited as niche core cloud service players will go after the buyers as contract renewal approaches.

- Fixed Contracts will come under fire as demand of variable contracts will increase. Traditional vendors will have a problem as their models are based of one-time fixed with incremental charges while with hardware/software costs dropping dramatically customers will end up paying more

Cloud computing is surprisingly incremental – software provides a potential risk, hardware provides a potential risk but platform as a service offerings and ability to tap huge new markets tilt the scale. We believe cloud models, like SaaS, will take more years to go mainstream in a true sense of the term, ie, relative to the size of the overall software market. This implies that while SaaS has already displaced traditional software in some categories (eg, CRM), a widespread shift will only be gradual. This provides a window of opportunity for IT-services vendors. Newer offerings like platform BPO adds to the service provider pie. Platform BPO can be described in many ways. For example, it can be considered a SaaS model implemented across a business process, or something akin to “process as a service”.

In effect, platform BPO is about:
- Automating a business process to ensure lower dependence on manpower resources;
- Hosting that process on shared infrastructure (of the vendor or rented by the vendor) rather than on the customer’s premises;
- Pricing the offering on a transaction-based model rather than on a cost centric - people deployed model;
- Sharing of the core platform across many customers (multi-tenant) rather than proprietary to a single customer. Platform BPO is, thus, a new way to deliver processes that have already been around, and it is much influenced by the emerging cloud-computing concept.

Three years from now I would say a good number of enterprises will have their strategies in place for how they plan to use the cloud. The smaller the enterprise the greater the potential exploitation of a third-party cloud will happen. The larger customers will likely use cloud-like technologies, but internally private clouds. Our customers are unlikely to use any public platforms except for what they would deem as commodity activities, such as payroll, F&A, HR, etc (not core strategies). How does that impact companies like us? Large enterprise market, our predominant market space, is likely to adopt cloud technologies internally, and try to charge back their internal clients/departments on a cloud-like billing pattern. Therefore, they would like to reduce the complexity of what they have to take advantage of the variable costs from a costing perspective. In a three-year timeframe that will be a substantial transformation for a lot of companies, which translates into opportunities for people like us.

On a three to five-year perspective, most of the enterprises will be transitioning to a cloud-based delivery strategy for tech and services. Not likely too big, as big a transformation as ERP, CRM, etc as it will be technology-driven transformation on the inside. Business benefits that the end user will see will be greater efficiency, from infrastructure, hardware perspective. There will be a better sharing of resources. What we are seeing in a lot of pilots is speed. Change and speed will be much better in a cloud environment.

And that provides an opportunity for outsourcing service providers when it comes to cloud integration, cloud enablement of existing applications or creating new applications all together on the cloud.

As long as the outsourcing partners’ providers continue to be innovative, and adapt themselves to the market conditions, there should be no room to get worried. After all, the outsourcing service providers do carry with us the knowledge of the applications, ecosystem and architecture for the customers we operate with. Value gets created only by assisting the end client in their business process. In its true sense, value is not just created by cloud infrastructure providers, as they are only going to commoditize the data centre service.

So, overall, well run offshore headquartered firms shall see net positive opportunities with large enterprises embracing the cloud.

Networking with Purpose" Made Easy Thanks to New Partnership Between The Global Leaders and JUJAMA

This week The Global Leaders Holdings LLC (TGL) announced a partnership with rising star JUJAMA, Inc. to power TGL InPerson^TM, a new interactive networking and meeting-planning tool.

The Global Leaders (TGL), a networking platform for senior level executives in business, education, government and philanthropy, has introduced yet another tool designed to help their members “Network with Purpose”...TGL InPerson^TM.

TGL InPerson^TM is a result of a strategic partnership with JUJAMA, Inc., a rising star in interactive technology solutions. TGL InPerson^TM allows participants to interact and network before, during and after any of The Global Leaders networking events or conferences.

“We are very excited about our partnership with JUJAMA. They are a young, vibrant company who is on the cutting edge”, stated TGL Co-Founder and CEO, Jim Gitney. “As a result of this new tool powered by JUJAMA, members attending our events will be able to...

• Post their bio, company information, videos, links and important information that they would like to share with other registrants
• Request and schedule meetings with event participants that are then recorded in a personal calendar that tracks their schedules for not only the day of the event, but the day(s) prior and after
• E-mail communication to fellow attendees
• View the list of attendees and create alerts to let them know when someone they want to meet has registered
• Experience an open environment for discussions to take place

This is an amazing step forward in providing our members a vehicle to not only get the most out of the dollars they spend on conferences and networking events, but to ensure they are optimizing their time while at the conference and interacting with the right people.”

“Our professionally focused networking platform fits nicely with The Global Leaders model of bringing professionals together to connect and stay connected”, added JuJaMa Co-Founder and CEO, Nadia Dailey. “We are very excited to be a part of their events, which draws the best in class thought leaders from around the world”.

TGL InPerson^TM is fully operational and making its debut for a special networking event in San Francisco on April 20th that will connect the brightest MBA students from Berkeley, Stanford, Hult and other universities with senior level executives from the top companies in the greater San Francisco area. To read more about this event and see a complete listing of other upcoming TGL InPerson^TM events go to http://www.jujama.com/tgl-inperson/events.aspx. To register for a free membership with The Global Leaders visit http://www.tgleaders.com.

For more information about The Global Leaders contact Jayme Porkolab at jayme.porkolab@tgleaders.com

JuJaMa provides professionals an interactive community to network before, during and after a seminar, conference, trade show or gathering—it is the glue that holds together local, regional, national or global societies, industries or initiatives. This intuitive environment makes event planning, scheduling, networking and information exchange more efficient. Go to http://www.jujama.com for more information.

The Current Theater of American Prejudice

Creativity flowers from the foliage of diversity, whether a diversity of ideas, perspectives, cultures, or, alas, religions. Here is video-food-for-thought from our current theater of American prejudice, starring Stephen Colbert (semi-actual person) and Reza Aslan (an actual Muslim). If you don't see the video below, click here.

The Colbert Report	Mon - Thurs 11:30pm / 10:30c
Reza Aslan
http://www.colbertnation.com/
Colbert Report Full Episodes Political Humor & Satire Blog Video Archive

If you'd like a real opportunity to improve interfaith relations here in Chicago, join me to help with the non-profit Poetry Pals, a kids program bringing Muslim, Catholic, Jewish and other diverse students and their communities together to share in creativity. Check out this video and facebook page for more.

Unconventional Author to Speak at Los Angeles Event

Kim Shepherd, author of "The Bite Me School of Management: Taking a Bite Out of Conventional Thinking" and CEO of Decision Toolbox, has been added as the keynote speaker for the May 11th networking event at the Los Angeles Athletic Club from 6p to 9p.

Kim is a nationally recognized Recruitment Executive, Ex-Broadcast Journalist, Author, Thought Leader and National Speaker.

Attention to People is More Important Than Ever.  In recent years, attracting and retaining talent has emerged as one of the top three challenges facing senior executives annually --- alongside competition and the health of the global economy.  And because recruiting is the business of people, the players, playing field and rules of the game change constantly.

Drawing on her background as a broadcast journalist, Kim Shepherd invites her audience out onto the recruitment playing field to see what trends are directly impacting talent today and how the smartest companies are out-recruiting the competition.

With real-world examples, proven processes and in-the-trenches recruitment advice, Kim inspires and equips her audience to compete in the War for Talent.  REGISTER TODAY and join fellow members of The Global Leaders to 'network with purpose' and meet Kim Shepherd.

Understanding Tungsten Replication Services

If you follow Giuseppe Maxia's Datacharmer blog you have seen several recent articles on Tungsten Replicator.  Giuseppe and I work closely together on replication at Continuent, and I have promised a matching set of articles about replication internals that match the practical introduction provided by Giuseppe.  In this first article I will describe replication services, which are message processing flows that run in the Tungsten Replicator.

Unlike many replication engines, Tungsten Replicator can run multiple replication services concurrently.  There is a central management interface that allows you to start new replication services without disturbing services that are already running.  Each replication service also has its own management interface so that you can put the loaded service online, offline, etc. without disturbing other replication work.  As Tungsten is written in Java, the management interfaces are based on JMX, a standard administrative interface for Java applications.

Here is a simple diagram that shows a Tungsten Replicator with two replication services named fra and nyc that replicate from separate DBMS masters in Frankfurt and NYC into a single slave in San Francisco.   You can immediately see the power of replication services--a single Tungsten Replicator process can simultaneously replicate between several locations.  Replication services are an important building block for the type of complex setups that Giuseppe Maxia discussed in his blog article on Replication Topologies. 

Users who are handy with Java can write their own programs to manipulate the JMX interfaces directly.  If not, there is the trepctl utility, which is supplied with the Tungsten Replicator and works off the command line. 

If the Tungsten Replicator architecture reminds you of a Java application server, you are absolutely right.  Java VMs have a relatively large resource footprint compared to ordinary C programs, so it is typically more efficient to put multiple applications in a single VM rather than running a lot of individual Java processes.  Tungsten replication services follow the same design pattern, except that instead of serving web pages they replicate database transactions.  

Let's now look a little more deeply at how Tungsten Replicator organizes replication services.  Each replication service runs a single pipeline, which is Tungsten parlance for a configurable message flow.  (For more on pipelines, read here.)  When the service starts, it loads an instance of a Java class called OpenReplicatorManager that handles the state machine for replication (online, offline, etc.) and provides the management interfaces for the services.  The OpenRepicatorManager instance in turn depends on a number of external resources from the file system and DBMS. 

Here is another diagram showing how Tungsten Replicator organizes all of the various parts for services.  Services need a configuration file for the pipeline, as well as various bits of disk space to store transaction logs and replication metadata.  The big challenge is to ensure things do not accidentally collide.

This layout seems a bit complex at first but is reasonably simple once you get used to it.  Let's start with service configuration using our fra service as an example. 

Service configuration files are stored in the tungsten-replicator/conf directory.   There are up to two files for each service.  The static-fra.properties file defines all properties of the service, pipeline organization and properties like the replication role or master address that may change during operation.  The dynamic-fra.properties contains overrides to selected properties.  For instance, if you switch the replication role from slave to master as part of a failover operation, it goes in the dynamic-fra.properties file.  Tungsten Replicator reads the static file first, then applies the overrides when it starts the service.

Next, we have Tungsten transaction logs, also known as the Transaction History Log.  This is a list of all transactions to be replicated along with metadata like global transaction IDs and shard IDs.  THL files for each service are normally stored in the logs directory at the same level as the tungsten release directory itself.  There is a separate directory for each service, as for example logs/fra. 

Next we have Tungsten relay logs.  These are downloaded binlogs from a MySQL master DBMS from which the replication service creates the Tungsten transaction logs.  Not every replication service uses these.  They are required when the MySQL master is on another host, or the binlogs are on an NFS-mounted file system, which Tungsten does not parse very efficiently yet.  Tungsten relay logs use the same pattern as the THL--everything is stored under relay-logs with a separate subdirectory for each service, for example relay-logs/fra. 

Finally, there is metadata in the DBMS itself.  Each replication service has a database that it uses to store restart points for replication (table trep_commit_seqno) as well as heartbeats and consistency checks (tables heartbeat and consistency, respectively).   The name of this database is tungsten_servicename as in tungsten_fra. 

Setting up services is difficult to do manually, so Tungsten Replicator 2.0 has a program named configure-service that defines new replication services and removes old ones by deleting all traces including the database catalogs. You can find out all about installation and starting services by looking the Tungsten Replicator 2.0 Installation and Configuration Guide, which is located here. 

Services have been part of Tungsten Replicator for a while but we have only recently begun to talk about them more widely as part of the release of Tungsten Replicator 2.0.0 in February 2011, especially as we are start to do more work with multi-master topologies.  One of the comments we get is that replication services make Tungsten seem complicated and therefore harder to use, especially compared with MySQL replication, which is relatively easy to set up.  That's a fair criticism.  Tungsten Replicator is really a very configurable toolkit for replication and does far more than MySQL replication or just about any other open source replicator for that matter.   Like most toolkits, the trade-off for power is complexity.

We are therefore working on automating as much of the configuration as possible, so that you can set up even relatively complex topologies with just a couple of commands.  You'll see more of this as we make additional replicator releases (version 2.0.1 will be out shortly) and push features fully into open source.   Meanwhile, if you have comments on Tungsten 2.0.0 please feel free to post them back to us.

Social Media in a Grown Up World Means.....Business

The following is an advance copy of a press release for TGL scheduled for this Monday, March 7th, announcing TGL Exchange^TM.

While other businesses have been focused on creating fans on Facebook and followers on Twitter, The Global Leaders has been laser-focused on creating a networking platform that puts business development in high gear for their members.

Several months ago The Global Leaders Holdings LLC quietly launched www.tgleaders.com, a networking platform that brings together senior level executives, heads of government, top educators and philanthropists.  Their members have led over 10,000 companies in more than 150 countries.  “Our member base is unique in the fact that while they are very tech savvy, the majority do not belong to other networking sites.  They tend to ‘network with purpose’ as we like to call it”, stated co-founder, George Bickerstaff.  “By that we mean that they get on the information highway with a specific purpose in mind; whether it is to search for an investment, market their business opportunity, find talent, research best practices or ask experts in their field of interest for advice.  Our network provides them a pool of resources that includes their global peers allowing them to find what they need quickly and efficiently.”

Fellow co-founder, Jim Gitney, added “Rather than going out and making a big splash about our network to recruit new members, we have been more intent on listening to our current members to understand what tools and resources they would like to see on www.tgleaders.com.  As a result, we are constantly enhancing our site.”

Among the major enhancements revealed on www.tgleaders.com today was TGL Exchange^TM; an exclusive feature on The Global Leaders’ proprietary platform that facilitates business development for their members.  TGL Exchange^TM brings together both sides of the equation for a successful business endeavor:  Opportunity (TGL Deals) and Money (TGL Investment Funds).

TGL Exchange^TM has categorized the current opportunities and funds into 5 main areas that include Real Estate, Business, Philanthropy, Green Technology and Healthcare; allowing potential investors and businesses to easily find each other.

Businesses can post their current opportunities; i.e. selling a business, seeking a joint venture partner, licensing a new product line or finding people to collaborate with on a specific project through TGL Deals.

While investors can post the specific types of opportunities, sectors, countries, etc. that they have available capital for in the relevant category of TGL Investment Funds.

Members have access to all postings via their homepage on www.tgleaders.com; however, posters who want to insure that their offers get noticed have the option of posting them in specific groups within The Global Leaders website.  Posting in targeted groups ensures that their message is communicated to and seen by the audience they want to reach. 

“Our goal is to continue to find ways to make it easy for our members to ‘network with purpose’ and TGL Exchange^TM is another tool that will allow them to do so”, stated Jim Gitney. 

When asked about TGL’s plan to expand their membership base, Jim responded “Right now we are relying on word of mouth….and its working.  Our members like what they are experiencing and they’re inviting their peers and colleagues to join.  We’re adding new members every day and growing at a steady, rapid pace.  So, by staying focused on our members’ needs, they’re helping us spread the word and recruit new members”.

So Your Business Failed. Wonder Why?

Every year, millions of new businesses declare bankruptcy; in fact, the majority of new businesses fail within five years.

For a business to thrive, it should have the following four components - a vision, a strategic plan, an operating plan, and a financing plan.

This article summarizes the common pitfalls awaiting the new business owner and explains the key actions needed to avoid these problems and build a sustainable business.

Check out "What Happened to My Business" at

http://www.tgleaders.com/what-happened-my

The Global Leaders launches Social Media for Business

We launched our network group for Social Media yesterday.

Over 500 key executives joined the group within the first 24 hours.  A proof statement that senior leaders are warming up to and understanding the power of social media for business.

Our website for the Social Media group is:  http://www.tgleaders.com/group/global-leaders-social-media

Our related LinkedIn group is:  http://www.linkedin.com/groups?about=&gid=3813086

Join the group and learn the lingo and tools.

A sampling of Mary Oliver

Time for a quick dip into the poetic vault of Mary Oliver, the great American poet, starting with one of my favorite poems and then a few excerpts to spur your creative imagination. All of these can be found in her New and Selected Poems. Happy anniversary, Mom and Dad!

The Journey
One day you finally knew

what you had to do, and began,

though the voices around you

kept shouting

their bad advice--

though the whole house

began to tremble

and you felt the old tug

at your ankles.

"Mend my life!"

each voice cried.

But you didn't stop.

You knew what you had to do,

though the wind pried

with its stiff fingers

at the very foundations,

though their melancholy

was terrible.

It was already late

enough, and a wild night,

and the road full of fallen

branches and stones.

But little by little,

as you left their voices behind,

the stars began to burn

through the sheets of clouds,

and there was a new voice

which you slowly

recognized as your own,

that kept you company

as you strode deeper and deeper

into the world,

determined to do

the only thing you could do--

determined to save

the only life you could save.

from Rice

I don't want you just to sit down at the table.

I don't want you just to eat, and be content.

I want you to walk out into the fields

where the water is shining, and the rice has risen.

I want you to stand there, far from the white tablecloth.

I want you to fill your hands with the mud, like a blessing.

from When Death Comes

When it's over, I want to say: all my life

I was a bride married to amazement.

I was the bridegroom, taking the world into my arms.

When it's over, I don't want to wonder

if I have made of my life something particular, and real.

I don't want to find myself sighing and frightened,

or full of argument.

I don't want to end up simply having visited this world.

from In Blackwater Woods

To live in this world

you must be able to

to do three things:

to love what is mortal;

to hold it

against your bones knowing

your own life depends on it;

and, when the time comes to let it go,

to let it go.

from Wild Geese

You do not have to be good.

You do not have to walk on your knees

for a hundred miles through the desert, repenting.

You only have to let the soft animal of your body

love what it loves.

The Good, The Bad & The Ugly

The following article was written with inputs from members of The Global Leaders.....

Experiences executives have with management consultants

Dr. Sarah Layton CMC, FIMC

In preparation for the keynote at the 2010 Confab meeting of professional management consultants, we asked about a thousand global business leaders to respond to an online survey about their use of and experience with management consultants. When over a hundred executives responded within a few hours, I knew I had hit a nerve.

Eighty three respondents out of 107 replied that they hire management consultants.  They represent a broad range of industries from healthcare/pharma and financial services to manufacturing, retail and agriculture. Here are some of their comments, minus specific identifiers. They range in size from start-ups to about USD $120 Billion

Ninety three percent said they were very satisfied to moderately satisfied with their experience with a management consultant. Here are a few of their comments:

The good ....there are tales of how management consultants have saved the company with their recommendations.

Worked with (major consulting firm) consultants on an SG&A rationalization project that was well thought out and very valuable to the organization.

Great experience with a former industry leader that helped the company develop key evaluation hurdles for new projects and prepare a commercialization plan

The bad ...there are tales of how the consultant just rephrased into consultant speak what the client told them, adding no value.

Our CEO hired a consultant to evaluate the sales team. He found him via the internet. The evaluation tests he used were not relevant to our distribution model or industry. He didn't understand our business and he failed miserably. Waste of money.

All talk and theory - no action and practical application

The ugly ... Consultant discovered a major weakness in our CEO's leadership skills and tried to force him out by exposing it to key leaders, very unprofessional approach...should have worked through it at an executive/owner level.

So what would executives like management consultants to know? When given a chance to tell management consultants a thing or two, here is just a sampling what they said:

1. Work with the client. He probably knows what he wants but may not have sufficient resources such as time or people, to carry it out as quickly or to the standards required. Building trust that the consultant has a deep understanding of the clients needs, both spoken and unspoken, and then consistently acting to achieve those goals will always have satisfactory results

2. Sarah, take a lesson from small business. They have to produce results or they don't eat. The days of charging outrageous sums for less than stellar results are over. Success comes from deep commitment and  respect on both sides.

3. Spend the extra time up front understanding the problem and don't make the client spend too much time and effort educating you.

4. a. Study the potential client prior to meeting with him/her
   b. Do not assume analytical illiteracy on the part of the client
   c. Do not assume the client will tell you what she wants to hear
   d. Fancy slides do not compensate for mediocre work

Dr. Sarah Layton speaks, consults and writes on value innovation and strategy and can be reached at info@corporatestrategy.com. She is a qualified Blue Ocean Strategy practitioner and managing partner of Corporate Strategy Institute based in Orlando, Fl. She presented the Keynote address at the 2010 Confab in Reno and has travelled Asia, North, Central and South America, and India working with clients to find new markets that have no competition.  (www.corporatestrategy.com) (www.blueoceanstrategicplanning.blogspot.com

For more informative articles on business, check out the "What Happened to My.....?" series of articles from TGL members and staff at www.tgleaders.com/what-happened-my.