Happy Halloween: Is it scary to be an American?

There is nothing quite like waking up in one city, flying hundreds of miles, and then walking the streets of another later that day. The wild Chicago winds delayed but did not cancel my early Boston-bound flight so I found myself at Faneuil Hall Tuesday afternoon, surrounded by stunning fall foliage, curving streets and history everywhere around me. This new reality stimulated new insights, old college memories and hopeful thoughts for me, confirming that travel is indeed a creativity lubricant. During my few-day trip I facilitated a teambuilding program with an enthusiastic company, visited with old friends, and was a guest on a cable TV program called "Wake up your Magic." I buzzed with ideas and schemes.

But by doing a little reading, eavesdropping and now TV-watching back in Chicago, I am once again reminded of another pervasive reality--which is not optimistic, not conducive to new ideas. Just turn on your TV now in the week before an election and here is the voice of that reality: "Happy Halloween: It's scary to be an American." Here's one political ad example:

Now, creative thinkers are particularly adept at shifting among different realities, enabling them to see from different perspectives, flexibly consider possibilities, and shift their mindset to come up with new solutions. But the current "reality" that for too long has infected the American discourse and media--that these are scary times, and that extreme and often irrational views trump rational ones--blocks our creativity and squelches innovation and change. That's because fear and creativity are not compatible. The more afraid you are, the less ability you have to open to divergent ideas, consider new options and access your heart--all creative keystones.

We (meaning our predominant cultural conversation) appear to be stuck so deep--and constantly reinforced by media--in this particular "reality" of pessimism and fear that we can't honor other valid realities, like my time in Boston, which was great. It can be as scary to read rational commentary (I'll discuss the Time Magazine cover story on restoring the American Dream next time) as it is to watch Democrats and Republicans hurl fear-bombs in the current political campaigns on everyone's television. I wanted some relief so I read though my old college newspaper when I was on campus and found the same partisan and fear-inducing views parroted there. So it's everywhere. Right? America is in decline and we all should be afraid, right? Preachers are burning Korans and Obamacare is taking over our freedom.

Are we insane to ignore these threats or are we insane to listen to them?

This is the question that Jon Stewart and Steven Colbert (click on banners for more info) are addressing in this weekend's rally in D.C. Usually I'm on the side of crazy, as creativity is not possible without coming up with way-out-there ideas that some will call crazy. I am an advocate of letting our freak flags fly, engaging our individual and original passions as a birthright to being human. But right now we need more sanity. We need to stop reinforcing extreme views and irrational arguments--and subscribing to the "reality" that creative solutions are impossible in America right now. Stewart and Colbert will try their own creative take on shifting this reality, and I'm curious to see how they will attempt to pull it off.

I want to encourage you to take this Halloween opportunity to explore a different reality, to try on another identity for a little while --perhaps someone less crazy than our current politics is--to see where it leads.

Parallel Replication on MySQL: Report from the Trenches

Single-threaded apply is one of the big downsides of MySQL's built-in replication, as Baron Schwartz pointed out a couple of days ago.  While a master can process dozens of updates at once, slaves must apply them one after the other on a single thread.  Add in disk I/O, and the result is very slow performance indeed.  The obvious answer is parallel apply, namely writing multiple non-conflicting updates to the slave at once.

I have spent the last few months implementing parallel apply for Tungsten 2.0, which we are now testing at customer sites.  In this article I would like to describe how Tungsten's parallel apply works as well as some of the lessons that have become apparent through the implementation.

There are a couple of big challenges in parallel apply.  There is of course the practical problem of separating transactions into parallel streams, for example splitting them by database.  This is known as sharding.   Row updates are easy enough but MySQL also has statement replication.  Transactions with statements require parsing, and there are ambiguous cases.  If that's not enough, features like LOAD DATA INFILE have a complex implementation in the binlog and require specialized logic to shard correctly.  In addition, parallel apply of any kind has a lot of corner cases that you have to solve completely or risk unpredictable failures.  Here's an example:  skipping transactions on the slave.  You have to wait for the event, but what if some of the threads are already past it when you ask to skip?  How do you synchronize access to the list of transactions to skip without creating a choke point for threads?  

The next challenge is performance.  Parallel replication offers a rich choice of ways to lower throughput, not raise it.  Multiple disk logs are the best I have found so far, as they can convert sequential reads and writes on the disk log to random I/O when more replication threads contend for different parts of the disk.  Implementing multiple queues in memory is far faster and simpler but limits the queue sizes.  Another excellent way to slow things down is to try to parallelize SQL transactions with a lot of dependencies, which means you end up effectively serialized *and* paying the extra cost of parsing transactions and synchronizing threads.  In this case it can be better to keep everything sequential but use block commit to apply 50 or 100 transactions simultaneously on the slave.

With all that said, the parallel apply problem is still quite tractable, but you need to pick your battles carefully.  Tungsten's parallel apply implementation has a very clear problem focus:  speeding up slave updates for multi-tenant applications that have a high degree of natural partitioning and concurrent updates across customers.  This is not as limiting as it might sound to readers unfamiliar with MySQL.  SaaS applications for the most part follow the multi-tenant model on MySQL, with each customer assigned to a particular database.  So do large ISPs or cloud providers that host customers on shared servers using separate databases.

Tungsten parallel apply is based on automatic sharding of transactions.   The following diagram shows the parallel apply algorithm conceptually. 

Tungsten Parallel Apply

Tungsten has a flexible architecture based on replication pipelines, described in a previous article on this blog.  To recap the model, pipelines are divided into stages, which represent processing steps.  Each stage consists of an extract-filter-apply loop with symmetric interfaces and identical processing logic for each stage.  The parallel apply implementation builds on replication pipelines as follows:

1. A new filter called EventMetadataFilter automatically parses incoming transactions to figure out which database(s) they affect.  This is simple for row updates but involves parsing for statements and specialized extract handling for odd-ball operations like LOAD DATA INFILE. 
2. The shard ID is assigned from the database name. This is glommed into the EventMetadataFilter but will shortly be broken out into a separate filter so that it is possible to support alternate shard assignment algorithms. 
3. There is a new kind of in-memory buffer between stages called a ParallelQueue that supports multiple queues that feed the final apply stage.   Stages have a corresponding extension to allow them to have multiple threads, which must match the number of parallel queues or you get an error. 
4. The ParallelQueue implementation calls a new component called a Partitioner to assign transactions a partition number (i.e., a parallel queue).  You can substitute different algorithms by providing different partitioner implementations.  The default implementation uses a configuration file called shard.list to map shards to queues.  Unless you say otherwise it hashes on the shard ID to make this assignment.

Extensions #1 and #2 run on the master, while #3 and #4 run on the slave.  I really like diagrams, so here is a picture of the fully implemented parallel apply architecture.  The master replicator extracts, assigns the shard, and logs each transaction.  The slave replicator fetches transactions, logs them locally, then applies in parallel.

Full Master/Slave Architecture for Parallel Apply

So how does this work?  Pretty well actually.  Local lab tests indicate that parellel apply roughly doubles throughput on a multi-database TPC-B benchmark we use for testing.   We should be able to publish some real-world performance numbers in the near future, but so far things look quite promising.  During the implementation a number of interesting issues have arisen, which I would like to discuss now.

The first issue is the ratio between parallel apply threads and shards.  While it might seem obvious to have a thread per shard, in real deployments the situation is not so clear.  For one thing actual deployments in SaaS and ISP situations often have hundreds or even thousands of databases, which has a number of practical consequences for implementation.  Less obviously, spreading transactions thinly across a lot of queues means fewer opportunities to use block commit, hence more work for slave servers and less overall throughput.  Performance optimization is a very uncertain matter, so Tungsten lets users configure the ratio.

Dependencies between shards are yet another issue.  While I mentioned that Tungsten is designed for applications with "a high degree of natural partitioning," dependencies between databases as well as individual transactions do occur and cannot be ignored.  For example, many SaaS applications have reference data that are used by all customer databases.  Even if parallel SQL works here, applications may get sick from seeing updates appear in the wrong order.  Or you could have global operations like CREATE USER that affect all databases.  Or you might not be able to tell which shard a piece of SQL belongs to.  Tungsten allows users to declare reference databases and automatically serializes these databases as well as global or "don't know" cases. 

There are also numerous issues around startup and shutdown.  Remember how MySQL replication slaves will not restart after unclean shutdown with open temp tables?  (If not, take a quick break and read this now.  You'll thank me later.)  Parallel apply introduces similar issues, because you have multiple threads all updating different positions in the database.  Tungsten handles crash recovery by tracking the apply position of each queue in InnoDB and then recommencing from that point on restart in each queue.  I am putting finishing touches on clean shutdown, which ensures that all queues are empty, much like automatically checking that temp tables are closed on MySQL.  


In short, over the last few months Tungsten has climbed a fair distance up a pretty big hill to get parallel apply to work.  The flexibility of the replicator architecture, particularly pipelines, has been very helpful as it is quite easy to extend.  The parallelization algorithm builds on terrific work by other colleagues at Continuent, especially Stephane Giron and Linas Virbalas.  They have both put enormous effort into building up MySQL and PostgreSQL replication capabilities.

Here are a couple of parting thoughts about parallelization based on the experience so far.

Thought number one:  parallel replication is not magic.  To use parallel apply effectively, applications need to play nice:  mostly short transactions and not too many dependencies between shards are the biggest requirements to see a substantial boost in throughput.  For example, if you let one user write 50M statements to the binlog in a single transaction, things are going to get kind of quiet on the slave no matter what you do.  Also, you can forget about MyISAM or other non-transactional engines.  As I have written before, these engines offer a number of opportunities for databases to get messed up or out-of-sync even using conventional MySQL replication.  Tungsten's block commit and parallel apply increase the window for problems significantly.  If you are still using MyISAM for replicated data, it's time to man up and convert to InnoDB. 

Thought number two: The long-term answer to effective parallel replication is to change how MySQL works by interleaving transactions within the binlog along the lines suggested by Kristian Nielsen and others.  MySQL currently completely serializes transactions to the binlog, an accomplishment that makes slave apply logic a lot simpler.   Tungsten parallel apply then has to undo this good work and recreate streams of non-conflicting updates, which is complex and does not help all workloads.

It is doubtful that replicating interleaved transactions will be less complex than handling a serial binlog as it stands today.  There is also some heavy lifting inside MySQL to get to an interleaved binlog.  However, interleaved transactions would have the advantage that transactions for any workload would be parallelized, which would widen the scope of benefits to users.  I'm happy to see that Kristian and other people are now working this feature for future releases of MySQL.

Meanwhile, we have a workable solution for Tungsten and are pushing it forward as quickly as we can.  Contact Continuent if you would like to test it out.

Cloud : Vision & Tactful Deployment

I was moderating an informal discussion amongst decision makers on how much resources to allocate to cloud computing efforts inside their respective enterprises. Someone wanted to know if there is a benchmark available as to how much individual enterprises spend on cloud efforts to stay ahead of the curve. I murmured that this may not be so relevant as a factor for taking decisions inside their respective enterprise but at best can act as an aide – either to reinforce that we are doing what everyone else is doing – little or more as the case may be, but can’t become a guiding mantra. Remember best practice vs next practice dilemma. Cloud actually provides an opportunity to create transformational change inside IT and enterprise. I told myself that the key remains evolving a vision for cloud inside every enterprise.

Cloud is a truly disruptive use of technology and many of the traditional principles such as vision, strategy and trusted execution partners are highly relevant in assessing and managing cloud adoption progress inside enterprises.What I see all around is the fact that, many enterprises are not developing a full vision for their cloud computing efforts. I see that in a number of cases, cloud efforts are pilot efforts, departmental efforts, proof of concepts etc. Seldom does one get to see a full blown real time cloud efforts inside enterprises. Why these half steps – while testing the water is per se not bad, winners need to learn to swim –fast and in an easy manner. Enter cloud computing vision – connected dots, big picture ….

Since it is the early days of the cloud, it is understandable that people take baby steps on an experimental basis as a start point , but the rate of adoption of the technology is getting faster and faster. This is irrespective of the nature of the cloud – private, hybrid, public – all are seeing faster rates of adoption and enterprises are sooner than later going to be confronted with the question – how fast are we moving on their cloud initiatives.

The early initiatives of virtualzation hold an ominous parallel here. When server virtualization fever began to grip enterprises, we saw that everyone wanted to rush into that, so much so that many parallel initiatives were happening inside enterprises - so much so no one owned these at a central level and mostly many purused them as independent efforts – the result, enterprises began to face the rising prospect of VM sprawl. This created opportunities for vendors to sell new services around managing VM sprawl. The point here is – without an overaching vision, efforts which look successful in the short term but may create a formdable set of problems to manage – medium to long term!

Simple questions like who manages the process inside to manage these deployments – whats the overall security compliance etc.. In many organizations, individual efforts begin to bubble up at a central level only when security, risk , compliance efforts get pushed from the CFO/CIO organizations, That’s when most of the so called local efforts come together to show a gargantuan picture of the number of moving parts so to say that need to be shephered together..

When enterprises focus on shepherding and untangling the messy knots – already time has passed to examine the business value such efforts could bring in. The shift to cloud does not bestow long lasting benefits by just doing all over - rather I argue , it is going to come out of having a lasting vision, with a well detailed out program plan and aided by flawless execution. A strategy that speaks for itself and a concerted plan of action communicated well enough inside enterprises would go a long way in realizing benefits out of cloud efforts.

Cloud computing is all about driving holistic change in IT, but comes with the flexibility to start locally – this is a double edged sword – you can move fast locally thinking that bottoms up could work but harldly true based on real life experiences. What helps is having an overarching vision and tactically executing smaller programs locally in alignment with the overall vision.Such vision/strategy ought to illustrate how cloud facilitates pursuit and accomplishment of a set of goals articulated at the enterprise level. The plan should espouse the plan of adoption and address issues like compliance, security, governance, change management, partnership plans, auditing, provisioning , chargeback, exit strategies etc. Departmental efforts can draw from this plan the appropriate linkages and drive towards achieving the stated goals through their efforts. Think about this : such effors would help drive business value by bringing alignment across all internal initiatives, bringing cohesion across all efforts and help achieve larger goals such as capex to opex centric spending models, compliance, security etc. Such efforts would truly provide the basis of creating a platform centric IT infrastructure for enterprises to enable IT leverage for lot more benefits.

For cloud initiatives to be successful, enterprises need to focus on the larger end goal and have a game plan to achieve them in a truly centrally driven but localy executed model of execution. Success in large enterprise initiatives are always predicated upon their vision and execution – cloud adoption fits into this model very well. Is the partner system ready for this? The answer is Yes. The benefits of what is available today to service providers in the Cloud ecosystem provides more than ever the promise that they can focus on the business goals of the organizations they seek to support.

Typically larger enterprises need to focus on an array of things centrally to make progress on cloud adoption. Enterprises need to worry about new scale, different levels of security, reliability to support the new order of things. This means putting in place a new model of information and infrastructure governance, deployment, training, support - needless to say all these need to be reframed inside the enterprise. The IT environments need to be upgraded to provide for very high transaction workloads, security infrastructure needs to be retuned and app experiences need to be completely rehauled to provide a consumer feel for look and usage.

We have been waiting for the moment when transformative disruptions of this nature and scale happen. Learning organizations always had a charter to capture the latent knowledge that reside within people, process and systems, Existing technologies and methods could provide a lending hand to capture that vision in a limited way. People across the ranks inside organization can now participate in such transformation efforts quite easily and this could become the bedrock of new forms of collaboration leading to innovation and productivity improvements. Ironically this is mostly facilitated by advances in technology and communications and those responsible for technology management need to be ready to make this transformation possible inside their enterprises.

As I wrote elsewhere earlier, too often, inside enterprises, IT today is seen as the dampener towards embracing change at a mega-scale. IT organizations have an urgent need to transform themselves to keep pace with the change and become more relevant and stay aligned with business to leverage the opportunities that revolve around improving productivity, improving the efficiencies of operation, making organizations more innovative etc.. This also becomes a major force towards attracting talent. The clarion call here is for the enterprises to adopt to this change and create a new basis of competing – to distinguish from competition and create a leading distance from others through good plans, design and practice.

Swinging through the Trees on a Path Less Travelled

I had the doubly natural pleasure this week of facilitating a retreat for the Nature Conservancy in lovely Door County, Wisconsin. Like most organizations, this leading advocate for preserving lands and water is looking for innovative ways to more effectively accomplish their mission and attract more supporters (this was the marketing group). In keeping with the retreat's theme, "Innovation: Freeing your Inner Brilliance," I invoked our surroundings to make the case that individually and collectively they needed more often to swing from the trees (as in the classic Piraro cartoon, left), not only during our time together but also when they returned to the workplace.

For any culture to be to be truly innovative, it must honor individual divergence and originality. Are you empowered to follow your own passion and encouraged to support and build on your colleague's tree-swinging ideas without squelching them? We can learn to do this better when we separate diverging time from converging time, when we put off judging an idea much, much longer than we normally would. Instead of jumping to what is weak or wrong with a suggestion, we instead place it into consideration, let it breathe, build on it and play with it for a while. Later we can converge and decide which course to go.

Too many organizations today do not allow for breathing room--the instinct to judge and dismiss quickly (that's what smart people have learned to do) is so strong, that most half-baked ideas get smothered before they even have the possibility of rising. Without an environment for at least some half-baked, oddball or zany ideas--originality disappears and innovation becomes impossible.

I left the Conservancy group to brainstorm among themselves yesterday and I took to the road around the still-leafy County. I found myself walking through the woods, stimulated by how yellow the leaves were. Then it hit me--the yellow woods took me right back to a memory of the classic Robert Frost poem that I hadn't looked at for years. Let me refresh your own memory with this excerpt of Frost's "Road not Taken":

Two roads diverged in a yellow wood,
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth;

Then took the other, as just as fair,
And having perhaps the better claim,
Because it was grassy and wanted wear...

I shall be telling this with a sigh
Somewhere ages and ages hence:
Two roads diverged in a wood, and I—
I took the one less traveled by,
And that has made all the difference.

If you can't literally swing through the trees, at least you can more often explore paths less travelled.

Apple hits Home Run with latest quarterly earnings

Apple hit major milestones today in the latest quarterly earnings:

1. Highest Quarterly Sales of $20.3 billion, rising 67% from previous year
2. Highest Quarterly Profits of $4.3 billion, rising 70% from previous year
3. Highest Unit Sales of Mac, iPhone and iPad

“We are blown away to report over $20 billion in revenue and over $4 billion in after-tax earnings—both all-time records for Apple,” said Steve Jobs, Apple’s CEO. “iPhone sales of 14.1 million were up 91 percent year-over-year, handily beating the 12.1 million phones RIM sold in their most recent quarter. We still have a few surprises left for the remainder of this calendar year.”

Apple raised the outlook for the holiday quarter, and as Steve Jobs indicated, is planning to launch something exciting this year... Apple TV sold 250,000 units for the quarter.

In the conference call, Steve Jobs was elated, talked about winning, catching "tiger by the tail", and even joked about loving "Flash" ( memory). Mr. Jobs talked about creating the best products for the markets and customers Apple serves, and maintaining higher margins by providing the better Apple experience out of the box. Mr. Jobs quipped that he would like to learn from Nokia on how to create Smart phones for less than $50. Apple had a phenomenal quarter for the new iPad, and importantly, the businesses and education are finding new applications for the iPad.

“7-inch tablets are tweeners: too big to compete with a smartphone and too small to compete with the iPad,” said Jobs, adding that competing manufacturers were struggling to meet the price point of the iPad, which starts at $500. Both Samsung and RIM have not announced pricing on their tablets.

“These are among the reasons that the current crop of 7-inch tablets are going to be DOA — dead on arrival,” Jobs said during the earnings call!

After hitting another record daily high, Apple stock was down more than 6% in after hours, perhaps reacting to lower gross margins and pent up expectations.

Learn to innovate like Apple and Steve Jobs.
How does Apple, the #1 innovative company in the world, innovate and create game-changing innovations such as the iPod, iTunes, iPhone, iPad and more? What is Apple's secret recipe for innovation success?


Download Apple's Innovation Strategy, and learn how Apple became the #1 innovator through:

• Creativity and Innovation
• Innovation in Products
• Innovation in Business Model
• Innovation in Customer Experience
• Innovation and Leadership
• Steve Jobs Visionary Leadership
Revised in 2011! Steve Jobs interview

Learn more...

Read more:
http://www.wired.com/gadgetlab/2010/10/tablets-steve-jobs/#ixzz12ou3xuLP

Building Business Networks with Purpose

Many people have asked “what does it take to build a business network”.  There are three basic steps.

A.  You need an inspirational, aspirational and measurable mission and vision.

B.  Second, you need clear, aligning and measurable short and long term objectives.

C.  Third, you need to bring together groups of peers who can share knowledge and "Do Stuff".  

Most businesses and leaders do points A and B above, with some degree of success.  I am going to focus on point C, forming peer groups and getting stuff done through networks. This is the part that most leaders have trouble with because they confuse business networks and social networks.

 

What do you need to do to achieve the benefits of group dynamics, or point C above?

1.  First you need to build peer groups.  "Peers" are people or organizations interested in your mission and vision, which is point A above.   Peer groups can include businesses, government organizations, educational organizations, philanthropic organizations, customer groups, supplier groups, employee group, and so forth.

2.  Second, you need to engage your peer groups by sharing knowledge, information and insights.   This can be done by mailings, education sessions, news broadcasts, internet, or face to face meetings, among many other communication approaches.

3. Third, you need to encourage and support the groups to "Do Stuff" that helps to achieve your objectives (point B above).  For example, host a meeting, teach a course, write a blog, be an expert, form a support group, write a survey, create a group, etc, etc, etc.


The benefits and mathematics of these three steps are defined by Professor David Reed, commonly referred to as Reed's Law. Professor Reed notes that the value and impact of networks scale exponentially with the size of the network – expressed mathematically as 2 to the nth power.

For example, if your goal is to sell more product by making your company better know, if you can do this  through a network of five groups, your impact is 32 times (2 to the 5^th power) what it would be if you spent the same time and effort and only got to one person. Adding a sixth group would take the benefit to 64 times (2 to the 6^th power).

 

What do you need to do to achieve group dynamics and the directional benefits?

1.  You need a clear, aspirational and inspirational mission and vision that is translated into sound bites that the groups can understand and explain themselves.

2. You need quantifiable short and long term objectives that each group can help to execute.

3.  You need a communication strategy and program that gets the mission, vision and relevant objectives out as broadly and as inexpensively as possible.

4.   You group leaders who are viewed by their peers as experts or thought leaders.  Their job is to "garden" the group, engage members, and facilitate "Doing Stuff" to achieve the objectives.

5.  You need people who understand and can use the social and communication media technology.

6. You need a process to measure and report your performance against the objectives on a regular periodic basis.  The measurements and progress needs to be reported back to the groups so they can see the impact of what they are doing.


To recap, first you need a clear, aspirational and inspirational mission and vision with quantifiable short term and long term measurements of success.

The you can exponentially leverage your resources by building physical and virtual groups who share knowledge and “do stuff” in support of the objectives.


This model can be used by all organizations, whether business, educational, philanthropic, governmental, faith based, or special interest.  But its biggest impact will be in business networking with purpose.

One of the best examples of this model in action is The Global Leaders (www.tgleaders.com), or TGL. TGL gives you many of the tools, techniques, and platforms to form and benefit from group dynamics.  The Global Leaders (www.tgleaders.com) is free.

    

 

Sincerely,


George Bickerstaff

MySQL Disaster Recovery With Tungsten

Disaster recovery (DR) is not the first thing most DBAs think of when putting up a new database application.   However, it's one of the top issues for people using the data--what happens if the site goes down and everything disappears?   So even if DR is not the first issue in every deployment, it is a very high priority as soon as your application is the least bit successful.

At the database level DR has a fairly simple solution:  keep copies of data on a backup site that is up-to-date at all times.  This article explains the architecture for MySQL DR with Tungsten and a couple of key features that make it work, namely floating IP addresses and global transation IDs.  We will dig into those at the end.

First a bit of introduction.  Tungsten manages clusters of off-the-shelf database connected by master/slave replication.  There are replication and management services on each host with automated policies to handle failover as well as low-level tasks like recognizing new cluster members.  There is a simple management client that lets you log into any host and manage all nodes in the cluster.  Tungsten also has connectivity options to let applications find databases easily.  However, for this article we are going to focus on the database only and how you solve the problem of ensuring your data are protected. 

DR Setup

To implement disaster recovery, you actually create two clusters--one on your main site and one on a backup site which we will henceforth call the DR site.  It looks like the following picture.

Standard Main/DR Architecture with Backups

Here is an outline of the setup.  There are additional details of course but those are covered in Tungsten documentation and support procedures.  The goal here is to give you a sense of how things work at the top level. 

1. Main site.  Set up the main site cluster as a master/slave pair with a floating IP address on the master.  Enable automatic policy mode so that in the event of a master failure the local slave will immediately take over.  Set up backups and run them on the slave on a regular basis. 
2. DR site.  Next, set up the DR cluster by provisioning both databases with a recent backup from the main cluster.  Configure it identically to the main site with a master IP address and with backups but with two exceptions.  First, use manual policy mode so that the cluster does not try to fail over. Second, do not start replication automatically.  Instead, manually configure the DR master to be a slave of the main site master using the master floating IP address and start services.  Set up backups on this site as well. 

Handling Failures

At the end of setup you have a main site with a cluster and a DR site with a cluster that slaves efficiently off the main site master.  Both sites have regular backups.  As long as there are no failures, you operate both sites and everything is fine.  Let us now consider a couple of different types of failures and how to handle them.

Let's suppose the main site master fails.  Tungsten will automatically fail over to the main site slave and move the master floating IP address.  The DR site relay slave TCP/IP connection to the master will then break, or more accurately time out.  When the relay slave reconnects to the floating IP,  it will have shifted to the new master and replication to the DR site will continue without any human intervention.

Failed Master on Main Site

This protocol is handy because failures are not the only reason that the main site master may move.  You can also move masters for maintenance or upgrades.  Tungsten has a switch command that makes this very easy to do.  The floating IP moves as before and the DR site continues to receive updates properly after it reconnects. 

If you lose the main site, you initiate a site switch procedure.  At the database level this consists of running a script to "unconfigure" your DR relay slave node so that it becomes a master again and then reload the configuration.  When the node comes up as a master it will then automatically install its own master floating IP address.  The commands are simple and run in a few seconds.  In most cases it will take a lot longer to switch applications properly than switch databases, because you have to change DNS entries, start and/or reconfigure applications, and potentially activate other resources to have a functioning system. 

In fact, the real problem with site failover at the database level is not so much failing over but getting the main site back in operation without losing too much data and with as little interruption to users as possible.  You first need to check for any transactions that did not make it off the main site and apply them to the DR site master.  In MySQL you can do this by carefully applying transactions from the main site binlog.  You can help yourself considerably by including a step in the site failover process where you fence (i.e., turn off) the old site as quickly as possible by shutting down applications and taking applications offline.   The fewer extra transactions on the main site, the simpler it is to clean up.

Next, you need to get the master site resynchronized with the slave.  If there are more than a few differences, you will probably just restore the main site master and slave from local backups, then manually configure them to make the main site master a relay slave of the DR site.    If you have large databases, you may want to look at SAN or NAS products like NetAPP that offer snapshot capabilities.  I have been working lately with NetApp; the snap restore command is really impressive for rolling back file system state quickly. 

DR Site Operation and Main Site Recovery

Once the main site is caught up, you can switch applications back the main site by taking a short outage to move applications.  This step is not fully transparent, but unlike the original DR failover, you get to pick the time that is least inconvenient for your users.  Also, you can use Tungsten features like consistency checks to verify that data are consistent across sites. 

Underlying Tungsten Features to Enable DR

As promised at the beginning, here is a look at the Tungsten features that make DR work.  First, there is automated failover with floating IP address management.  Tungsten uses a rules engine combined with group communications to manage failover and master floating IPs efficiently.  The rules take care of many of the weird failure cases as well as handling tasks like automatically making slave servers readonly, etc.  Setting up DR without floating IP addresses is more complex because it means your relay slave needs to know when the main site master moves for any reason.

As useful as floating IP addresses are, Tungsten has a much more important feature that underlies the entire DR site architecture:  global transaction IDs.   Unlike native MySQL replication, Tungsten assigns a global ID or seqno to each transaction as it is read from the binlog.  Tungsten replicator processes track position using the seqno values rather than the file name and offset used by MySQL slaves.  Here is a picture that illustrates how the replicator log works.

Global IDs, Epoch Numbers, and Backups

As already mentioned, the Tungsten master replicator assigns the seqno to each transaction as it is extracted.  Tungsten slave replicators always use the seqno to request the next event from the master.  This means that you can switch the master without worrying whether slaves will lose track of their positions, because they will just ask for the seqno from the new master.

The other important feature of global IDs is that they make backups fungible across different databases and even sites.  Tungsten marks the database with the current seqno and epoch number.  As long as your backup (or file system snapshot) is transactionally consistent, you can load it on any server and bring it back online as a slave.  The new slave will connect to and catch up with the master, wherever it happens to be.  This makes database recovery both simple and very flexible.

The phrase "transactional consistency" brings up another issue.  To make the disaster recovery architecture work reliably I strongly recommend you switch to InnoDB or another fully transactional engine.  MyISAM does not have a place in this architecture--there are just too many ways to end up with corrupt data and a massive outage.

There is one final aspect of Global IDs in Tungsten that is worth mentioning.  What if the master log is corrupted or a slave from a different cluster accidentally logs into the wrong master?  In both cases the slave could get bad data if it just asked for the next seqno without some how checking that the master and slave logs are somehow consistent.  This would at best lead to errors and in the worst case to badly messed up data.

Tungsten deals with log consistency problems using epoch numbers. Whenever the master goes online it sets a new epoch number, which works like a parity check on the sequence number.   Each time a slave connects to the master, it offers the last seqno it received along with the epoch number.  If the values match the same seqno/epoch number in the master log, we assume the logs have the same master and proceed.  Otherwise, we assume somebody is confused and do not allow the slave to fetch transactions. 

Conclusion

DR site setup is complex and this article obviously glosses over a lot of details even for databases.  One final bit of advice is that whatever you do, test the daylights out of it before deploying.  Site failures may be karmic but dealing with them is certainly not.  Site failover is a really bad time to find out you don't have the password to your DNS provider handy or that you have a network configuration problem on the DR site.  One customer I know put all the computers from his main site and DR site in a pile on his conference room table and tested (and retested and retested and retested) until he was completely satisfied with the results.  That is the soul of true disaster recovery.

Apple Stock Explodes - On Fire - Before Earnings

Apple is all set to announce the quarterly earnings on Oct 18 after market close!

Apple hit an all-time high (again) today, closing the regular session at $314.74, a surge of $12.43 in one day, or 4.11%. Apple's market cap is now at $287.53 billion.

Apple is poised to announce quarterly revenue of at least $18.86 billion, earnings of at least $4.06 a share, and provide an uplifting outlook for the last holiday quarter, traditionally Apple's best quarter. This could mean Apple's stock has more upside going into the holiday season... An Apple analyst raised the Apple stock price estimate to $500 today.

Apple could report sales of 4.5 million iPads (not a typo), 9 million iPhones, 9 million iPods, and 4 million Macs... for a dizzying total sales of at least $18.86 billion. This is going to be a Monster quarter for Apple by all measures.

Important side note: This will mark the first quarter in Apple's storied history when it's quarterly revenue will exceed that of Microsoft's! Amazing indeed!

Apple's Innovation Strategy - Innovation eBook & Process

How does Apple, the #1 innovative company in the world, innovate and create game-changing innovations such as the iPod, iTunes, iPhone, iPad and more? What is Apple's secret recipe for innovation success?

What is Apple's Innovation Strategy? Download these Apple Innovation eBook insights and learn to be like Apple... like Steve Jobs, the innovator and CEO of Apple.

"There's an old Wayne Gretzky quote
that I love. 'I skate to where the puck
is going to be, not where it has been.'
And we've always tried to do that at
Apple. Since the very very beginning.
And we always will." -Steve Jobs

Apple innovates through:
• Creativity and Innovation
• Innovation in Products
• Innovation in Business Model
• Innovation in Customer Experience
• Innovation and Leadership
• Steve Jobs

This Apple Innovation Strategy ebook provides insights, strategy, best practices, facts and much more...

Apple has built an Innovation Factory – one that harnesses creativity in its people, stimulating new ideas, and launching successful, profitable new innovations... Apple leverages its diverse culture, innovation processes, partners and networks to seize the new opportunities in the marketplace and grow its business...exponentially…

How did Apple do it?
• Increase revenue more than 400% in 8 years…
• Increase net profit more than 650% in 8 years…
• Increase market cap more than thirty times to over $250 billion and
counting…

Download Apple's Innovation Strategy and learn to innovate, like Apple, today!!

Creativity highlights so far in 2010

As we head into our final quarter of 2010, I wanted to recap some of the most important headlines on creativity and innovation this year. In a year full of lingering economic and cultural malaise, the innovation imperative--our need to be more creative as a culture and as individuals--is as urgent as ever, particularly here in the U.S.

Stunning news (at least for me) broke in the spring when the largest IBM CEO survey ever identified "Creativity" as the "single most important leadership competency." In a business world that loves to overuse the word "Innovation" (most often defined as "applied creativity") but shies away from the more personal "C" word, creativity itself was heralded like never before, as I summarized previously. Now you can read the full report, called "Capitalizing on Complexity," and even access an interactive version by clicking on the graphic on the right. "CEOs now realize that creativity trumps other leadership characteristics," states the report. "Creative leaders are comfortable with ambiguity and experimentation. To connect with and inspire a new generation, they lead and interact in entirely new ways."

In July, Newsweek weighed in with a cover story called "The Creativity Crisis," which explored new research that has found that creativity test scores have declined since 1990 in the United States. The authors note that other countries are making creative development more of a national priority, with the European Union actually designated 2009 as the "European Year of Creativity and Innovation." "While our creativity scores decline unchecked," write authors Po Bronson and Ashley Merryman, "the current national strategy for creativity consists of little more than praying for a Greek muse to drop by our houses. The problems we face now, and in the future, simply demand that we do more than just hope for inspiration to strike." You can read the story and hear a fascinating radio interview about Torrance creativity tests, schools and adult creativity myths here, and can experience your own creativity test here.

Both of these headlines are calls for action, and one new source of inspiration comes from Steven Johnson, whose subtitle alone in his new book "Where Good Ideas Come From: The Natural History of Innovation," makes its release a worthy event this month. While his book takes a wide-lens and historical approach to human innovation, he shared some of his more contemporary conclusions about creativity in the workplace in this interview with Salon.com yesterday. "The problem," he says, "is that most traditional companies...talk a big game about innovation and making their workforce more creative" but do very little to change the culture or allow for ideas to be nurtured in the normal structure of daily work. He argues for "innovation time off"--like the "20% passion time" Google allows for employees to work on what they wish--so that "you're always spending a little bit of your time working on something weird that's not part of the official plan," a "permanent track of hunches and half-baked ideas that runs alongside the regular work-week with its immediate deadlines and fixed concepts." Yes.

A local upcoming highlight here in Chicago local inspiration is the always entertaining and eye-opening Chicago Innovation Awards, scheduled for November 1, which honors our region's most innovative new products and services. If you live here, rush to reserve a free ticket (this link should work) and come join me as we see some positive examples of how humans are demonstrating the #1 leadership competency and combatting the creativity crisis as best we can during this time in our natural history...

Making Consumer Technologies Help Enterprises Create A Winning Edge

There’s a revolution taking place in the IT ecosystem today. When more and more focus is put on innovation, its evolution, growth and in managing innovation while looking through what conventional collaborative mechanism in fusion with powerful mechanisms like internet enabled collaboration could help achieve –all these point to a world of immense possibilities. With a dominant number of internet users poised to take a dip in the virtual world, the virtual world could become more and more real!! Ten years back it is told that less than 3% of U.S. households had access to broadband. Today, it is estimated that nearly 90% of U.S. households have access to broadband.

Apple and the high tech semicon industry can vouch for the pull from the consumer segment – for both of them, consumer segment happens to be the largest consuming class!
Let’s look at the numbers : Morgan Stanley data tells us there are more than 1.2 billion consumers with Internet access, 700 million consumers with their own PCs, and 2 billion consumers with mobile phones, of which more than 400 million users have access to the internet through 3G connections. 1 billion users around the world access the internet through wi-fi connections. Let’s look at what customers do with these devices. Obviously, around the world, they use these devices to send billions and trillions of emails and text messages


-600 + million customers watch videos or YouTube or listen to music on the Web - most of them download music, video etc.

- 600+ million customers use instant messaging - a majority of them participate in online communities and a 100 million plus users actively blog

- The mobile brigade is not far behind. More than half a billion users access the internet from the mobile today around the world. The growth of the mobile markets is reinforcing the growth of the mobile apps market as well in a big way.

Of those 1–2 billion consumers using mobile phones, computers, and the Internet, 300 million people work inside enterprises. End of last year, the estimated number of connected devices at 6 billion devices on the planet — growing to 7 billion this year. These devices include camera phones, computers, , digital cameras, GPS devices , mobile devices, printers, smart phones, Internet phones, MP3 players, surveillance cameras, sensing devices and many other types of gadgets, gizmos, and chip-powered devices. Interestingly, amongst the population of connected devices, it can be seen that four out of five of these devices were not computers (either PCs or servers). Atleast, half of these devices are of the consumer electronics genre. The projections show that over the next three years, this category will double in size, growing much faster outpacing the growth of the traditional computer system.

Plain common sense thinking shows that while all these devices may not arrive inside the enterprise, the reality is that a majority of these devices would begin to show up in the enterprise. Obviously, consumers are bringing the chariots of fire with these connected devices into the enterprise – and in large numbers. The key thing to note here is that way above the investments made by the enterprise; we see that consumers in large measures are self investing in learning to use the connected devices and along the line are creating a huge market for tools and apps for connected devices.

This large group of consumers works with a huge swath of connected devices and wants similar access and experience of the applications and access mechanisms inside enterprises. They are getting used to accessing all information at near zero latency levels. The connected world may not respect the boundaries of enterprise and consumer in terms of experience and access - granted the security and audit mechanisms could be different between these two worlds. This expectation is creating a huge pressure on enterprise IT organizations to harmonize and integrate consumer-oriented devices and applications. Most of the young people joining the workforce have become quite accustomed to the experience that the world of mobile, social networking and connected devices . Clearly this expectation would keep increasing with time as more and younger workforce joins the enterprise.

Just like enterprises began to come under pressure to webify their enterprise(s) in the start of this decade, we see that the consumer driven revolution centering around connected devices are creating a new wave of pressure on enterprises to make their systems and processes ready for the connected devices. The Smartphones, Superphones, iPads, Social Networks get used /consumed quite extensively across the enterprise and at home - to stay informed, connected and productive in their professional as well as their personal lives. Add to that the changing usage demands of an always-on environment with anytime/anywhere access - this fundamentally changes the nature of support and service requirements.

A number of factors go towards making this shift happen: ranging from perceived gaps in the capabilities (in either functionality or ease of use) of existing corporate tools; employees incorporating their favorite social and collaborative tools into daily workflow; the low or absent cost of most consumer-grade tools and economic pressure to do more with less; and a narrowing of the differences between tools designed for the consumer and those built for enterprise. The gap between consumer and enterprise tools is narrowing quite rapidly. Gartner’s Nick Jones says he expects there will essentially be no difference between enterprise and consumer mobile tools within five years. The wave of change is indeed very powerful. New real-time cloud applications, platforms, and infrastructure offer the path to redefine the future of collaboration. The challenge before enterprise IT is to marry this phenomenon to business. We need to transform the business conversation the same way Facebook has changed the consumer conversation. If we look at the real world, Market shifts happen in real time, deals are won and lost in real time, and data changes in real time. Yet, all of us know that inside the enterprise, the software used to run is anything but real time. This forces enterprises to look for tools that work smarter, make better use of new technology (like the mobile devices in everyone’s hands), and fully leverage the opportunities of the Internet.

This “Consumer-Powered IT” trend is having a prolific effect inside enterprise IT. They are clearly changing the rules of the game in respect of the IT support models. It’s a powerful new way to work that will transform organizations over the next three to five years . Against this background, if we were to assess the readiness of most of the enterprises, we find that a majority of enterprises woefully unprepared to leverage this huge impactful opportunities. Such opportunities would revolve around improving productivity, improving the efficiencies of operation, making organizations more innovative etc..
The interesting thing here is change and transformation is happening bottoms up here – so we will see employees not waiting for things to change but their own expectations and expertise, becomes the change that enterprise IT needs to have to realize the potential of this changing nature of connectedness. The security risks, management issues, and policy and governance implications that arise from mass introduction of consumer devices and applications into the enterprise becomes a concern for enterprise IT to handle on their own. One of the things that internet did to traditional business is the deflationary effect it brought on them. Similarly the connected devices and social networks are going to bring in an unusually high degree of delayering within organizations so much so what appeared to be very sophisticated and confined to certain class of users may become very common and accessible to all those who seek them.

This means enterprises need to worry about new scale, different levels of security, reliability to support the new order of things. This means putting in place a new model of information and infrastructure governance, deployment, training, support - needless to say all these need to be reframed inside the enterprise. The IT environments need to be upgraded to provide for very high transaction workloads, security infrastructure needs to be retuned and app experiences need to be completely rehauled to provide a consumer feel for look and usage.

We have been waiting for the moment when transformative disruptions of this nature and scale happen. Learning organizations always had a charter to capture the latent knowledge that reside within people, process and systems, Existing technologies and methods could provide a lending hand to capture that vision in a limited way. People across the ranks inside organization can now participate in such transformation efforts quite easily and this could become the bedrock of new forms of collaboration leading to innovation and productivity improvements. Ironically this is mostly facilitated by advances in technology and communications and those responsible for technology management need to be ready to make this transformation possible inside their enterprises.

Too often, inside enterprises, IT today is seen as the dampener towards embracing change at a mega-scale. IT organizations have an urgent need to transform themselves to keep pace with the change and become more relevant and stay aligned with business to leverage the opportunities that revolve around improving productivity, improving the efficiencies of operation, making organizations more innovative etc.. This also becomes a major force towards attracting talent. The clarion call here is for the enterprises to adopt to this change and create a new basis of competing – to distinguish from competition and create a leading distance from others through good plans, design and practice.

Apple's Innovation Strategy - Innovation eBook & Process

How does Apple, the #1 innovative company in the world, innovate and create game-changing innovations such as the iPod, iTunes, iPhone, iPad and more? What is Apple's secret recipe for innovation success?

What is Apple's Innovation Strategy? Download these Apple Innovation eBook insights and learn to be like Apple... like Steve Jobs, the innovator and CEO of Apple. The eBooks has been revised in 2011, and contains Steve Jobs interview.

"There's an old Wayne Gretzky quote
that I love. 'I skate to where the puck
is going to be, not where it has been.'
And we've always tried to do that at
Apple. Since the very very beginning.
And we always will." -Steve Jobs

Apple innovates through:
• Creativity and Innovation
• Innovation in Products
• Innovation in Business Model
• Innovation in Customer Experience
• Innovation and Leadership
• Steve Jobs

This Apple Innovation Strategy ebook provides insights, strategy, best practices, facts and much more...

Apple has built an Innovation Factory – one that harnesses creativity in its people, stimulating new ideas, and launching successful, profitable new innovations... Apple leverages its diverse culture, innovation processes, partners and networks to seize the new opportunities in the marketplace and grow its business...exponentially…

How did Apple do it?
• Increase revenue more than 400% in 8 years…
• Increase net profit more than 650% in 8 years…
• Increase market cap more than twenty times to over $170 billion and
counting…


How does Apple, the #1 innovative company in the world, innovate and create game-changing innovations such as the iPod, iTunes, iPhone, iPad and more? What is Apple's secret recipe for innovation success?

Download Apple's Innovation Strategy, and learn how Apple became the #1 innovator:

If you are having download issues, send us an email at: info at innovationmain dot com

About the author:
Sanjay Dalal is an innovator and entrepreneur with over fifteen years of leadership experience in Silicon Valley and High Tech companies. Dalal authored and launched the Innovation Faculty eBook and Definitive Guide on Creativity and Innovation in business in 2008, used by over 650 leading organizations and professionals all over the world including HP, Hallmark, Cleveland Clinic, Pepsi, EDS, TATA, and major universities. Dalal published over 200 articles in the last two years on the real-time state of innovation in business at this blog on Creativity and Innovation Driving Business, and introduced the Innovation Index in December 2006 that correlates business, innovation and stock performance. Dalal filed joint U.S. Patent on "Hands-On Labs" for delivering live, hands-on training over Web Meetings by simulating a training lab environment. Dalal has launched innovative products such as WebEx Training Center and WebEx Sales Center to market, and grown product line revenue to tens of million dollars in annual revenue. Dalal holds executive certification on Leading Management Teams from Cornell University, and is an engineering scholar graduate in Computer Engineering from The University of Texas at Austin. Dalal attended Arizona State University for graduate education in Computer Science. Dalal is the CEO & Founder of OGoing Inc., small business social networking community.

Innovation eBook is brought to you by Creativity And Innovation Driving Business based in Irvine, CA.

Address:
111 Academy Way, Suite 100, Irvine, CA 92617
Main Phone #:
1-949-288-6880 (call-in only)

Committing Acts of Innovation: Help Needed...

I keep thinking about my friend George asking me how much I actually "commit acts of innovation"--it's one thing to write about creativity and innovation but quite another to actually do it. In its rawest form, the creative process is simply the opening up—and then closing—of possibilities. We are engaging in it all the time. When you have a decision to make, you open your closet door of ideas, rummage around to see what’s available, and then choose one as you close the door. A more complicated decision or challenge may require much opening and closing of different doors, the pivot between divergent and convergent thinking. To be creative, we have to open wide to a divergent stream of possibility—fully turn on our faucet of ideas—but also know how and when to filter, evaluate and narrow the stream so that we’re left with the best, nourishing drops.

Innovation requires sustained creativity, repeated diverging and converging, leading to a result--a product, an outcome, a more courageous and risky perseverance that makes a positive impact. That's not easy. Right now I'm trying to bring two acts of innovation more fully into the world--a new kind of theatrical show, the Malaise County Fair, and an interfaith program for kids, called Poetry Pals. Perhaps like other acts of innovation, I realize there are two challenging realities for me here: 1. I don't really know how to do it. 2. I need help from others. And I'm trying to bring these two projects to life while at the same time sustaining my consulting business of live learning, leadership and innovation events that provide me with the money I need to live (also a particularly difficult enterprise in this economy).

Both of these projects have the goal of helping us get out of our malaise, the numbness I believe we all feel living in a culture of information overload and political paralysis, where we express ourselves less and isolate ourselves more, where we distrust our neighbors and prefer cynicism to creative possibility and personal participation to change the world. I know for many these attempts are "unrealistic" and have questionable "market value." But I've chosen to take them on. And I need help.

The Malaise County Fair is designed to be a new form of entertainment, one in which the audience participates and stretches their own comfort zone, steps beyond the typical spectator role and acts as part of the story. It's a love story and multi-arts performance that takes place at an unusual fair--where creativity abounds and there is singing and opportunities for the whole community to jam together. You can experience a free introduction to it yourself this Sunday as part of the Gypsy Jam showcase (this Sunday at 3pm at Arts at Large, 3318 N. Lake Shore Drive, just north of Belmont, in Chicago). I'm looking for more creative folks to participate, help shape it and share their talents. Can you help? Can you come to the event to support us, get involved, recommend it to friends who might be itching to be involved in an innovative project?

Poetry Pals brings together kids--and their teachers and community--who usually would not meet. Our main program has been partnering Muslim, Catholic and Jewish school kids to get to know each other and write poetry together. Everyone involved agrees that it's been wonderful and so needed in a too-divisive culture that focuses on extremism and keeps us insulated from others who are different. Click here to read more and see me on a television program sharing what we're up to. But we need help. Are you interested in helping interfaith communication and learning about other cultures? Are you good at facilitating groups of kids and fostering friendship, conversation and a little poetry writing? Do you have access to diverse groups and can help us bring in more schools and partners and money? Do you know others who might? Please contact me for a flyer, help spread the word and come to a meeting.

I know we're all busy and that these may not be the projects that interest you. I would appreciate any way you might be able to help, including donations. I also encourage you to commit your own acts of innovation however you can, starting today.