Virtual IP Addresses and Their Discontents for Database Availability

Virtual IP addresses or VIPs are commonly used to enable database high availability.   A standard failover design uses an active/passive DBMS server pair connected by replication and watched by a cluster manager.  The active database listens on a virtual IP address; applications use it for database connections instead of the normal host IP address. Should the active database fail, the cluster manager promotes the passive database server and shifts the floating IP address to the newly promoted host.  Application connections break and then reconnect to the VIP again, which points them to the new database.
VIP-Based Database Availability Design
Virtual IP addresses are enticing because they are completely transparent to applications--no changes to database API behavior, no changes to connection strings, etc.  While virtual IP addresses seem simple, they depend on arcane TCP/IP behavior that is not especially well understood and not always consistent across different TCP/IP implementations.  Unless properly managed, virtual IP addresses can induce problems that run the gamut from simple lack of availability to split-brain, which can in turn lead to unrepairable data corruption. 

This blog article describes in some detail how virtual IP addresses work, specifically the behavior of Address Resolution Protocol (ARP) which is a core part of the TCP/IP stack that maps IP numbers to hardware addresses and permits VIPs to move.  We will then dig into how split-brain arises as a consequence of ARP behavior.  Finally we will look at ways to avoid split-brain or at least make it less dangerous when it occurs.

Note: the examples below use MySQL, in part because mysqld has a convenient way to show the server host name.  However you can set up the same test scenarios with PostgreSQL or most other DBMS implementations for that matter. 

What is a Virtual IP Address? 

Network interface cards (NICs) typically bind to a single IP address in TCP/IP networks.   However, you can also tell the NIC to listen on extra addresses using the Linux ifconfig command.  Such addresses are called virtual IP addresses or VIPs for short.

Let's say for example you have an existing interface eth0 listening on port 192.168.128.114.  Here's the configuration of that interface as shown by the ifconfig command:
saturn# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 08:00:27:ce:5f:8e
inet addr:192.168.128.114 Bcast:192.168.128.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fece:5f8e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10057681 errors:0 dropped:0 overruns:0 frame:0
TX packets:8902384 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6941125495 (6.9 GB) TX bytes:6305062533 (6.3 GB)
You can allow the eth0 interface to accept traffic for another address 192.168.128.130 using the following simple command. 
saturn# ifconfig eth0:0 192.168.128.130 up
This command tells the TCP/IP stack to accept packets directed to IP address 192.168.128.130 as well as the original address 192.168.128.114.   This means that if you are running a MySQL server on the host users can connect using either of the following commands:
mysql -utungsten -psecret -h192.168.128.114 (or)
mysql -utungsten -psecret -h192.168.128.130
Finally, you can move VIPs from host to host very easily by dropping them on the first host and binding to them on a second host, as shown in the following example:
saturn# ifconfig eth0:0 192.168.128.130 down
...
neptune# ifconfig eth0:0 192.168.128.130 up
Meanwhile, virtual IP addresses behave in every other way like standard IP addresses.  You can put them in DNS, applications can connect to them, etc.  VIP-based high availability is therefore a minimally invasive implementation for most applications--about the only requirement is that applications need to reconnect if their connection breaks.

How Virtual IP Addresses Work

To understand the weaknesses of virtual IP addresses for database high availability, it helps to understand exactly how the TCP/IP stack actually routes messages between IP addresses, including those that correspond to VIPs.   The following diagram shows moving parts of the TCP/IP stack in a typical active/passive database set-up with a single client host.  In this diagram host saturn currently has virtual IP address 192.168.128.130. Neptune is the other database host.  Mercury is the client host.


Applications direct TCP/IP packets using IP addresses, which in IPV4 are four byte numbers.  The IP destination address is written into the header by the IP layer of the sending host and read by the IP layer on the receiving host.

However, this is not enough to get packets from one host to another.  At the hardware level within a single LAN, network interfaces use MAC addresses to route messages over physical connections like Ethernet.   MAC addresses are 48-bit addresses that are typically burned into the NIC firmware or set in a configuration file if you are running a virtual machine.  To forward a packet from host mercury to saturn, the link layer writes in the proper MAC address, in this case 08:00:27:ce:5f:8e.  The link layer on host saturn accepts this packet, since it corresponds to its MAC address.  It forwards the packet up into the IP layer for acceptance and further processing.

Yet how does host mercury figure out which MAC address to use for particular IP addresses?  This is the job of the ARP cache, which maintains an up-to-date mapping between IP and MAC addresses.  You can view the ARP cache contents using the arp command, as shown in the following example.
mercury# arp -an
? (192.168.128.41) at 00:25:00:44:f3:ce [ether] on eth0
? (192.168.128.1) at 00:0f:cc:74:64:5c [ether] on eth0
Note that the ARP cache does not have a mapping for the VIP address 192.168.128.130.  Let's say we now make a client connection to the MySQL server at the other end of the VIP address on mercury:
# mysql -utungsten -psecret -h192.168.128.130
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 33826
...
mysql>
To route traffic, host mercury gets the IP address to MAC address mapping using an ARP request.  You can watch this happen in real time using the tcpdump command to track ARP traffic.
mercury# tcpdump -n -i eth0 arp 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:37:43.755081 ARP, Request who-has 192.168.128.130 tell 192.168.128.110, length 28
09:37:43.755360 ARP, Reply 192.168.128.130 is-at 08:00:27:ce:5f:8e, length 46
Now if you look at the ARP cache again on host mercury, you will see a proper mapping for the VIP in mercury's ARP cache:
# arp -an
? (192.168.128.130) at 08:00:27:ce:5f:8e [ether] on eth0
? (192.168.128.41) at 00:25:00:44:f3:ce [ether] on eth0
? (192.168.128.1) at 00:0f:cc:74:64:5c [ether] on eth0
Now if you go back and look at the picture (or still recall the details), you will notice that the MAC address maps to the NIC on host saturn.  This is exactly what we expect since saturn is listening on the corresponding virtual IP address 192.168.128.130. 

Virtual IP Addresses and Split-Brain

Most real problems with VIPs appear when you try to move them.  The reason is simple:  TCP/IP does not stop you from having multiple hosts listening on the same virtual IP address.  For instance, let's say you issue the following command on host neptune without first dropping the virtual IP address on saturn.  
neptune# ifconfig eth0:0 192.168.128.130 up
Let's further clear the ARP mapping for the virtual IP on mercury using the handy arp -d command and reconnect to MySQL.
mercury # arp -d 192.168.128.130
root@logos1:~# mysql -utungsten -psecret -h192.168.128.130
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 294
...
mysql>
So far so good--we logged into MySQL and everything appears normal.  But in fact it is not normal at all.  If you run tcpdump and watch the ARP requests during this login, you see the following:
# tcpdump -n -i eth0 arp 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:59:32.643518 ARP, Request who-has 192.168.128.130 tell 192.168.128.110, length 28
09:59:32.643768 ARP, Reply 192.168.128.130 is-at 08:00:27:68:cd:7d, length 46
09:59:32.643793 ARP, Reply 192.168.128.130 is-at 08:00:27:ce:5f:8e, length 46
This is not not just bad--it's very bad.  Both saturn and neptune responded to mercury's ARP request!  Mercury can pick only one for the mapping; which one it picks depends on timing as well as the exact implementation of the TCP/IP stack.  In other words we have a race condition and the winner is essentially random.

You can demonstrate the randomness for yourself with a simple experiment.  Let's create a test script named mysql-arp-flush.sh, which clears the ARP cache entry for the VIP and then connects to MySQL, all in a loop.  
#!/bin/bash
for i in {1..5};
do
arp -d 192.168.128.130
sleep 1
mysql -utungsten -psecret -h192.168.128.130 -N \
-e "show variables like 'host%'"
done
If you run the script you'll see results like the following.  Note the random switch to Neptune on the fourth connection.  
# ./mysql-arp-flush.sh
+----------+---------+
| hostname | saturn |
+----------+---------+
+----------+---------+
| hostname | saturn |
+----------+---------+
+----------+---------+
| hostname | saturn |
+----------+---------+
+----------+---------+
| hostname | neptune |
+----------+---------+
+----------+---------+
| hostname | saturn |
+----------+---------+
At this point you have successfully created a split-brain.  If you use database replication and both databases are open for writes, as would be the default case with MySQL replication, Tungsten, or any of the PostgreSQL replication solutions like Londiste, your applications will randomly connect to each DBMS server.  Your data will quickly become irreparably mixed up.  All you can do is hope that the problem will be discovered quickly. 

A Half-Hearted Solution using Gratuitous ARP

You might think that it would be handy if the ARP protocol provided a way to get around split-brain problems by invalidating client host ARP caches.  In fact, there is such a feature in ARP--it's called gratuitous ARP.  While useful, it is not a solution for split-brain issues.  Let's look closely to see why. 

Gratuitous ARP works by sending an unsolicited ARP response to let hosts on the LAN know that an IP address mapping has changed.  On Linux systems you can issue the arping command as shown below to generate a gratuitous ARP response. 
neptune# arping -q -c 3 -A -I eth0 192.168.128.130
This tells host neptune to send 3 ARP reply messages with its MAC address for the VIP address.  If we look at tcpdump output again, we see the following:
# tcpdump -n -i eth0 arp 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:02:27.154279 ARP, Reply 192.168.128.130 is-at 08:00:27:68:cd:7d, length 46
11:02:28.159291 ARP, Reply 192.168.128.130 is-at 08:00:27:68:cd:7d, length 46
11:02:29.162403 ARP, Reply 192.168.128.130 is-at 08:00:27:68:cd:7d, length 46
Linux hosts that receive the response will generally then update their ARP caches, though as we will see, there are some important exceptions.   But first, we need to show the effect of gratuitous ARP on MySQL connections.  Let's start with the following ARP cache contents on host mercury.  This shows an existing mapping to the MAC address of host neptune, which is what we expect from the previous arping command on neptune. 
mercury# arp -an
? (192.168.128.130) at 08:00:27:68:cd:7d [ether] on eth0
? (192.168.128.41) at 00:25:00:44:f3:ce [ether] on eth0
? (192.168.128.1) at 00:0f:cc:74:64:5c [ether] on eth0
Next, we run a loop that connects to MySQL and prints the host name every second.  The loop code is shown below and stored in a script named mysql-no-arp-flush.sh.  Unlike the previous script this does not release the ARP cache mapping between connections to MySQL.
#!/bin/bash
for i in {1..30};
do
sleep 1
mysql -utungsten -psecret -h192.168.128.130 -N \
-e "show variables like 'host%'"
done
While the test script is running is running, we run an arping command from saturn. 
saturn# arping -q -c 3 -A -I eth0 192.168.128.130
What we see in the MySQL output is the following.  Once the gratuitous ARP is received, mercury switches its connection from neptune to saturn and stays there, at least for the time being.
mercury# ./mysql-no-arp-flush.sh
+----------+---------+
| hostname | neptune |
+----------+---------+
+----------+---------+
| hostname | neptune |
+----------+---------+
+----------+---------+
| hostname | saturn |
+----------+---------+
+----------+---------+
| hostname | saturn |
+----------+---------+
There is one more interesting property of gratuitous ARP responses.  If you issue one during a session, it will cause client sessions to switch between hosts without waiting for a timeout.  Here's an example.  First login with MySQL and see which host we are on.
root@logos1:~# mysql -utungsten -psecret -h192.168.128.130
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 33853
mysql> show variables like 'hostname';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| hostname | neptune |
+---------------+---------+
1 row in set (0.00 sec)
Now issue an arping command on saturn using a separate window.
saturn# arping -q -c 3 -A -I eth0 192.168.128.130
Finally, go back and check the host name again in the MySQL session.   The session switches over to the other server, which you see at the client level as a lost connection followed by a reconnect.
mysql> show variables like 'hostname'; 
ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 33854
Current database: *** NONE ***
+---------------+--------+
| Variable_name | Value |
+---------------+--------+
| hostname | saturn |
+---------------+--------+
1 row in set (0.00 sec)
So is gratuitous ARP the solution to virtual IP split-brain?  It announces that there is a mapping change, which can make failover work much more quickly.  This is useful in its own right.  However, it does not solve split-brain. 

First, not all TCP/IP stacks even recognize gratuitous ARP responses.  Second, gratuitous ARP only takes effect on hosts that actually have a current mapping in their ARP cache.  Other hosts will  wait until they actually need the mapping and then issue a new ARP request.  Finally, ARP mappings automatically time out after a few minutes.   In that case the host will issue a new ARP request, which as in the two preceding cases brings us right back to the split-brain scenario we were trying to cure.

Avoiding Virtual IP Split -Brains

Avoiding a VIP-induced split-brain not a simple problem.  The best approach is combination of sound cluster management, amelioration, and paranoia. 

Proper cluster management is the first line of defense.  VIPs are an example of a unique resource in the system that only one host may hold at a time.   An old saying that has been attributed to everyone from Genghis Khan to Larry Ellison sums up the problem succinctly:
It is not enough to succeed.  All others must fail.  
The standard technique to implement this policy is called STONITH, which stands for "Shoot the other node in the head."  Basically it means that before one node acquires the virtual IP address the cluster manager must make every effort to ensure no other node has it, using violent means if necessary.   Moving the VIP thus has the following steps.
  1. The cluster manager executes a procedure to drop the VIP on all other hosts (for example using ssh or by cutting off power).  Once these procedures are complete, the cluster manager executes a command to assign the VIP to the new owner. 
  2. Isolated nodes automatically release their VIP.  "Isolated" is usually defined as being cut off from the cluster manager and unable to access certain agreed-upon network resources such as routers or public DNS servers. 
  3. In cases of doubt, everybody stops.  For most systems it is far better to be unavailable than to mix up data randomly.  
Cluster managers like Tungsten and Pacemaker handle this kind of process very well.   PaceMaker for example has a number of specialized hooks to cut power or otherwise use extreme violence to ensure proper fencing of databases.  Tungsten has fewer such hooks but has a much richer set of operations for databases and also has a wide set of connectivity options for HA besides using VIPs.

Incidentally, you want to be very wary about re-inventing the wheel, especially when it comes to DBMS clustering and high availability.  Clustering has a lot of non-obvious corner cases; even the "easy" problems like planned failover are quite hard to implement correctly.  You are almost always better off using something that already exists instead of trying to roll your own solution.

Amelioration is the next line of defense, namely to make split-brain situations less dangerous when they actually occur.  Failover using shared disks or non-writable slaves (e.g., with DRBD or PostgreSQL streaming replication) have a degree of protection because it is somewhat harder to have multiple databases open for writes.  However, it is definitely possible and the cluster manager is your best bet to prevent this.  However, when using MySQL with either native or Tungsten replication, databases are open and therefore susceptible to data corruption, unless you ensure slaves are not writable.

Fortunately, this is very easy to do in MySQL.   To make a database readonly to all accounts other than those with SUPER privilege, just issue the following commands to make the server readonly and ensure the setting is in effect:
neptune# mysql -uroot -e "set global read_only=1"
neptune# mysql -uroot -e "show variables like 'read_only'"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | ON |
+---------------+-------+
This protects you not just in cases of actual failover but also from administrative mistakes or software failures that switch the VIP by accident.   Many cluster managers implement read-only slaves.   Tungsten clustering has explicit support for read-only slaves.  Other cluster managers like MMM and Pacemaker can do the same.

Lastly paranoia is always a good thing.  You should test the daylights out of clusters that depend on VIPs before deployment, and also check regularly afterwards to ensure there are no unexpected writes on slaves.  Regular checks of logs are a good idea.  Another good way to check for problems in MySQL master/slave setups is to run consistency checks.   Tungsten Replicator has built-in consistency checking designed for exactly this purpose.  You can also run Maatkit mk-table-checksum at regular intervals.   Another best practice is to "flip" masters and slaves on a regular basis to ensure switch and failover procedures work properly.   Don't avoid trouble--look for it!

Conclusion and Note on Sources

Virtual IP addresses are a convenient way to set up database high availability but can lead to very severe split-brain situations if used incorrectly.   To deploy virtual IP addresses without problems you must first of all understand how they work and second use a sound cluster management approach that avoids split-brain and minimizes the impact if it does occur.  As with all problems of this kind you need to test the implementation thoroughly before deployment as well as regularly during operations.  This will help avoid nasty surprises and corrupt data that are otherwise all but inevitable.

Finally it is worth talking a bit about sources.  I wrote this article because I could not find a single location that explained virtual IP addresses in a way that drew out the consequences of their behavior for database failover.  That said, there are a couple of good general sources for information on Internet tools and high availability:
Beyond that you can look at general networking sources like Radia Perlman's Interconnections, Second Edition or Internetworking with TCP/IP by Douglas Comer.  These are more high-level.  If you get really desperate for details, try the RFCs, for example RFC-826, which is the original specification for ARP.  Some of them are surprisingly good reads even 30 years after the fact.

Tungsten Replicator Overview Webinar

On Thursday January 27th at 10am PST I will doing a webinar on Tungsten Replicator together with my colleague Giuseppe Maxia.  The title is "What MySQL Replication Cannot Do.  And How to Get Around It."  Basically it is a nuts and bolts description of Tungsten Replicator capabilities like multi-master replication, failover, parallel apply, and using replication for zero-downtime upgrade.  If you have ever wanted an in-depth look at the Tungsten Replicator this is a good opportunity. 

During 2010 we implemented an amazing number of new replication features ranging from pipelines early in the year to fast disk logs, multiple replication services per process, bi-directional replication, and parallel apply by the end.  We will be building out all of these in the coming year and releasing increasingly capable features into open source as well. 

This presentation is part of Continuent's regular webinar series which means we will also talk a bit about commercial products and services at the end.  However, it's mostly cool replication stuff.   You can sign up on the Continuent webinar page.  Hope to see you there.

We Just Made Finding Talent Easier!


While the Internet has made it easier for applicants to search for open positions, it has also created unexpected challenges for the applicants and the companies seeking top tier talent.  For the job seeker it's getting noticed in a sea of resumes and for the search firm or hiring company, it's wading through the large number of unqualified applicants.

Introducing TGL Jobs.  TGL Jobs allows search firms & companies to leverage our proprietary networking platform's feature of industry and position specific groups whose members have already been vetted.  Job searches can be placed in front of those individuals that have the experience that matches the position.

For a fraction of the cost that other sites charge, you can place your post in specific groups; i.e. Chief Financial Officer, Chief Marketing Officer, etc. to insure that you are recruiting qualified candidates or getting leads & referrals from executives who understand your needs.  Get started today and find your next rising star...read more

Stressed out childhood and squelched out imaginations

Students are so overscheduled they can't think straight in Race to Nowhere, a recent documentary I screened with a community group of questioning parents and frustrated educators this weekend. Between the pressure for kids, earlier-than-ever, to compete to get into the "right" college and on educators to teach to the tests that may or may not measure "achievement," we now have an education culture that more often than not squelches the imagination of its best students--the ones whose creativity we need more than ever.

The film examines mostly high-achieving communities, and uncovers what Stanford professor Denise Pope captured almost a decade ago in a study turned book called Doing School: How We Are Creating a Generation of Stressed-Out, Materialistic, and Miseducated Students. The way we are doing school and the cost to our students, she revealed, is out of wack from our real goals of education. Race to Nowhere (click for trailer here if you don't see it below) reveals what too many parents know--the demands on our kids to succeed have led to grueling routines, sacrificed sleep, cheating and stress, depression and anxiety. The result is that students "do school," chase grades and college application impressiveness, going from one activity, homework assignment and memorized-before-forgotten information gulp to another. The question is, what are our students actually "learning" about how to live their lives? "Things that actually get our students to think are pushed aside," says one dedicated teacher from the film, who left her job as a teacher because she could no longer abide by her own district's test-taking demands.



This last week has seen related conversations percolate about how parents and kids deal with competition, discipline and being the best, thanks to Amy Chua's Battle Hymn of the Tiger Mother, which makes the case that our "weak-willed" and "indulgent" culture, compared perhaps to China or at least the demands of some Chinese-American moms like Chua, has our kids growing up ill equipped to compete in a fierce global marketplace. While Chua's arguments for "tenacious practice" and no excuses are worth being reminded of, Race to Nowhere shows how out of balance we've become. And keep in mind, as this week's TIME cover story about Tiger Moms points out, that many educated Chinese are seeking out the more "relaxed" American style of education--wanting to move away from rote memorization to more right-brained learning "because they know they must produce more creative and innovative graduates to power the high-end economy they want to develop."

Just like our too-busy adult culture today, a too-busy, overscheduled, and digitally addicted childhood ends up squelching creativity, indviduality and passion. We need an education system that inspires and engages and allows families to spend time together. Kids need downtime and free time to process and to imagine and to play, for social and emotional health, as well as the creative future of our culture.

Right now it's a race that is running us. It reminds me of the time I was in Japan and met up with a top English teacher there. She laboriously tutored kids into the night, almost every night, after their full day of classes. Turns out she could not communicate with me in English at all! Japanese kids were memorizing and studying a faux-English language that didn't really exist just so they could pass a test! More and more American kids are doing six hours of homework a night and forgetting everything they learn by the next week.

Every teacher and administrator I know is overwhelmed with standards and testing. Every parent fears their kid isn't doing enough to make it to the next school or threshold. Watch Race to Nowhere to remind you of other truths: that elementary kids don't need to do homework to thrive, that there is honor and smarts in kids who don't excel academically, and you can be successful if you don't go to the best college. The filmmakers describe the film as a call "to mobilize families, educators, and policy makers to challenge current assumptions on how to best prepare the youth of America to become healthy, bright, contributing and leading citizens." I call it a sanity check.

SFO Event a Big Hit

The Global Leaders first networking & awards ceremony for 2011 was a huge success and received favorable reviews by members, as well as first time guests. 

"The TGL event was my first, but was utterly enjoyable. I met a diverse group of smart and experienced people, was inspired by two marvelous speakers whom I also got to speak with and get to know a bit, and found myself making connections with a vast array of folks I could not imagine meeting all in the same place before I joined the global leaders!"…..Bruce Berkoff, Chairman, LCD TV Association.

Held in San Francisco during the JP Morgan Healthcare Conference week, TGL had almost 200 senior level executives in attendance at their first ever event on the west coast and the first event of the new year.  During their careers, those in attendance,  had led over 300 companies, headquartered in 26 countries, supporting 57 industries, with revenue exceeding $480 billion, assets of $3.1 trillion and employing 1 million.  These executives took the time out of their busy schedules to mingle with colleagues and exchange ideas & thoughts about the opportunities that 2011 has to offer.

"I rarely have time dedicated to networking and the TGL forum provided me with the opportunity to connect with a number of leaders facing the same challenges as me and taking advantages of opportunities in front of each of us.  I am also looking forward to continuing to network with these leaders through the private group that TGL created especially for this event and its participants; that is an added bonus that is greatly appreciated."…..Steve Collins, Director M & A Advisory Services at PwC


G. Bickerstaff, M. Potter, D. Hee & J. Gitney

Guest speakers included Myrtle Potter, renowned leader & innovator in Healthcare, and Olympic Gold Medalist & inspirational speaker, Dana Hee.  In addition, TGL took the opportunity to recognize three promising “Future Global Leaders” from the greater San Francisco Area:  David Boyer, Edna Mabanes Casteel and Darian Rodriguez Heyman.  (Read more about these remarkable individuals who were recognized for their Innovation, Leadership & Service to Community.)
To encourage attendees to keep the lines of communication and exchange of ideas open, TGL created a private group for all of the registered participants. “As Mr. Collins of PricewaterhouseCoopers eluded, this is a unique feature that TGL has added to benefit our members.  Most individuals, during any event, will only have an opportunity to have 5-6 meaningful dialogues and lose track of other contacts made during casual introductions & conversations.  By creating a private group, that is only accessible to those who registered for this event via a personal invitation to join, members can easily continue to network with each other long after the event is over”, cited Jim Gitney, CEO & Co-founder of The Global Leaders.  “Our plan is to create these ‘invitation only’ groups for each of our networking meetings that we have scheduled for 2011 with each meeting having their own unique group”, added Gitney.
The Global Leaders will be releasing details for their next event to be held in Los Angeles during the month of May next week.   Other events have already been scheduled to coincide with the World Innovation Forum (June 7th) and The World Business Forum (October 5th) in New York City; TGL is a supporting sponsor for both the WIF & WBF.  Tickets for the events in NYC are already on sale, as well as special packages which include admission to both the WIF and WBF.  More information is available at Upcoming TGl Sponsored Events.

A couple of thoughts for MLK Day



"Almost always, the creative dedicated minority
has made the world better."
~ Martin Luther King, Jr.

"Human Salvation lies in the hands
of the creatively maladjusted."
~ Martin Luther King, Jr.


Click here for video if you don't see it below

What Happened to My Business?

pastedGraphic.pdf

What Happened to My Business?

January 2011


Overview

Every year, millions of new businesses declare bankruptcy; in fact, the majority of new businesses fail within five years. For a business to thrive, it should have the following four components - a vision, a strategic plan, an operating plan, and a financing plan. This article summarizes the common pitfalls awaiting the new business owner and explains the key actions needed to avoid these problems and build a sustainable business.

Background

There are over two million companies in the world big enough to be tracked by the big business information services, such as Standard & Poor’s and Dun & Bradstreet, as well as millions of smaller businesses. The total revenue generated by these companies is about 53 trillion dollars per year, which is over 91% of the global gross domestic product, currently at 58 trillion dollars according to the CIA Factbook. These companies employ millions of people around the world, although a lack of transparency makes the exact numbers difficult to ascertain as the corresponding data are unavailable.


The following chart provides the pertinent information for a representative sample of these large businesses. It includes the number of operating companies in each major geographic region of the world, as well as the annual revenue generated by these operating companies and the number of employees of each.


Global Statistics for Reporting Companies




Geographic Region

Operating Companies

Revenue ($mm)

Employees

United States of America

185,479

$ 17,320,530

40,147,419

Europe

46,501

10,612,093

77,698,783

Asia / Pacific

32,673

13,252,804

47,332,914

Africa/Middle East

6,307

1,536,132

3,903,746

Canada

4,918

1,750,966

5,563,282

Latin America and Caribbean

1,970

2,062,460

5,954,212

Total

277,848

$ 46,534,986

180,600,358


Source: Public global business data services

Every year, many of these large companies as well as countless smaller companies declare bankruptcy. This article illustrates the issues involved, and will provide advice on how to hurdle the obstacles facing the new business owner.

The Issues

The global gross domestic product is currently estimated at 58 trillion dollars and about 24% of this amount is generated in the United States alone. According to the Small Business Administration, over 50% of new businesses fail within five years of inception. There are several reasons for this, but virtually all of them result from a lack of foresight and can be traced back to the vision, the strategic plan, the operating plan, and the financing plan.


A close look at the bankruptcies reveals that large, established companies are not necessarily immune. The following graph represents a small sample of the companies tracked by information services, such as Standard & Poor’s and Dun & Bradstreet.


pastedGraphic_1.pdf

It shows that even big businesses are susceptible to the effects of a weak economy, as illustrated by the large increase in companies declaring bankruptcy in early 2009, coinciding with the economic downturn. As the economy improved, the number of bankruptcies steadily decreased since the third quarter of 2009. It is important to remember that large, well-funded companies are vulnerable. In fact, the sheer size of these businesses means that they require even more precise planning than a smaller company that can be kept running on very little funding.


It is also important to investigate what kind of companies are most likely to be forced to declare bankruptcy. The following graph illustrates the bankruptcy count by sector; consumer discretionary goods are the hardest hit in a poor economy, because the general public will cut down on unnecessary or luxury purchases. The Industrial and financial sectors also show high bankruptcy rates, because their success is closely tied with the economy’s ups and downs.

pastedGraphic_2.pdf

The effects of these bankruptcies are illustrated in the following chart, which shows the five-year sector stock growth. Currently, the only sector not experiencing any growth is financials. While the consumer discretionary goods are hardest hit, the sector rebounds more quickly than financial markets.

pastedGraphic_3.pdf

Bankruptcy is a very real danger for any new business; especially within the first five years. It is important for every business owner to recognize and understand how critical it is to have all four key components to avoid failure. Vision, planning, execution and funding are all essential for survival.


Whereas most new businesses have a vision, many of them are without a clear plan, fail at execution, and lack adequate funds, especially during the initial start-up phase. This lack of funding gives management no time to recover from poor planning and execution. A proper vision, planning, execution, and funding will prevent these mistakes, enabling a new business owner to build a successful and sustainable business.

The Solutions

The Vision

The first component of a successful business is the simplest; every business must have a vision. The vision should be articulated in a mission statement that encapsulates the expectations and goals of the business. A good mission statement should include both the vision and the core values that guide the company and its employees, serving as the foundation of the business. The vision itself should be clear, compelling and achievable.


The Strategic Plan

The second step to success is the strategic plan, which is a broad framework addressing the needs and direction of a company over the first three to five years. This is where the most common mistakes are made; numbering among them are errors such as an inappropriate location, ill-advised staffing decisions, ineffective marketing strategies, and lack of technological support, such as a website or calling center. Many new businesses also have problems with the structure and direction of the company, such as little to no internal controls and accountability, as well as overexpansion. A successful strategic plan must be clear and specific, containing provisions for all aspects of the business. It must also provide for all eventualities, addressing both the best and worst case scenarios. Keep in mind that a plan must be focused, but flexible. If the situation changes, one must allow for deviations from the proscribed path. It is, simply put, a living document.


The Operating Plan

The operating plan is just as important as the strategic plan; it also must contain clear directions and specific provisions. However, unlike the strategic plan, the operating plan focuses on the actions and issues of the current year, detailing the precise steps to be taken on a month-to-month basis. The most common mistakes at this stage are poor management, lack of experience, hasty or imprudent decisions, and inability to adapt. In order to ensure success, a business owner must follow the operating plan closely, approaching critical junctures with composure and making sure not to deviate from the set policy. This is not to say, however, that one must follow the plan to the detriment of the business; as with the strategic plan, the operating plan must be prepared to adapt to different situations as the need arises. If something changes drastically, a savvy business owner should make the necessary alterations, or if the situation warrants it, a new plan should be constructed. Danger lies in willful abandon of a successful plan simply because the confusion of unexpected circumstances.


The Financing Plan

The final component of a successful business is an effective financing plan. It takes at least five years for most businesses to turn a profit, so a business must be supplied with enough funding to last through the profitless years, with surplus funds for unexpected circumstances. Specifically, a new business must avoid high overhead costs, overspending, and poor credit arrangements. A good financing plan will make provisions for all financial decisions, as well as stress the value of long-term gains over short-term profits.


These four crucial requirements are easily explained, but not quite as easily fulfilled. The vision must be viable and the strategic and operating plans must be effective. The financing plan must include sufficient funding, including reserve funds for unforeseen situations. These four components should ensure the success of any new business.

Interview

In the following interview, George Bickerstaff, co-founder of The Global Leaders, explores the characteristics of a successful company.


Question: Can you give us an example of an effective vision or mission statement?

Answer: Every good mission statement should be clear, concise, aspirational and inspirational. The following companies have effective mission statements that are simple, yet summarize the ethos of the company as a whole.


Facebook - To give people the power to share and make the world more open and connected

General Electric - Imagine, solve, build and lead

Google - To organize the world’s information and make it universally accessible and useful

Harvard Business School - We educate leaders who make a difference in the world


Question: Who should be able to articulate the mission statement?

Answer: Every stakeholder in the company should understand and be able to articulate the goals and expectations of the company.


Question: And how and when should this vision be communicated?

Answer: The vision should be communicated often and to everyone. A good company must have a consistent and clear message, so that everyone involved, from low-level employees to the CEO to the clients, knows what the company is all about.


Question: So, once a company has a vision, what’s the next step?

Answer: Well, the vision is the business dream and that dream cannot be achieved without a plan. The next step is to prepare a strategic plan to highlight the key issues and the actions that the organization must take in order to achieve that dream.

Question: How often should this strategic plan be prepared?

Answer: A good strategic plan should address the first three to five years, but it needs to be updated annually order to incorporate changing information, such as the markets, customer needs and competition.


Question: And how many strategic issues and actions should the strategic plan identify?

Answer: A successful plan will have about ten of each. Less than this means that management has not considered all of the issues or actions to be taken, and more than this is probably not doable and certainly not strategic. These should be considered your “top ten strategic actions”.


Question: Should the operating plan include specific measurements, such as dollar amounts, percentages, and so forth?

Answer: Definitely. As the old saying goes, you cannot manage what you cannot measure. High-level, specific measurements are necessary. For example, a goal could be to increase the market share from 20 to 30% over five years, or to increase the revenue from emerging markets from 10 to 25 million dollars within five years, or to improve quality to less than 0.1% error rates.


Question: How does this strategic plan tie into the operating plan?

Answer: The strategic plan is the broad framework and covers the next three to five years, while the operating plan is the issues and actions that should be addressed in the current year. It is also sometimes called the budget.


Question: Who owns the operating plan?

Answer: Well, the CEO is responsible for the preparation of the operating plan, and the board of directors is responsible for reviewing and approving the plan on behalf of the shareholders. However, key management must be involved in the preparation of the plan and have to completely understand its provisions its order to manage effectively.


Question: How important is it to achieve the annual budget?

Answer: Every great business should plan and achieve its plans with good governance and accounting controls. Clearly, a business should never fall short of its goals, but in a sense it is also undesirable to exceed the parameters of the plan in place, because it means that the plan failed to provide for the circumstances. A good plan should include cover all eventualities.


Question: Now let’s talk about the financing plan. Why is it so critical?

Answer: The financing plan is the key to success. Almost all good businesses fail before they can achieve their objectives - they simply run out of money. A financing plan is essential to ensure the business has sufficient funds and will last long enough to start making a profit.


Question: How difficult is it to create a financing plan?

Answer: Like with the vision, strategic plan, and operating plan, it is necessary to have expert input both from inside and outside the organization. There are as many options to finance as there are different types of companies. It is absolutely imperative to have the best advisors; they should be competent people that you can trust.


Question: How far out should the financing plan look?

Answer: It should look out as far as the strategic plan and operating plan, which is generally three to five years, but under no circumstances should it address less than the next two years.


Question: When should a company raise money externally?

Answer: Whenever it can. Companies should look to external money-raising opportunities whenever the markets let them, and they should always have funding for at least two years.


Question: Finally, what are the different funding sources available to new businesses?

Answer: That’s a great question, but it’s one I’ll hold off on answering for now, as I’m planning to address it in future publications.

Additional Information

For more information, please visit The Global Leaders bookstore, which carries supplementary materials, including detailed versions of the spreadsheet in this article.


Furthermore, each regional group of The Global Leaders provides its members with information and documents specific to that region.

Acknowledgements

Author: George Bickerstaff

George is a board member, investor, partner and trustee for various businesses and philanthropic organizations; he has also held senior positions with companies such as Dun & Bradstreet, General Electric and Novartis. George has been instrumental in establishing the strategic, operating and financial direction for numerous private and public companies and has provided financial leadership in mergers valued at more than $50 billion during his career.

For more information, please visit: http://www.linkedin.com/in/bickerstaff


Contributor: Caroline Tara Frey

Caroline graduated with a law degree from Oxford University in 2009. She is now pursuing her legal career in the United States, having successfully completed the New York State Bar Exam in July 2010.

For more information, please visit: http://www.linkedin.com/in/carolinetarafrey

Fixing Replication with Replication

A couple of days ago I ran into a Tungsten Replicator case where several MySQL tables became corrupted on slaves and needed to be restored from the master.   We identified the tables that had problems fairly quickly using Tungsten Replicator's consistency checks.  However, that led to another problem:  how to restore the slave tables efficiently from the master.  The MySQL server in question processes around 10M tranactions per day--there is virtually no downtime.  Though the tables were not large, we could not be sure whether they were in use. 

Fortunately, you can use a simple MySQL trick to get all the rows of a table to replicate through to slaves.  The idea is to dump the table, delete the rows, then reload it again.  The delete and subsequent reload replicate out to slaves, after which everything is consistent again.  Let's say we have a table called tpcb.history that needs to be fixed.  Login with mysql and run the following commands:
BEGIN;
SELECT * FROM tpcb.history
INTO OUTFILE '/tmp/tpcb.history.dmp' FOR UPDATE;
DELETE FROM tpcb.history;
LOAD DATA INFILE '/tmp/tpcb.history.dmp' REPLACE
INTO TABLE tpcb.history FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n';
COMMIT;
You can do the reload several ways in MySQL, but this particular code has some advantages over other approaches, such as using LOCK TABLES.  First, it uses a transaction, so if something goes wrong the changes roll back and you do not lose your data.  Second, the SELECT ... FOR UPDATE locks your data and ensures serialization.  You can run this while applications are running without problems.

This seems useful enough that I put together a simple script called reload-table.sh with a README and checked them into the Tungsten Replicator codeline on SourgeForge.net.  You can refresh the same table shown above using the following command:
[sudo] ./reload-table-b.sh -u tungsten -p secret -t tpcb.history
I tested the reload using Tungsten 1.3.1 on MySQL 5.1 with statement replication.  However, it would work equally well with row replication.  Moreover, you can do the same trick in MySQL replication, as this involves base replication capabilities that are directly equivalent.  There are a few caveats:  you need to use InnoDB (or another transactional engine), large tables may be a problem, and you would need to tread carefully in cases where tables contain referential constraints.  Finally, it would be wise to save the master table somewhere else before running the script.

Future Global Leaders Honorees Announced

We're proud to announce three incredible individuals who will be recognized this Thursday at our networking event being held at the Marines' Memorial Club.  Check out the honorees now.

If you'd like to come celebrate with the honorees, there is still time to register for our first networking event on the West Coast.  Featured speakers include Myrtle Potter, renowned Health-care Leader & Innovator, and Olympic Gold Medalist & Inspirational Speaker, Dana Hee.  Register now at http://the-global-leader.eventbrite.com/

New Year, New Call: for United States of Innovation

As we start a new year (and, some would say, a new decade) as an already reeling country now reeling even more from the shooting of a Congresswoman, I'm grappling with the state of the United States. There is no question that we are poorer than we were when I grew up, and there is much evidence that we are more divided and pessimistic than we've ever been. We need something (as opposed to "someone," which hasn't seemed to work) to rally behind. And I'm going to vote, once again, for "Innovation." Hear me out.

If there is anything that we've learned as Americans in this past decade it's that there are many versions of "America," and plenty of other Americans who don't see it the way you do. Red state-Blue state-Tea Party-Who's-the-smarty?-Obama-Drama, Hey! Our politics and media--which feed our sense of country and confidence--are so far out of wack that we don't know what is true or what to believe. We do know that from healthcare to housing, we've lost a lot of our wealth lately, and that as a culture and as individuals, we have more challenges than ever before. Some say that we are seeing the irrevocable fall of the American empire. Some say technology can save us, empowering us to write blogs like this that lead to less alienation than in previous years. Not all of us believe that though. What can really unite us?


It's "Innovation." I put it in quotes because we still need to collectively define what that means and to better understand creativity, the engine of innovation. But right now the political left and right, the CEOs and the artists, miraculously agree that innovation is needed, and that our future is dependent on leveraging our innovation capacity. I've called this the Innovation Imperative--that we need creativity and innovation more than ever, for economic, cultural and personal reasons.

Economic: America, with its diversity of ideas, free enterprise, university research and available capital, still has the raw materials for economic innovation. The Obama administration had been more quiet about "innovation" in 2010 than it had been the year before and, as I've discussed before, certain measures of innovation seem to be on the decline. But both political sides, as well as leaders everywhere, know that "innovation" is the key to us getting out of our economic hole. The "United States of Innovation" is the best rallying cry for getting us aligned and talking (!) under one unified banner.

Cultural: For economic success, we need an innovative culture in our cities, communities, schools and organizations. We have really difficult challenges now--from our healthcare to the environment to under-educated kids, that require a new mindset of innovation that can lead to real breakthroughs and better solutions. This blog has been dedicated to offering tools for an innovative mindset and culture, which include openness to new ideas, diverse perspectives coming together in new combinations, and the fostering of the "4Cs" of creativity, critical thinking, communication and collaboration.

Personal: We live in a world without long-term job security, and that means we have to be more creative individually, able to learn constantly as adults and re-invent ourselves as needed. The world is asking us to truly leverage our unique talents in ways that provide value to others, and to do that we have to be aware of and build our competencies of creativity, which I've described throughout this blog. Perhaps the most important is flexibility--which is our ability to see things differently, seek out new perspectives, challenge our assumptions and embrace change.

Despite our current malaise, the United States is still a young country and a small shift of mindset, perhaps an inner rather than outer revolution, could lead to needed change much quicker than we think. What do you think? Can we get people to rally behind innovation? What else can unify us? Here's to a 2011 where more Americans embrace their own creativity and where we're more of a united state of innovation and collaboration.

See Who is Attending the San Francisco Reception

We have a Who's Who of business attending The Global Leaders Reception in San Francisco on Thursday, January 13, 2011.

See a list of attendees and their sphere of influence at our website.  See who they are, what they have done, who big their companies are, who their customers, suppliers, and competitors are.  And see relative performance.

To down load the 150 page plus report, go to:

https://www.tgleaders.com/group/san-francisco-0

To attend the reception and be listed with these leaders, go to:

http://the-global-leader.eventbrite.com/?ref=ecount

As an indication, below is a list of the top 50 companies by market capitalization.

Company Name
General Electric Co. (NYSE:GE)
GlaxoSmithKline plc (LSE:GSK)
The Goldman Sachs Group, Inc. (NYSE:GS)
Amazon.com Inc. (NasdaqGS:AMZN)
Morgan Stanley (NYSE:MS)
Medtronic, Inc. (NYSE:MDT)
MedcoHealth Solutions Inc. (NYSE:MHS)
Merck KGaA (DB:MRK)
LG Display Co., Ltd. (KOSE:A034220)
Mylan, Inc. (NasdaqGS:MYL)
Vertex Pharmaceuticals Incorporated (NasdaqGS:VRTX)
International Game Technology (NYSE:IGT)
Nasdaq OMX Group Inc. (NasdaqGS:NDAQ)
Dun & Bradstreet Corp. (NYSE:DNB)
Elan Corp. plc (NYSE:ELN)
Pharmaceutical Product Development Inc. (NasdaqGS:PPDI)
Accenture plc (NYSE:ACN)
InterMune Inc. (NasdaqGS:ITMN)
Impax Laboratories Inc. (NasdaqGS:IPXL)
Fauji Fertilizer Co. Ltd. (KASE:FFC)
Questcor Pharmaceuticals, Inc. (NasdaqGS:QCOR)
PDL BioPharma, Inc. (NasdaqGS:PDLI)
Micromet, Inc. (NasdaqGS:MITI)
Ariad Pharmaceuticals Inc. (NasdaqGM:ARIA)
Paladin Labs Inc. (TSX:PLB)
Geron Corporation (NasdaqGS:GERN)
3i Group plc (LSE:III)
Novartis India Ltd. (BSE:500672)
BMP Sunstone Corporation (NasdaqGS:BJGP)
Fauji Fertilizer Bin Qasim Limited (KASE:FFBL)
Bacterin International Holdings, Inc. (OTCBB:BIHI)
Durect Corp. (NasdaqGM:DRRX)
Capstone Turbine Corp. (NasdaqGM:CPST)
Synta Pharmaceuticals Corp. (NasdaqGM:SNTA)
Arena Pharmaceuticals, Inc. (NasdaqGS:ARNA)
Evolva Holding SA. (SWX:EVE)
4SC AG (XTRA:VSC)
Anacor Pharmaceuticals, Inc. (NasdaqGM:ANAC)
Enzo Biochem Inc. (NYSE:ENZ)
Vital Images Inc. (NasdaqGS:VTAL)
Solta Medical, Inc. (NasdaqGS:SLTM)
Landec Corp. (NasdaqGS:LNDC)
SuperGen Inc. (NasdaqGS:SUPG)
Curis Inc. (NasdaqGM:CRIS)
Pharmasset, Inc. (NasdaqGS:VRUS)
Æterna Zentaris Inc. (TSX:AEZ)
American DG Energy, Inc. (AMEX:ADGE)
CardioNet, Inc. (NasdaqGM:BEAT)
Amicus Therapeutics, Inc. (NasdaqGM:FOLD)