Replicating from MySQL to Drizzle and Beyond

Drizzle is one of the really great pieces of technology to emerge from the MySQL diaspora--a lightweight, scalable, and pluggable database for web applications. I am therefore delighted that Marcus Erikkson has published a patch to Tungsten that allows replication from MySQL to Drizzle. He's also working on implementing Drizzle-to-Drizzle support, which will be very exciting.

Marcus has submitted the patch to us and I have reviewed the code. It's quite supportable, so I plan to integrate it as soon as we are done with our next Tungsten release, which will post around 5 November. You will be able to build and run it using our new community builds.

This brings up a question--what about replicating from MySQL to PostgreSQL? What about other databases? I get the PostgreSQL replication question fairly often but it may be a while before our in-house team can implement plug-in support for it. Anybody want to submit a patch in the meantime? Post in the Tungsten forums if you have ideas and need help to get the work done. Tungsten Replicator code is very modular and it is not hard to add new database support.

Meanwhile, go Marcus!!

Community Builds for Tungsten Clustering

It's been almost two months since I have posted anything on the Scale-Out Blog, as our entire team has been heads-down working on Tungsten. We now have a number of accomplishments that are worth writing articles about. Item one on that list is community builds for Tungsten clusters.

Tungsten community builds offer a bone-simple process to check out and build Tungsten clustering software. The result is a fully integrated package that includes replication, management, monitoring, and SQL routing. The community builds work for MySQL 5.0 and 5.1 and also allow you to set up basic replication from MySQL to Oracle.

Community builds do not include much logic for autonomic management, including automated failover and sophisticated rules that keep databases up and running rain or shine. Those and other features like floating IP address support are part of the commercial Tungsten software. PostgreSQL and Oracle-to-Oracle support is also commercial only at least for the time being.

Community builds do include our standard installation process, which allows you to set up a working cluster a few minutes. You can back up and restore datebases, check liveness of cluster members, failover master databases for maintenance and a lot of other handy features. There is also full documentation, located here.

To get started, you need a host running Mac OS X, Linux, or Solaris that meets the following prerequisites. On Linux you can usually satisfy these requirements using Yum or Apt-get if the required software is not already there.

• Java JDK 1.5 or higher.
• Ant 1.7.0 or higher for builds
• Subversion. We use version 1.6.1
• MySQL 5.0 or 5.1 (only on hosts where cluster is installed)
• Ruby 1.8.5 or greater (only on hosts where cluster is installed)
  

Now you can grab the software and do a build. Make a work directory, cd to it, and enter the following commands. (Due to truncation on the blog the SVN URL looks a little funny. Don't be fooled.)

svn checkout \
https://tungsten.svn.sourceforge.net/\
svnroot/tungsten/trunk/community

cd community
./release-community.sh    # (Press ENTER when prompted)

The release-community.sh script checks out most of the Tungsten code for you and does a build. IMPORTANT NOTE: The command shown above builds SVN HEAD, which means you may have a life of adventure. You can also build off branches which are more or less stable. Look at the available config files in the community directory.

After the build finishes, you have ready-to-install clustering software. You can scp the resulting tar.gz file out to another host or just cd directly into the build itself as shown below and run the configure script, which sets up Tungsten software on a single host.

cd build/tungsten-community-2009-1.2
./configure

You may need to read the manuals so you get all the answers right. The installation manual is posted here at www.continuent.com. You'll also need to look at the Replication Guide, Chapter 2 to see how to set up MySQL properly. We'll do that automatically in the future, but for now it's help yourself. (Don't worry: the database set-up is easy.)

To make the cluster interesting you should install on at least a couple of hosts. Here's what an installed cluster looks like using the Tungsten cluster control (cctrl) program.

[tungsten@centos5a tungsten-community-2009-1.2]$ tungsten-manager/bin/cctrl
[LOGICAL] /cluster/comm/> ls

COORDINATOR[centos5a:MANUAL]

ROUTERS:
+-----------------------------------------------------------------------+
|NONE                                                                   |
+-----------------------------------------------------------------------+

DATASOURCES:
+-----------------------------------------------------------------------+
|centos5a(master:ONLINE, progress=3)                                    |
+-----------------------------------------------------------------------+
|  REPLICATOR(role=master, state=ONLINE)                                |
|  DATASERVER(state=ONLINE)                                             |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
|centos5b(slave:ONLINE, progress=3, latency=0.0)                        |
+-----------------------------------------------------------------------+
|  REPLICATOR(role=slave, master=centos5a, state=ONLINE)                |
|  DATASERVER(state=ONLINE)                                             |
+-----------------------------------------------------------------------+

Starting from scratch and pulling code from SourceForge, it takes me about 30 minutes to get to an installed cluster with two nodes. At this point you have access to a very powerful set of tools to protect data, keep your databases available, and scale performance. Look at the manuals. Try it out. If you have questions or feedback, post them in the Tungsten forums. In the meantime, have fun with your database cluster.

p.s., We will post binary builds next week. The current build is in final release checks, so you may notice a few problems--I hit a Ruby warning on configuration that will be fixed shortly.

R&D And Innovation In A Downturn : A Misnomer?

I have always followed Booz Allen’s annual innovation study for the last several years and have written about it as well here, here etc.. This year’s study looks refreshing and asks the question how do smart companies spending good money &effort on innovation approach spending when the economy is down? The answer: Most of the leading innovators are actually accelerating their innovation efforts in the recession – in doing so they are actually betting that customers will demand new products and services as the economy gets out of the downturn. Many of the corporate decision makers in the list who control/manage budgets for innovation – running into hundreds of billion dollars of business every year believe that innovation is critical as they prepare for the upturn, and a majority have maintained or expanded their portfolios and are pursuing new products to improve growth and margins. Practically everyone wants to be as focused and result oriented as could be The recent trends, observed over the last few years suggest that as a rule, companies are performing less pure and applied research. Instead, they are concentrating their R&D budgets on product development and engineering. The software sector is now brimming with new strength on SaaS & Cloud centric products and services while behemoths like SAP struggling to maintain growth and leadership! Apple had the best quarters when the global economy was perhaps at its lowest ebb last few quarters!
I found it interesting to see an IBM executive’s statement suggesting that in a recession or downturn "...that really drives a need for innovation and a level of creativity that you might no otherwise have in normal times." I thought that the new normal is that normalcy is a product of bygone era! The reality is that last two decades have seen more changes in growth /slowdown and indications are that this may become the new normal – so downturn or upswing ought to marginally influence yearly spend on innovation – particularly when product development cycles can run into multiple years. If anything the sensitivity could prove itself in reverse direction of spending more/differently in a downturn, Read Applied material’s dilemma – the revenues are down significantly this year. Yet it’s customers continue to demand innovative new products to maintain their own competitive positions. The stronger companies want to stay on the same innovation page so that at the end of the cycle, they have a stronger competitive position. Taking a long view of economy, markets and relating to the context are done differently by different companies. The top three spenders on R&D in the world happen to be non-american companies! Matter of fact in many industries today, predicting the economy for the next few years and its implication for them may become the most coveted job and an art for its executive board. In fact, I would think that not being able to predict and adjust to an ever changing economic environment may become a disadvantage for any enterprise – being able to constantly test the assumptions about economic environment may have to become a top item for corporate strategy planning, adjusting internally and to succeed in the marketplace.

Lets look at how this would impact executives inside the organization. Business needs to look at talent horizontally across the business classified as per the roles and not necessarily as per the organizational norms and this would be the key in effectively harnessing talent inside enterprises. The differences ought to be expressed in terms of talent valuation - that is, such attributes as knowledge, experience, skills, and personal interaction capabilities - and not in terms of organizational structures (such as business units) or in human resources management terms (such as age, education, seniority, or compensation). In an age where the rules and procedures of an organization can be an obstacle to segmentation and a force for “averaging” the treatment of individuals’ roles, and in a situation where organizations need to offer very specialized services, definitely a radical new look at the way talent gets categorized, nurtured and reviewed are called for and on an ongoing basis, needs to be reviewed to provide a definitively fresh and dynamic approach. After all winning the people war is a crucial determinant of success for any organization. In most of the knowledge business, the future values of enterprises are centered on building seemingly intangible assets vs the conventional measures of capital assets. That’s where good, capable people, well aligned team, well conceived strategy and top quality leadership matters. With all these in place, the cutting edge would from innovation - in all its forms starting from management innovation to disruptive innovation to innovation of the incremental kind. The ability to recruit/ mould the leaders that will be able to create the future innovations that will make enterprise more successful is a major responsibility & talent management in many ways will determine the organization’s growth as much as the overall business strategies will. This will shape organizations in a significant way and if organizations get this right along with purposeful innovation models, competitiveness increases and success would follow.