If June and July were a Single Day...

I woke up early this morning intent on inspiring creativity outside and in, first by facilitating two days of innovation exercises at an educators conference at a farm-camp in mid-state New York, and then by walking dozens of miles through NYC and Philly, climbing the Statue of Liberty* and ringing the Liberty Bell, couple-clutching through Times Square and Washington Square Park, watching kids play baseball in Brooklyn and dance a recital in Upper Dublin, PA, navigating and negotiating my way through Canal St., Port Authority, deer crossings* and crossing the Delaware, mixing it up with family, old friends, rude people with East-coast accents, and my girl.

I got back in Chicago just in time for exploring leadership with new MBA students and seasoned risk managers and then hitting a homer over the left fielder's softball glove before I celebrated at a party featuring the largest pink penis pinata* ever seen. Then we ate and swayed with the rain and heat along with several bands at Chicago street festivals before heading north to Summerfest in Milwaukee, taking a view from the sky* before dancing dangerously on benches to Michael Franti (video below or here) and his frantic love vibe, and then as the heat souffled us we were better off leaving the beach where alewives had flooded the shore and drove back down to the Chicago lakefront for late night swimming and a thousand-person, fire-twirling full moon celebration.



By afternoon I shared the sorrow and joy of a death and a bris, took moped rides to interfaith program meetings, and watched fireworks lighting the sky. I officially finished my own musical script, packed my car with musical instruments and chairs, held auditions* and am now ready for our first Malaise County Fair rehearsal, taking the next step with my most personal creative project, guitar polished and ready to strum into the unknown.

As evening began to roll around, I facilitated an "idea jam"* for a super-smart group of engineers whose buildings can swivel, and this past weekend I swiveled my way out to Ravinia to listen to Five for Fighting under the stars and then took in the ultimate theater-on-steroids show--30 "plays" in 60 minutes by the Neo-Futurists. Now I sit at a cafe, typing this, reminded that there is no better place to spend the summer than Chicago (with a little travel thrown in) and wondering what is in store for the remainder of these simmering months.

Mobile Internet Access in Germany for Open Source Road Warriors

Reliable Internet access is a long-standing problem for road warriors visiting foreign countries.  Open source developers in particular have problems reconciling travel with addiction to high-bandwidth network access from laptop computers.  Wi-Fi hotspots are scarce, costly, often slow, and in some cases complicated by inconvenient local laws like Italy's Pisanu Decree.  International mobile network access plans are ridiculously expensive or like DROAM have download limits that make them useless for serious programming.

The best solution in many cases is to look for a local pre-paid mobile access plan in each country you visit.   Mobile networks are widely available and fast in developed regions, and there are cheap plans that limit the amount you pay while providing solid connectivity.  On a recent trip to Germany I found a great solution for local Internet access from FONIC, which offers a prepaid data plan using their FONIC Surf-Stick.  The Surf-Stick is a USB modem that plugs into a USB port on your laptop and looks like the following. 

After you buy the Surf Stick, you can add money to it using credits purchased in local stores.  It's a relatively simple solution provided you understand a little about networking and can work your way through a bit of German.

The rest of this article is a description of how to use FONIC in Germany, as well as a review of the performance and a couple of downsides.  I do not have any connection with FONIC other than using their products.  You may have different experiences or find something better.  If so, please write an article about it.

Getting Started

I bought got my Surf Stick for 49 Euro at Saturn, a big German consumer electronics chain.  You get the modem itself plus a SIM card and one day of free surfing.  You can get also the stick from other FONIC partners or order it off the FONIC website.  Surprisingly it does not appear to be available in places like Frankfurt or Munich Airport.

The first step once you have the modem is to initialize the SIM on FONIC's site by registering yourself as a new user.  To do this you'll need to have your FONIC phone number and activation code ("Freischaltungsnummer"), which are written on the side of the envelope that contains the SIM and that you should try to avoid losing.  You'll also need a German address that you enter as part of the sign-up.  A hotel address is fine.  It takes a couple of hours after registration to complete provisioning.  

Once provisioned, connecting is easy.  Put the SIM in the surf stick and plug the stick in a USB port on your computer.  On Mac OS X, you'll see an application called "Mobile Partner."  Here are the steps to activate a connection: 

1. Click the Mobile Partner app.  
2. Enter your PIN (also on the SIM envelope) and press Return. 
3. The application will stall for a few seconds while it looks for the modem.  Once complete it will show a screen entitled "Verbinden" (connect).   
4. Click Verbinden.  A couple of pop-ups will appear and you are online. 
5. When you want to drop the connection click "Trennen" (disconnect).  

You will now need to load some money into your account.  There's no way to top up online with a credit card--you either have to register a German bank account or purchase credits.  For foreigners the only practical thing is therefore to buy a credit, which is called an "Aufladebon."  You can get them at stores like Lidl, Rossman, and Jet as well as Esso gas stations.  Just ask for the following in German:  FONIC Aufladebon für 20 Euro, which is a 20 Euro credit and gives you 8 days of surfing.   I recommend buying this at the same store where you get your modem, which means the start-up cost is about 70 Euros.  

The Aufladebon itself a bit dissapointing:  a printed receipt with a 12 digit number and some instructions in German.  Start your Mobile Partner app, connect to the Internet, and click on "Guthaben verwalten" (manage credit).  Select "Guthabenkarte aufladen" (load credit), enter your 12 digit number, and press enter.   Your credit is now topped up for a while.  You can check the level using the "Guthabenabfrage" (check credit) button.  

Performance

I used the FONIC modem for about a week.  It functioned well in every location that had at all reasonable connectivity.  FONIC handles transitions across cells really well--provided there's connectivity it seems to use the fastest available protocol, such as HSPA, and switches seamlessly down to lower capacity protocols like Edge if that is all that is available.  You can use it in a car or train without experiencing application problems--I used it on the Autobahn between Hamburg and Lübeck and it worked surprisingly well.   (Just be clear, somebody else was driving at the time.) 

With good connectivity, the rated speed is 7.2Mbps down and 5.76 upload, which is comparable to average household connections.  PDFs and similar files downloaded at 150K/sec.  Upload was similarly snappy.  

Applications like ssh and even remote debugging/profiling worked very well.  In my case this included running Yourkit Java profiler connecting into an application running on servers elsewhere in Germany.  I also hosted GotoMeeting web conferences.  The latency seems higher than DSL--typical latencies run about 100 to 200ms, which is about the same latency you get when connecting from the US West Coast to sites in Europe. 

Daily transfer limits are very competitive.  The current limit is 500Mb per day at full speed access, though in practice it seemed to cut off around 600Mb for me.  This is better than offerings from the big guys like T-Mobile and Vodafone.  FONIC implements the limit quite elegantly.  First you get a nice SMS that you are near the limit, followed by another that you are over the limit.  However, after that you do not get cut off.  The connection just switches to 64kbps until the next day.  This is still faster than the Wi-Fi connectivity in many hotel rooms.

If you don't want to wait for the friendly SMS message about limits, you can check your limits easily on using the Mobile Partner app.

Speaking of the app, Mobile Partner seems really solid.  There were no bugs, crashes or weird hangs the way you get with some of the US providers like Verizon's prepaid Internet app. I only ran it on Mac OS X but I would guess it is just as good on Windows.  

Caveats

In general FONIC is outstanding.   Life is not perfect, so here are a few of the limitations I found.   

1. German.  FONIC is 100% German, including the website and the app.  You'll need to know some German or make friends fast.  Also, the data plan apparently only works in Germany.  
2. Phone connectivity.  If your cell phone does not work, FONIC won't either.  The nice thing is that you can have connectivity drop out for a bit while crossing cells in a train or car without losing TCP/IP sessions.  
3. Occasional routing problems. On one evening in one particular location the FONIC modem did not work at all well, reporting ping latencies of 50+ seconds or dropping out entirely.  It was not apparent whether this was due to poor cell coverage or another issue.  It worked fine everywhere else.

There appear to be some competitors to FONIC but I have no idea how well they work.  One of the advantages of FONIC is that they have many partners in Germany that sell both modems as well as the credits.  That would also be a consideration in evaluating competing products.

Summary

Some day it may be possible to get decent mobile data plans that work for international travel.  Given the complexity of telecom regulation and the motivations of local carriers to prevent competition, you probably don't want to hold your breath until they arrive.

In the meantime, FONIC looks like a good Internet access solution for anybody traveling to Germany for more than a few days who needs constant, high-capacity access.  I was stuck in a couple of locations that had no network access, so it was a life-saver in those cases.  However, FONIC is also cheaper than hotel Wi-Fi and has better performance than many Wi-Fi hotspots.   If you move about or uncertain about the quality of the Internet connectivity where you are staying, I highly recommend it.

If you know of similar plans for other countries, please share your experiences.  I would love to find something like FONIC for Italy, Spain, and the US.  

The Incredible Apple

Yesterday, Apple did the unthinkable, yet again! Apple blew away the revenue and earnings by a long shot. Apple announced record quarterly sales of over $28.57 billion, and profits of $7.31 billion ( Apple, what are you planning to do with all that cash? Perhaps you can buy out a country or two! )

Apple sold 20.34 million iPhones in the quarter, representing 142 percent unit growth over the year-ago quarter. Apple sold 9.25 million iPads during the quarter, a 183 percent unit increase over the year-ago quarter. The Company sold 3.95 million Macs during the quarter, a 14 percent unit increase over the year-ago quarter. Apple sold 7.54 million iPods, a 20 percent unit decline from the year-ago quarter.

“We’re thrilled to deliver our best quarter ever, with revenue up 82 percent and profits up 125 percent,” said Steve Jobs, Apple’s CEO. “Right now, we’re very focused and excited about bringing iOS 5 and iCloud to our users this fall.” (This means, iPhone 5 is coming this Fall as well!)

Apple Destroys Estimates AND the Stock Is Still Cheap! Here's what stood out:
* Apple's Cash on Hand grew by 16% to $76.2 billion. The number works out to roughly $81 per share. Were they so inclined, Apple could write checks for Netflix (NFLX) and Ebay (EBAY) combined and still remain liquid.
* Apple's revenue growth is actually accelerating.
* The company beat EPS estimates by over 30% and revenue forecasts by 15%. These numbers speak ill of the people analyzing Apple but are jaw-dropping nonetheless.
* The stock continues to perform yet somehow gets cheaper; trading at over 18x earnings yesterday and in the vicinity of a 15 P/E today.

Cramer on Wednesday raised his price target on Apple from $400 to $500 a share. Yet at the $500 level, the technology stock will be selling at just 11.5 times his estimates for next fiscal year's earnings, only three quarters of what the average company in the S&P 500 index sells for.
To Cramer, Apple is a great example of why capitalism is worth cheering for. Unfortunately, Apple also puts U.S. leaders to shame because all they can do is bicker. "The United States of Apple?" Cramer said, "Now that's a place Cramericans can call home."

Apple Earnings Call: "We are very excited to be launching Mac OS X Lion tomorrow. The Lion includes more than 250 new features such as a multi-touch gestures, system wide support for full screen apps, Mission Control, which is an innovative view of everything running on your Mac and much more."
"iPhone continues to be adopted as the standard across the enterprise with 91% of the Fortune 500 deployed or testing the device, up from 88% last quarter. We're also seeing great growth in scale worldwide. Today, 57% of Global 500 companies are testing or deploying iPhone, fueled by strong employee demand and opportunities for custom App development. Some examples of Global 500 companies supporting iPhones on their corporate networks, include AXA, Credit Agricole, Nestlé, Dow Chemical, GlaxoSmithKline, SUPERVALU and Comcast."
"We are also seeing strong adoption internationally with 47% of Global 500 companies testing or deploying iPad. In the 15 months since iPad has shipped, we've seen iPads used in the enterprise in ways we could have never imagined. Companies like Boston Scientific, Xerox and Salesforce.com are deploying thousands of iPads and revolutionize in how their sales teams engage their customers. iPad is being used inside the country's top hospitals like HCA and Cedars-Sinai and in retail at Nordstrom and at Esteé Lauder's Clinique counters. General Electric, SAP and Standard Charter have developed internal Apps for training, currency tracking and business process management to help make employees even more productive. And Alaska Airlines and American Airlines are using the iPad in cockpit to replace paper-based navigational and reference information pilots carry with him on every flight."
- Peter Oppenheimer, Apple CFO
"China was very key to our results. As a reminder, we define Greater China as Mainland China, Hong Kong and Taiwan. Year-over-year, it was up over 6x and the revenue was approximately $3.8 billion during the quarter and that makes the year-to-date numbers through the 3 quarters that we've had thus far around $8.8 billion. And so this has been a substantial opportunity for Apple and I firmly believe that we're just scratching the surface right now. I think there is an incredible opportunity for Apple there." - Timothy Cook, Apple COO

Bottomline:
Apple dropped a huge sales bomb again! Apple's quarterly revenue and earnings explosion is just the beginning! Expect Apple to deliver successive quarters of record sales and earnings in the foreseeable future. About the only thing that can bring Apple down is supplier constraints. But Apple has planned well in this department as well; Apple's inventory is healthy, and current generation products are getting rolled out without any delays. As schools resume in Fall, and as we get closer to the holiday season, Apple revenue machine will accelerate faster! A $40 billion holiday quarter is indeed a possibility... Apple makes more money internationally than in the U.S.; 62% of sales came from international operations in the latest quarter. This means that Apple has become a true global company, and has even less dependency on local market going forward. It also means that the world has more buying power and buys more Apple products thatn U.S. does. This is both a challenge and opportunity for Apple going forward. Apple now has to design products that cater to international buyers. And assuming that the world is still warming up to Apple, the global demand for future Apple products will be very high. As the U.S. economy improves, Apple will derive even more sales locally, giving its quarterly revenue numbers bigger boost! Apple is a winner no matter what!
When a billion cell phones are sold every year worldwide, if Apple is able to increase its marketshare to 10%, this would translate to 100 million iPhone annual sales. Apple just sold over 20 million iPhones. Expect a 25 million iPhone quarter in the near future. As for the iPad, there are many choice tablets available. However, at $499, iPad 2 offers an amazing mix of features and functionality in one incredible machine. Can Apple sell 15 million iPads next quarter? Quite possibly. Apple just announced the latest MacBook Air at starting price of $999. It's faster, ships with Mac OS X Lion, has Flash storage, backlit keyboard, camera and most everything you need as your entry notebook. This will help accelerate the sales of MacBooks in the coming quarters. Finally, the all important iOS 5 and iCloud are coming this year... along with the launch of iPhone 5. Apple has the whole world in awe with its product wizardry! The Incredible Magic of Apple!

How does Apple, the #1 innovative company in the world, innovate and create game-changing innovations such as the iPod, iTunes, iPhone, iPad and more? What is Apple's secret recipe for innovation success?


Download Apple's Innovation Strategy, and learn how Apple became the #1 innovator through:

• Creativity and Innovation
• Innovation in Products
• Innovation in Business Model
• Innovation in Customer Experience
• Innovation and Leadership
• Steve Jobs Visionary Leadership
Revised in 2011! Steve Jobs interview

Learn more...

Introducing Tungsten On-Disk Queues for Parallel Replication

Tungsten Replicator has offered shard-based parallel replication to slaves since late 2010.  The initial implementation uses in-memory queues.  Working purely in memory keeps latency low and throughput high.   On the other hand, working in memory consumes valuable RAM.  It also forces us to buffer all in-flight transactions and therefore greatly limits the span of time permissible between the slowest and fastest shard.

Hence our newest improvement:  on-disk parallel queues.  In this article I will cover how parallel replication works in general, how on-disk queues help with parallel replication, and then show how to set up from the latest builds.

First, let's review the basic mechanics.  Parallel replication is really "parallel apply," which means taking a stream of serialized transactions, splitting them into separate streams, and applying them in parallel to a slave DBMS.  In the Tungsten pipeline architecture, we implement this kind of flow using a combination of stages and stores.  One stage reads transactions from the persistent transaction history log (aka THL) to a "parallel store."  The parallel store splits the stream into a set of queues.  The next stage extracts from those queues and applies to the slave.   It looks like the following picture:

From a conceptual point of view the incoming thl-to-q task thread performs an indexing function.  It  guides construction of the queues read by task threads in the q-to-dbms stage.  Within this framework there are many ways to feed events into the parallel queues.    In the case of on-disk queues there are two obvious design options.  

1. Read data out of the THL and split them into separate transaction logs per parallel queue.  This is very similar to the in-memory approach, except that the queues are now on disk (or SSD or whatever storage you pick).  It can be implemented without adding any extra threads to the parallel store. 
2. Leave all data in THL.  Add a cursor for each parallel queue that scans the THL and picks only the transactions that belong in that parallel queue.   This requires extra threads to do the scans, hence is more complex to implement.  

Both approaches achieve the primary goal, which is to keep the transactions in storage until we actually need them and thereby minimize memory usage.  This in turn solves a major problem, namely that individual shards can now be many thousands or even millions of transactions apart in the serial history.  Beyond that, it is not completely obvious which approach is better.

For example, option 1 isolates the reads to individual files.  This minimizes overall I/O at the cost of making it more random, since reads and writes are spread over many files.  Option 2 avoids extra writes and keeps I/O sequential, but introduces a bunch of threads doing the equivalent of table scans across the same patch of storage.  Up to a point we can assume that pages are coming out of the OS page cache rather than storage but this assumption will not hold for all operating environments and workloads.  The only way to prove the trade-offs is to implement and test.  (We may end up implementing both.)

After some discussion internally at Continuent as well as with Harrison Fisk from Facebook, we picked option 2 for the initial implementation.  Here is a diagram that shows how it works.

Here is a quick tour of the implementation in Java class THLParallelQueue.  This class maintains an in-memory blocking queue for each channel.  Each queue has a corresponding read thread that scans the THL and places matching events into the queue.  The THLParallelQueue class synchronizes read threads and handles issues like serialization and clean shutdown.  Some memory is therefore consumed, for queues, but they are quite small and amount to far less than keeping all transactions in memory.

So much for the theoretical description. If you would like to test on-disk queues yourself, you can get started in three steps.

1. Download the latest nightly build of Tungsten Replicator to /tmp. 
2. Untar and cd into the resulting release directory.  
3. Install using the new tungsten-installer as described by Guiseppe Maxia.  

Here is an example of set-up commands.  My test system uses logos1 as the master and logos2 as the slave.  The MySQL DBMS build is Percona Server 5.1.54.  Tungsten is installed in /opt/tungsten.

# Download and unpack build.
cd /tmp
wget --no-check-certificate https://s3.amazonaws.com/files.continuent.com/builds/nightly/tungsten-2.0-snapshots/tungsten-replicator-2.0.4-154.tar.gz
# Untar.
tar -xvzf tungsten-replicator-2.0.4-154.tar.gz
cd tungsten-replicator-2.0.4-154
# Set up and start replicators. 
export TUNGSTEN_HOME=/opt/tungsten
/tmp/tungsten-replicator-2.0.4-154/tools/tungsten-installer \
  --master-slave  \
  --master-host=logos1  \
  --datasource-user=tungsten  \
  --datasource-password=secret  \
  --service-name=percona \
  --home-directory=${TUNGSTEN_HOME} \
  --cluster-hosts=logos1,logos2 \
  --relay-directory=${TUNGSTEN_HOME}/relay-logs \
  --datasource-log-directory=/usr/local/percona-5.1.54/data \
  --thl-directory=${TUNGSTEN_HOME}/thl-logs \
  --channels=10 \
  --svc-parallelization-type=disk \
  --start-and-report                                  

Note the bold options to select disk queues--"memory" is the other option--and the number of channels.  You can confirm you have the right queue installed by running the following command against any slave replicator.  You should see the storage class THLParallelQueue in the status output.

$ trepctl -host logos2 status -name stores
Processing status command (stores)...
NAME                VALUE
----                -----
doChecksum        : false
logDir            : /opt/rhodges4/thl-logs/percona
logFileRetention  : 7d
logFileSize       : 100000000
maximumStoredSeqNo: 0
minimumStoredSeqNo: 0
name              : thl
storeClass        : com.continuent.tungsten.replicator.thl.THL
NAME                VALUE
----                -----
criticalPartition : -1
discardCount      : 0
eventCount        : 1
headSeqno         : 0
maxSize           : 10
name              : parallel-queue
queues            : 10
serializationCount: 0
serialized        : false
stopRequested     : false
store.0           : THLParallelReadTask task_id=0 thread_name=store-thl-0 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.1           : THLParallelReadTask task_id=1 thread_name=store-thl-1 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.2           : THLParallelReadTask task_id=2 thread_name=store-thl-2 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.3           : THLParallelReadTask task_id=3 thread_name=store-thl-3 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.4           : THLParallelReadTask task_id=4 thread_name=store-thl-4 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.5           : THLParallelReadTask task_id=5 thread_name=store-thl-5 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.6           : THLParallelReadTask task_id=6 thread_name=store-thl-6 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.7           : THLParallelReadTask task_id=7 thread_name=store-thl-7 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.8           : THLParallelReadTask task_id=8 thread_name=store-thl-8 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.9           : THLParallelReadTask task_id=9 thread_name=store-thl-9 hi_seqno=0 lo_seqno=0 read=1 discarded=0 events=0
storeClass        : com.continuent.tungsten.replicator.thl.THLParallelQueue
syncEnabled       : true
syncInterval      : 2000
Finished status command (stores)...

Now for some fine print.  On-disk queues are implemented but are still undergoing QA.  There are bugs.  The most important problem is performance--the latency is a lot higher than expected on some of our systems, which I suspect is due to an as-yet undiagnosed bug.  If you try them out now you can expect to hit a few problems.  On the other hand, we take any and all feedback quite seriously, so this is your chance provide input and help guide the final implementation.  Please log issues on the Tungsten Replicator issue tracker or bring up questions on the tungsten-discuss mailing list.

Finally, if you would like to learn more about the parallel queue implementation, check out the design documentation on our wiki as well as the source code.  They are both pretty readable.