If you have a smartphone, you must be using LINE, KakaoTalk, or Whatsapp messaging app, if not all of them. But unlike KakaoTalk and Whatsapp, LINE also allows users to call other LINE users for free - great value-added service, isn't it!? Today I will explain more about LINE, its background story as well as how its developers use NoSQL together with an in-memory data storage to manage billions of user messages per month!

Behind Story

So, you already know about the first advantage of LINE - free calling service. But there is another one which gives even more convenience for users - it is available not only on leading mobile platforms (iOS and Android), but also on Mac OS X and Windows desktop operating systems. Now in order to send a message to your friend, you no longer need to take out your mobile phone and type on its tiny keyboard. You can use the desktop LINE to make the conversation fast and more convenient! You type on your computer, your friends and family read the message on their mobile or desktop app. Staying in touch for FREE has never been easier. I am sold!

Now the main question: how LINE app was created?

It may sound somewhat bizarre but it is all about the 2011 Japan Earthquake. You all know how tragic it was. Communication was disrupted, major damage was caused to many buildings, roads and transportation systems. In such situations cell phones still work but service is sketchy. Data works better than voice. This was the impetus to my colleagues at NHN Japan to design an app accessible on smartphones, tablet, and PC, which would work on data network and provide continuous and free instant messaging and calling service.

The name LINE was originated from the fact that after the incident people had to line up outside of public phones, because in Japan the public phones "are programmed to take priority over networks during and after an earthquake".

Later in June 2011 LINE app was launched.

LINE Today

Today LINE is one of the most popular beautifully designed instant messaging and calling app available for iOS, Android, tablet, and desktop users. It took only nine months for LINE to overcome 30 million registered users milestone. At the time of writing this article LINE has already 36 million users worldwide and is recognized as being a "Fast and Light" messenger that is considered as the "The Number 1 Free App" in many countries, especially in South East Asia.

LINE app allows group chatting with up to 100 people at once, sending photo video files as well as location info, making free voice calls both on Wi-Fi and cellular network. But one of the greatest values in LINE is its 250+ expressive sticker and emotions collection which make chatting even more exciting and fun.

Database Storage for Instant Messaging App

With increasing number of LINE users, the number of chat messages to store grows exponentially (see the graph below). These days LINE needs to store billions of rows in a database every month. However, back in early development stage LINE developers had anticipated the service load at most from only 1 million registered users. With such assumptions the developers had deployed Redis, an in-memory key-value store, which provides synchronous and asynchronous replication and enables DB administrators to take periodical disk snapshots.

Since Redis does not provide server-side sharding, LINE developers had to implement sharding on the client side and manage it with ZooKeeper. Thus, the entire LINE storage stack initially was based on single Redis cluster consisting of 3 nodes sharded on the application layer. With the growing service, more nodes were added to the cluster.

However, everything was changed around October 2011 when LINE started to experience extreme load due to unexpected growth worldwide (see above graphs). This have caused some outages, and delay in message delivery. The problem was in the core of Redis, in the fact that it is an in-memory store, which, unlike persistent storage systems, requires more servers. Managing LINE's tremendous growth with Redis would cost an arm and a leg. It was critical to find an alternative solution which would provide high scalability and availability at a relatively less cost.

To solve this problem, LINE developers had first identified the types of data they have to store depending on how much availability and scalability were important for each type. As a result they came to a conclusion that:

1. for user messages in delivery queue availability is critical;
2. for user profile info as well as contacts and groups both scalability and availability are important;
3. for messages in the Inbox as well as change-sets of (2) due to its massive size scalability is vital.

To achieve availability LINE developers chose to stick with Redis in-memory data store. For scalability they had three options: HBase, Cassandra, and MongoDB.

• HBase
• pros:
• Best matches the requirements
• Easy to operate (Storage system built on DFS, multiple ad hoc partitions per server)
• cons:
• Random read and deletion are somewhat slow.
• Slightly lower avaiability (there’re some SPOF)
• Cassandra
• pros:
• Also suitable for dealing with the latest workload
• High Availability (decentralized architecture, rack/DC-aware replication)
• cons:
• High operation costs due to weak consistency
• Counter increments are expected to be slightly slower.
• MongoDB
• pros:
• Auto sharding, auto failover
• A rich range of operations (but LINE storage doesn’t require most of them)
• cons:
• NOT suitable for the timeline workload (B-tree indexing)
• Ineffective disk and network utilization

Finally out of three, HBase became the successor as a primary storage for the third type of their user data on top of HDFS. Sharded Redis is still running as a front-end cache for data stored in HBase and as a primary storage for user profile, contacts and groups. Below is the current LINE storage stack architecture.

LINE storage stack: over 600 nodes and growing.

LINE developers at NHN Japan have posted a very detailed blog article where they explain how everything is designed, how they had migrated from original Redis based architecture to HBase based architecture. You are highly encouraged to read this article if you would like to learn more about how they have installed and configured separate components of their storage stack.

These days many websites (in fact, millions) are implemented by the use of popular open source content management systems (CMS). I would say, for everything you would imagine your site doing, there is one or another CMS which can do that. Thus, the reason for choosing a particular one depends on the tasks you want to accomplish. For blogs there is a common sense of using WordPress blog management system, for large scale websites Drupal CMS is perceived to be a better fit, while Joomla CMS is easier to learn for newbies. .NET developers prefer alternative DonNetNuke or umbraco CMS. Besides these, CMS Made Simple and Liferay are amongst the most popular content management systems based on the number of the number of downloads. Elgg and MODx are amongst the rising stars.

I will not tell you what a CMS is. If you want to learn, see the CMS Wikipedia page. In this blog I will give a comprehensive overview of top 14 content management systems based on the number of weekly downloads, installations, and brand familiarity. The full statistical data can be found in the 2010 OPEN SOURCE CMS MARKET SHARE REPORT by water&stone (2010). The report is distributed under the Creative Commons Attribution-Noncommercial License (3.0).

Today we will cover Drupal, Joomla, MODx, WordPress, DotNetNuke, umbraco, Liferay, TYPO3, CMS Made Simple, MOVABLE TYPE, Plone, eZ Publish, concrete5, and Alfresco content management systems. You might already be familiar with some of these, while some can be new to you. I bet you will be surprised to learn that some unknown for you systems suit your requirements much better than those you have got to use so far (my personal experience). So, let's learn about each of them, and do not forget to leave your comments below on what other CMS should have been included in this list and why.

Note: the order of the CMS does not represent its quality, performance, or superiority over the other. They are numbered to give a better perception to readers. The users feedback and opinions are retrieved from Stack Overflow, thus can be subjective as they are based on personal views and preferences.

# 1: Drupal

Official site: http://drupal.org/

1% of all websites on the Internet are based on this platform. An estimated 7.2 million sites were powered by Drupal as of July 2010. Drupal started out in 2001. In one year from May 2007 to April 2008, Drupal was downloaded from their official site more than 1.4 million times, an increase of approximately 125% from the previous year.

Weekly downloads: 33,671 (ranked #3 after Joomla and before DotNetNuke).

Installations: 575 according to the survey (#3 after WordPres and before DotNetNuke), but 1.4% (#3 after Joomla and before Typo3) of Alexa Top 1 million sites.

Brand Familiarity: Drupal is known to be the #3 most familiar content management system.

Major Features:

• You can manage multiple sites with Drupal in multiple languages. You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine.
• You can easily manage your site users, providing standard registration, including OpenID support. You can set various access control rules to limit the activity of your site users.
• Provides multiple-level menu system, template customization, advanced search, RSS feed aggregator.
• Officially Drupal supports several databases including MySQL, PostgreSQL, MariaDB, and SQLite.
• To increase its performance, you can use caching. At the same time it provides high security with notifications about the new update releases.
• Provides Search Engine Friendly descriptive URLs.
• Powered by jQuery JavaScript framework.

Distinguished Clients:

• http://www.whitehouse.gov/
• http://data.gov.uk/
• http://www.ubuntu.com/
• http://www.alrc.gov.au/
• http://www.nysenate.gov/
• http://london.gov.uk/
• http://www.rutgers.edu/
• http://www.economist.com/
• http://wfp.org/

• More difficult to master especially for newbies. It's more for advanced users. Though the new Drupal 7 claims to provide significantly improved usability (maybe more toward WordPress style). To achieve this, they hired web designers to specifically address the UX problems it had in previous versions.
• New major releases are not quite backward compatible. More focus on new features and functionality. External module developers should take care of compatibility themselves, except for data representation, which Drupal is intended to keep same.
• Drupal is often compared with Joomla and is perceived to be a bit slower. Though this depends on the type of website you develop. However, leveraging its caching and gzipping technologies, it is possible to achieve quite impressive results.
• jQuery, the default JavaScript framework in Drupal, allows to use alternative Mootools framework at the same time.

# 2: Joomla

Official site: http://www.joomla.org/

Joomla is the result of a fork of Mambo CMS on August 17, 2005. Within its first year of release, Joomla had been downloaded 2.5 million times.

Weekly downloads: 113,836 (ranked #2 after WordPress and before Drupal).

Installations: 1,297 according to the survey (#1 before WordPress), but 2.5% (#2 after WordPress and before Drupal) of Alexa Top 1 million sites.

Brand Familiarity: Joomla is known to be the #1 most familiar content management system.

Major Features:

• You can manage multiple site with Joomla in multiple languages natively (since Joomla 1.6). You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine.
• You can easily manage your site users, providing standard registration, including Google OpenID support. Full support for Access Control List.
• Provides Multiple-level menu and content category system, template customization, advanced search, RSS feed aggregator.
• Officially supports only MySQL.
• Page cashing for increased performance.
• Provides moderate descriptive URLs (still not fully customizable as you can do in WordPress).
• Powered by MooTools JavaScript framework.

Distinguished Clients:

• http://www.linux.com/
• http://www.itwire.com/
• http://www.quizilla.com
• http://www.ihop.com
• http://gsas.harvard.edu
• Citibank (Financial institution intranet) - Not publicly accessible
• http://www.greenmaven.com
• http://www.outdoorphotographer.com
• http://www.playshakespeare.com
• http://www.sensointeriors.co.za

• More intuitive and easy to use than Drupal, though still not like WordPress.
• Powerful. Fully-fledged content management system, so you can create whatever site you want.
• Really strong security. If security problems found, immediately fixed.
• The new Joomla 1.6 release is expected to be faster, more convenient, with more features.
• Its strong dependency on Mootools JavaScript framework sometimes bothers users as Joomla does not give easy workaround to disable it and use jQuery instead.
• Support of only one database, limits Joomla a lot in terms of the number of users. However, this is a compromise for high optimizations for MySQL, thus increased overall performance.
• Does not allow to fully customize URLs - a must feature for CMS.

#3: modx

Official site: http://www.modxcms.com/

modx is not just an open source CMS but also a web application framework. Raymond Irving and Ryan Thrash began the MODx CMS project in 2004 as a fork of Etomite. In 2008 MODx users created a new logo and branding for the project. Now MODx allows for full segregation of content (plain HTML), appearance and behavior (standards compliant CSS and JavaScript) and logic (PHP, snippets).

Weekly downloads: 4,500 (ranked #11 after umbraco and before Tiki).

Installations: 58 according to the survey (#12 after eZ Publish and before umbraco), has less than 0.1% among the Alexa Top 1 million sites.

Brand Familiarity: #14 (before Liferay and after eZ Publish).

Major Features:

• As with Joomla, modx officially supports only MySQL database.
• Not just CMS but a PHP framework for Web.
• Freedom to choose jQuery, Mootools, ExtJS, Prototype or any other JavaScript library.
• Supports PHP 4.3.11 and above.
• Complete control of all metadata and URL structure for SEO (Search Engine Optimization).
• Unlimited hierarchical page depth.
• Can create custom fields and widgets for templates.
• Role-based permissions for the Manager.
• Ability to customize the Manager on a per-deployment basis.
• Ecommerce integration via Foxy Cart.

Distinguished Clients:

• http://www.pippatoledo.com/
• http://www.aquevix.com/
• http://www.not1bug.com/
• http://everlight-uva.com/
• http://www.tritopora.ru/
• http://www.strategische-webloesungen.de/

• Good to have a choice for favorite JavaScript framework.
• Light CMS solution (but not necessarily the fastest).
• PHP 4 support for developers mean that a lot of compromises had to be made in terms of OOP (Object Oriented Programming) in order to offer PHP 4 support .
• Nice to have freedom to set custom URL.

#4: WordPress

Official site: http://wordpress.org/

WordPress was first released on May 27, 2003, by Matt Mullenweg as a fork of b2/cafelog. As of August 2010, version 3.0 had been downloaded over 12.5 million times. Nowadays, known as the #1 CMS for blogging.

Weekly downloads: 983,625 (ranked #1 before Joomla).

Installations: 1,012 according to the survey (#2 after Joomla and before Drupal), but 12.9% of the Alexa Top 1 million sites (#1 before Joomla).

Brand Familiarity: #2 (after Joomla and before Drupal).

Major Features:

• Highly optimized for blogging.
• Custom and easy to switch themes.
• Users can re-arrange widgets without editing PHP or HTML code.
• Official support for only MySQL.
• Custom URL, clean permalink structure, excellent for SEO.
• Nested, multiple categories to articles.
• Support for tagging. Advanced search by tags.
• Highly intuitive UI (User Interface).
• jQuery JavaScript framework.
• Supports the Trackback and Pingback standards for displaying links to other sites that have themselves linked to a post or article.
• Rich plugin architecture which allows users and developers to extend its functionality beyond the features that come as part of the base install.

Extensions: 12,780 plugins and 1,315 themes.

Distinguished Clients:

• http://www.nytimes.com/interactive/blogs/directory.html
• http://tmagazine.blogs.nytimes.com/
• http://crowdfavorite.com/
• http://blog.broadband.gov/
• http://blogs.america.gov/
• http://www.number10.gov.uk/
• http://www.speaker.gov/
• http://allthingsd.com/
• http://politicalticker.blogs.cnn.com/
• http://stylenews.peoplestylewatch.com/

• Perhaps, the most convenient, easy to use and intuitive CMS (or BMS) in the world. Perfect for blog sites. But if you need to develop a dynamic site with various components, perhaps, other CMS would fit better, though, WordPress provides enough plugins to accomplish almost all tasks.
• I would avoid Wordpress as a CMS in a professional environment. As stated earlier, it's a great blogging platform, but doesn't generally offer the robustness that most professional environments require.
• Availability of jQuery makes the plugin development a lot easier for external developers and site owners.
• Endless themes - no need to worry about the new design for your site, unless you really need something special.

#5: DotNetNuke

Official site: http://www.dotnetnuke.com/

DotNetNuke is an open source platform for building web sites based on Microsoft .NET technology. It is written in VB.NET and distributed under both a Community Edition BSD-style license and a commercial proprietary license. The Community Edition is a popular web content management (WCM) system and application development framework for ASP.NET, with over 6 million downloads and 600,000 production web sites as of October 2010. More than 8,000 DotNetNuke apps are available for purchase on Snowcovered.com. DotNetNuke.com has over 800,000 registered members as of October 2010.

Weekly downloads: 13,000 (ranked #4 after Drupal and before CMS Made Simple).

Installations: 402 according to the survey (#4 after Drupal and before Liferay), but 0.2% of the Alexa Top 1 million sites (#4 after Typo3 and before MOVABLE TYPE).

Brand Familiarity: #4 (after Drupal and before Typo3).

Major Features:

• Distinguishes between community (common features) and enterprise (full set of features) editions.
• Various modules, and data providers.
• Provides language packs for about 60 languages.
• Customizable through skins and templates.

• http://www.chamberlain.edu/
• http://magenic.com/
• http://www.graphiksolutions.ca/
• http://www.marly.com/
• http://www.dotcomsoftwaresolutions.com
• http://www.cityplaceascent.com/
• http://sites.kiwanis.org/kiwanis/en/home.aspx
• http://www.zonediet.com/

• A little bit difficult to create modules.
• It is perceived as a little bit bulky CMS.
• Does not provide extensive documentation and user guides.
• Similar to most CMS, does not provide full backward compatibility in its new major releases.
• Unlike its enterprise edition, the community edition is not tested and certified by the DotNetNuke Corporation.
• Even if it provides the language packs, the sites cannot be created to support multiple languages. Third party module should be used to enable this feature.
• Auto-upgrade, Advanced Site Search, Page Cashing, and many must-have features are not included in the Community edition, only in Professional or Enterprise editions.

#6: Umbraco

Official site: http://umbraco.org/

Umbraco is also an open source content management system. It was developed by Niels Hartvig in 2000 and released as open source software in 2004. It is written in C# and can be deployed on Microsoft based infrastructure. In 2010, with about 1,000 downloads a day, Umbraco was in the Top 5 most popular downloads via the Microsoft Web Platform Installer, two places below its main rival DotNetNuke.

Weekly downloads: 5,420 (ranked #10 after Alfresco and before MODx).

Installations: 57 according to the survey (#13 after MODx and before e107), but less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #16 (after Liferay and before e107).

Major Features:

• Can be deployed with several databases, including MySQL, SQL Server, and VistaDB.
• SEO-friendly URLs.

What users say:

• Limited number of extensions.
• Official support for Windows OS only.
• Umbraco is oriented to small low-cost sites.
• Video trainings have to be purchased.
• Many must-have features have to purchased.

#7: Liferay

Official site: http://www.liferay.com/

Liferay Portal is a free and open source enterprise portal written in Java and distributed under the GNU Lesser General Public License. It allows users to set up features common to websites. It is fundamentally constructed of functional units called portlets. Liferay is sometimes described as a content management framework or a web application framework. It comes with certain portlets preinstalled. These comprise the core functionality of the portal system.

Weekly downloads: 9,435 (ranked #6 after CMS Made Simple and before TYPO3).

Installations: 154 according to the survey (#5 after DotNetNuke and before TYPO3), but less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #15 (after MODx and before Umbraco).

Major Features/ Portlets:

• Can tag and categorize contents.
• Document Library Manager, Recent Documents.
• Alfresco, Documentum, and other document library integration.
• User management based on various roles and groups (ACL).
• WebDAV Integration (Web-based Distributed Authoring and Versioning which allows users to collaboratively edit and manage files on remote web servers).
• Nested Portlets
• User Directory
• LDAP Integration
• Microsoft Office Integration
• Calendar/Chat/Mail/Message Boards/Polls
• Wiki (supports Creole as well as MediaWiki syntax)
• Alerts and Announcements
• Knowledge Base
• Social Equity
• Can create multi-language sites.
• Asset Publisher to publish many contents, tagged by a specific term, at once.

Distinguished Clients: It is primarily used to power corporate business sites.

• http://developer.cisco.com
• http://www.t-mobile.cz
• http://www.betavine.net
• http://www.foxchannel.de/
• http://www.ixarm.com

• It is mostly used by enterprise companies rather than for powering personal or community sites, though it has social and collaboration features.
• More professional developers driven project (backed by Liferay Inc.) rather than the community driven.
• Provide paid Enterprise edition.
• All features are available in Community edition (except for support and customization related).

#8: Typo3

Official site: http://typo3.org/

TYPO3 is a free and open source CMS released under the GNU General Public License oriented to small to mid size enterprise-class users. TemplaVoila is an alternative template engine extension for TYPO3. A graphical mapping tool for creating templates is included, an alternative page module, the ability to create flexible content elements and an API for developers. New content element types can be created without programming. TemplaVoila facilitates more flexibility for maintaining web pages than TYPO3's standard templating, while making it possible to enforce a strict corporate design and allowing editors to work with content more intuitively.

Weekly downloads: 7,461 (ranked #7 after Liferay and before eZ Publish).

Installations: 122 according to the survey (#6 after Liferay and before Tiki), 0.6% of the Alexa Top 1 million sites (#4 after Drupal and before DotNetNuke).

Brand Familiarity: #5 (after DotNetNuke and before OpenCMS).

Major Features:

• Supports MySQL, Oracle, MS-SQL, PostgreSQL, ODBC, LDAP - virtually any external data source.
• You can undo any change you make on the site.
• Can create multiple sites with multiple domains for each.
• Can have multiple template per site.
• User management.
• Able to switch from administrator user to general user to check permissions.
• Sandbox: administrators can set up a section within the system to test new features without disturbing the main site.
• Versioning of content pages.
• Advanced caching: template, navigation or page level.
• Link management.
• Multi-language content.
• Search Engine friendly URLs.

Distinguished clients:

• http://www.volkswagen-ir.de
• http://www.eadssecureuk.com
• http://www.geveriwise-eu.com
• http://www.rewe-xxl.de

• The administrator's UI (User Interface) is not user-friendly, though intuitive and functional.
• It is extremely powerful which provide more enterprise level features, but the learning curve is incredibly steep.
• Great ease of multi-lingual site management.
• Difficult to customize templates. Need to learn TypoScript and TemplaVoila, two TYPO3-specific systems.

#9: CMS Made Simple

Official site: http://www.cmsmadesimple.org/

CMS Made Simple is an open source cms built using PHP with support for MySQL and PostgreSQL. The template system is driven using the Smarty Template Engine.

Weekly downloads: 9,948 (ranked #5 after DotNetNuke and before Liferay).

Installations: 72 according to the survey (#7 after Tiki and before Alfresco), 0.1% of the Alexa Top 1 million sites (#8 after Xoops and before eZ Publish).

Brand Familiarity: #12 (after Xoop and before Ez Publish).

Major Features:

• Officially supports MySQL and PostgreSQL.
• Search Engine Friendly URLs.
• Users management and group based permissions system.
• Content tagging.
• Site localization is available in 20 languages.

What users say:

• It is oriented more toward professional users.
• It does not provide many templates, so coding knowledge is a must.

#10: Movable Type

Official site: http://movabletype.org/

Movable Type is a weblog publishing system, similar to WordPress, developed by the Six Apart company in Perl programming language. At various times, this company maintained three other publishing systems - TypePad, Vox, and LiveJournal. Movable Type was publicly announced on September 3, 2001. Version 1.0 was publicly released on October 8, 2001, thus it is a blogging system older than WordPress. On 12 December 2007, Movable Type was relicensed as free software under the GNU General Public License. Based on the list of its customers, Movable Type is quite credible CMS.

Weekly downloads: unavailable.

Installations: 30 according to the survey (#17 after Silverstripe and before OpenCMS), 0.1% of the Alexa Top 1 million sites (#6 after DotNetNuke and before Xoops).

Brand Familiarity: #8 (after Plone and before Alfresco).

Major Features:

• Convenient blogging system with social community features.
• Since version 5 officially supports only MySQL. PostgreSQL and SQLite can be used via plugins. Databases such as Oracle can be integrated with Movable Type Enterprise edition.
• Manage user roles. OpenID support.
• Multiple site hosting.
• Easily customizable templates.
• Revision history.
• Can add custom fields.
• Content tags and categories.
• Feeds and trackback links.
• Available localization and internationalization.
• Can generate static pages (updated whenever the content of the site is changed).

Distinguished clients:

• http://www.britneyspears.com/
• http://www.barackobama.com/
• http://boeingblogs.com/
• http://blogs.oracle.com/
• http://gehealthcare.typepad.com/
• http://www.spd.org/
• http://blog.ted.com/

• Users often deploy Movable Type for their blog sites, newspaper or other type of publishing sites.
• A powerful alternative for WordPress.

#11: Plone

Official site: http://plone.org/

Plone, a free and open source CMS, started in 1999 by Alexander Limi, Alan Runyan, and Vidar Andersen. It was made as a usability layer on top of the Zope content management framework, thus Plone is written in Python. The first version was released in 2001. In 2004, Plone 2.0 was released. This release brought more customizable features to Plone, and enhanced the add-on functions. In 2007, Plone 3 was released. This new release brought inline editing, an upgraded visual editor, and strengthened security, among many other enhancements. Recently in 2010, Plone 4 was released with major improvements in performance.

Weekly downloads: unavailabe.

Installations: 34 according to the survey (#15 after e107 and before Silverstripe), 0.1% of the Alexa Top 1 million sites (#10 after eZ Publish).

Brand Familiarity: #7 (after OpenCMS and before Movable Type).

Major Features:

• Inline editing - no need to reload the page for editing.
• Localized into 40 languages.
• Plone can be integrated with Active Directory, Salesforce, LDAP, SQL, Web Services, and Oracle.
• Working Copy support - keep the old version of a content published until you publish a new version.
• Cut/copy/paste operations on content.
• Link and reference integrity checking - no more broken links within your site.
• Can create workflows - useful for organizations.
• LiveSearch - instant site search powered by AJAX.
• Full-text indexing of Word and PDF documents.
• Wiki support
• Collaboration and sharing
• Automatic locking and unlocking
• Versioning, history and reverting content
• Authentication back-end
• Collections
• Multilingual content management
• Automatic previous/next navigation
• Human-readable URLs
• Caching proxy integration
• Drag and drop reordering of content
• Adjustable templates on content
• RSS feed support
• Automatic image scaling and thumbnail generation
• Comment capabilities on any content
• WebDAV and FTP support
• Hot -backup support

Distinguished clients:

• http://www.amnesty.ch/en
• http://www.brasil.gov.br/
• http://www.chicagohistory.org/
• http://ccnmtl.columbia.edu/
• http://cnx.org/
• http://discovermagazine.com/
• http://www.ece.rice.edu/
• http://www.engagemedia.org/
• http://science.nasa.gov/

• It has a lot of features (several times more than listed above) built-in.
• To become a Plone expert is long and expensive. It has a steep learning curve.
• Plone is backed by Zope framework which is very powerful with support for caching, rollback, etc. - everything what your organization might need.
• Plone is really complex to deeply tweak.
• Plone is known for its high security.
• Plone runs slow if you don't know how to optimize it.

#12: eZ Publish

Official site: http://ez.no/

eZ Publish is an open source enterprise CMS developed by the Norwegian company eZ Systems in 1999 using PHP programming language. eZ Publish is freely available under the GPL licence, as well as under proprietary licenses that include commercial support. eZ Publish supports the development of customized web applications. Typical applications range from a personal homepage to a multilingual corporate website, which include role-based multi-user access, e-commerce functions and online communities.

Weekly downloads: 7,031 (ranked #8 after TYPO3 and before Alfresco).

Installations: 60 according to the survey (#11 after Concrete5 and before MODx), 0.1% of the Alexa Top 1 million sites (#9 after CMS Made Simple and before Plone).

Brand Familiarity: #13 (after CMS Made Simple and before MODx).

Major Features:

• Advanced Search feature is available in only Enterprise Edition.
• Online image editor.
• Can create building blocks for the site and later reuse them in several pages.

• http://www.schemexpert.com/
• http://www.ticotimes.net/
• http://jp.wsj.com/
• http://www.laborange.fr/
• http://canalstreet.canalplus.fr/
• http://www.hks.harvard.edu/
• http://www.vogue.com.au/

• The documentation is a bit limited especially there is almost no documentation for module development.

#13: Concrete 5

Official site: http://www.concrete5.org/

Concrete5 is an open source CMS started in 2003 as a rapid-design approach to building the now-defunct LewisAndClark200.org, the official site for the Ad Council's National Council for the Lewis & Clark Bicentennial. Concrete5 is developed in PHP and is distributed under MIT software license.

Weekly downloads: unavailable.

Installations: 62 according to the survey (#10 after Alfresco and before eZ Publish), has less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #20 (after TextPattern).

Major Features:

• Integrated server side caching.
• Support for only MySQL.
• Inline content editing.
• Image editing tool.
• Editable areas are defined in concrete5 templates which allow editors to insert 'blocks' of content. Additional blocks are available as add-ons.
• Automatic upgrade is available.
• Advanced Permissions to track content versions.

• http://www.genco.com
• http://www.cottonfrombluetogreen.org/
• http://www.cs.uh.edu/
• http://www.signals.ca/
• http://www.paulraymondgregory.com

• It's PHP based and quite new but it's quite a nice layout and it's really natural for new CMS users. You can go from a paper-based sitemap and PSD to a full site structure, ready for data entry, within a day, two at a push.
• Concrete5 is simple, suitable for creating sites quickly.
• Creating template is very easy with Concrete5.

#14: Alfresco

Official site: http://www.alfresco.com/

Alfresco is an open source enterprise content management system for Microsoft Windows and Unix-like operating systems. Alfresco includes a content repository, an out-of-the-box web portal framework for managing and using standard portal content, a CIFS interface that provides file system compatibility on Microsoft Windows and Unix-like operating systems, a web content management system capable of virtualizing web apps and static sites via Apache Tomcat, Lucene indexing, and jBPM workflow. The Alfresco system is developed using Java technology. John Newton (co-founder of Documentum) and John Powell (a former COO of Business Objects) founded Alfresco Software, Inc. in 2005.

Weekly downloads: 7,000 (ranked #9 after Ez Publish and before Umbraco).

Installations: 70 according to the survey (#9 after CMS Made Simple and before Concrete5) Alexa ranking is not available.

Brand Familiarity: #9 (after Movable Type and before Tiki).

Major Features:

• Document Management.
• Web Content Management (including full webapp & session virtualization).
• Repository-level versioning (similar to Subversion).
• Records Management, including 5015.2 certification.
• Repository access via CIFS /SMB, FTP, Web DAV, NFS and CMIS.
• j BPM workflow.
• Advanced search with Lucene.
• Multi-language support.
• Officially runs on Windows, Linux and Solaris.
• User Interface official supports Internet Explorer and Firefox.
• Desktop integration with Microsoft Office and OpenOffice.org.
• Clustering support.
• Pluggable authentication: NTLM, LDAP, Kerberos, CAS.

• France Air Force
• Harvard Business School Publishing
• Toyota
• Sony Pictures
• Fox
• National Academy of Sciences
• Cisco

• Alfresco is mostly for enterprises rather than for personal sites.
• Simple to install and use, flexible and open-ended.
• Alfresco is a solution with the broadest range of technical capabilities and the best feedback from users. In addition to demonstrating a promising roadmap for collaboration tools, Alfresco was highly attractive from a cost perspective, compared to the proprietary products offered by other ECM vendors.
• All in one solution for enterprises.

See also

Besides, the official home pages, you can refer to Wikipedia for general information on these content management systems. Also Stack Overflow provides very informative users feedbacks on each of these CMS.

1. http://en.wikipedia.org/wiki/Drupal
2. http://en.wikipedia.org/wiki/Joomla
3. http://en.wikipedia.org/wiki/Modx
4. http://en.wikipedia.org/wiki/Wordpress
5. http://en.wikipedia.org/wiki/DotNetNuke
6. http://en.wikipedia.org/wiki/Umbraco
7. http://en.wikipedia.org/wiki/Liferay
8. http://en.wikipedia.org/wiki/Typo3
9. http://en.wikipedia.org/wiki/CMS_Made_Simple
10. http://en.wikipedia.org/wiki/Movable_Type
11. http://en.wikipedia.org/wiki/Plone_(software)
12. http://en.wikipedia.org/wiki/EZ_Publish
13. http://en.wikipedia.org/wiki/Concrete5
14. http://en.wikipedia.org/wiki/Alfresco_(software)
15. http://www.waterandstone.com/sites/default/files/2010%20OSCMS%20Report.pdf
16. http://php.opensourcecms.com/
17. http://stackoverflow.com/

When developers review various IDE for PHP, it is not accident that PHPStorm is among the top ranked. Because it is one of the best feature-rich PHP Development Tools out there. The good news is JetBrains, the company behind this project, has just released a new version of PHPStorm 4.0. If you try PHPStorm 4.0, I bet you won't wish to come back to your current IDE. It's really the smartest PHP IDE I have ever used!

I am sure there are still many geeks who use Notepad++ or TextMate to write their code. I had personally used Notepad++ for over 6 years. It is great for source code editing. It is small and fast. But when it comes to working in a team on a large project, control the version of the code, commit, push to, or auto sync with the remote stage server which is accessible only through SSH, merge the remote changes, and all this with a press of a keyboard shortcut, you need a fully fledged IDE which will help you accomplish all these effortlessly. And PHPStorm is that IDE which lets you focus on creating the beautiful code and leave the rest to the IDE.

I won't enumerate all the features available in PHPStorm in this blog, but I want to tell you only about my favorite ones.

Intelligent PHP Editor

Auto completion

It is not just the usual code completion or syntax highlighting features which are available in almost all source code editors and IDEs. In addition to that PHPStorm shows auto complete suggestions for array indexes within your PHP code. This is very convenient!

What I like even more is how easy it is to replace the code by another code suggested by PHPStorm. Assume you have accidentally typed cubrid_close() instead of cubrid_connect().

Instead of manually pressing several times on Backspace/Delete key on your keyboard to delete the "close()" part of the function, all you have to do is to place the cursor after the underscore, start typing your desired function, then press Tab key on your keyboard. This will automatically replace the rest of the word with the one selected in the auto complete list. I like this feature very much! It saves so much time!

Support for CUBRID PHP API syntax

PHPStorm is the first IDE to provide code completion for CUBRID PHP API. I would give the credit to them just for this feature!

It is not just the code completion, but also the entire documentation of the selected API is available on keyboard shortcut (Ctrl+J).

Code quality analysis

Support for PHP Code Sniffer

PHPStorm has really valuable features to improve the quality of your PHP code. Its on-the-fly error checking allows you to detect coding standard violations using popular PHP Code Sniffer. Also, it provides PEAR, Zend as well as Drupal coding style support as well as various techniques for code refactoring. It is just perfect for PHP coding.

Integrated PHPUnit

It is so easy in PHPStorm to write test cases and run them with PHPUnit. It is even more pleasant to instantly see how much of your code is covered by the tests. It helps you immediately spot where you have to fill the gaps.

Visual PHP debugger

PHP debugging in PHPStorm is very easy. In fact you need zero configuration to have it working. Once you install your favorite debugger (I personally use Xdebug; Zend Debugger is also supported), it is ready to go. You can debug both local and remote servers. The debugging session is kept alive while you move between web pages. If you have difficulties configuring the debugger, there is a step by step tutorial and even a video tutorial.

Powerful set of keyboard bindings

This is perhaps the best thing in PHPStorm. There are so many predefined keyboard shortcuts in PHPStorm that you can accomplish almost anything instantly with just your keyboard be it code committing or pushing, adding or deleting a file from your local repository, uploading a file to the remote server via SFTP or FTP, syncing with the remote VCS repository (I personally use Git, but it also supports SVN, Mercury, CVS, TFS, and Perforce). Various server side operations can be triggered by a mere shortcut.

My most favorite shortcut is Ctrl+Cmd+↑ to upload the currently active file to the remote stage server via SFTP so that I can see the changes live on the development server. It is especially convenient when you do not have the full test server on your local machine. By the way, you will not find this shortcut in PHPStorm because I have set it for myself. In PHPStorm you can assign your own shortcut to any available action. For this go to Settings > Keymap. In my case I have assigned the above shortcut for Settings > Keymap > Plugins > Remote Hosts Access > Upload to Default server. Super convenient!

Full support for PHP 5.4

The latest version of PHPStorm has full support for PHP 5.4 including traits, class member access on instantiation, short array syntax, array dereferencing on function call, binary literals, expressions in static calls, etc. If you want to develop for PHP 5.4, PHPStorm will make it very easy! It is just perfect for PHP reliant Web applications development.

Frequent updates

Last but not least, I like how frequently PHPStorm gets patches. Once in a while it pops up a message window asking you to update the IDE. The updates are available immediately on all supported operating systems (Windows, Mac OS X, Linux). You сan feel how the developers are trying to make their product better.

But...

I should say that there is one BUT, maybe the BIG one. PHPStorm is a commercial IDE (99 USD). For some this may be the important criteria when choosing the code editor. But I will tell you what! Try it (free trial for 30 days)! You will really love it! They also provide free licenses to open source projects (our team is using PHPStorm under the open source license) as well as to educational institutions. So if you are an official member of an open source project, requests your free license and enjoy coding!

What do Web developers need from an SQL Client? An easy, uncluttered Web interface to execute and obtain the results for the most basic SQL queries: SELECT, INSERT, DELETE, UPDATE (SIDU). This is exactly what SIDU does and stands for! Even better - SIDU does not require any installation or configuration. Just extract the files into a directory accessible by your Web server, then open your browser. Done!

Written in PHP, SIDU is really the simplest and most intuitive Web based SQL Client I have ever used. The great news is that the latest version of SIDU 3.5 now officially supports CUBRID Database. At CUBRID, we are very excited about this because when we have tried the beta version of SIDU 3.5 we have fallen in love with it! If you try it, you will understand how simplicity and attention to small details make so much impression on a user.

Here is what SIDU has to offer:

• CRUD operations with CUBRID, MySQL, PostgreSQL, and SQLite databases.
• No need to install or configure. Just extract and open the browser.
• Minimalistic app (210KB). Provides only what it is designed for: SIDU (SELECT, INSERT, DELETE, UPDATE).
• Simultaneously connect to multiple databases.
• Database and table navigation panel.
• Multiple query execution. Displays query results for all statements.
• Provides SHOW CREATE TABLE.
• Keeps SQL execution history.
• Useful keyboard shortcuts.
• Built-in filter in the query results panel. No need to change the main query.
• Hide unnecessary columns in the query results panel.

So, if you need an instant access to your database from Web to quickly execute SQL, SIDU is the app you are looking for! We have created a short tutorial with screenshots to illustrate SIDU in action. Check it and see how easy it is to execute SQL in your browser with SIDU.

Information for our readers!

If you develop an open source application and would like to become a CUBRID Partner by supporting CUBRID Database in your project, contact us by email affiliates@cubrid.org. In your letter please provide an overview of your software, project links, and your statement on behalf of your project. We will be very glad to have you onboard!

Hello everybody!

I am very pleased to announce this great, really great news! Last week on March 18, 2011, CUBRID RDBMS was chosen to power the Official Web Site of the 17th Asian Games which will be held in 2014 in Incheon, South Korea.

The event officials say: "Considering the total cost of ownership of existing foreign database management systems, we think it is better to adopt the Korean DBMS which provides many features. This is going to be an impressive wave of about 4 billion Asians who are expected to communicate through the event homepage."

The event will last 18 days, and at this moment the Organizers have already started the preparations. They are developing a comprehensive plan which will guarantee the success of the event. Besides the Official Web Site, the Organizing Committee is planning to heavily use Social Networking Services to promote the new media.

It is both very impressive and very challenging for CUBRID to be the database management system for world events. I hope one day CUBRID will be deployed as part of IT systems for even bigger events such as 2012 London Olympics.

For more information about the 2014 Asian Games refer to the Wikipedia article at http://en.wikipedia.org/wiki/2014_Asian_Games.

Android, currently the most popular mobile operating system in the world, continues to further solidify its position as the major mobile platform i the world. In this article I will briefly cover how Android has appeared, and will explain its software structure and what has to be paid attention when developing apps for this platform.

What is Android?

Linux operating system is the main component in Android's software stack. It does not differ much from Microsoft's Windows OS or Apple's Mac OS which we use on the daily basis. Therefore, theoretically you can enjoy Android not only on mobile devices but also on other tablet devices as well just like Windows even to a point where various applications can be downloaded and used. Of course, if you want to make an Android application by yourself, you should also install ADK (Android Development Kit) to your PC and use Java programming language for development. For a lower level program, NDK (Native Development Kit) for C language shall be used.

It seemed that Android had been shunned publicly for not having as sensitive UI as iPhone. However, now it is considered as the most viable blue-chip mobile OS with advantageous openness, free, and Java-based characteristics.

Another interesting aspect is that most users mistakenly think that Android was developed by Google. In fact, it was developed in 2003 by "Android" company based in California. Back then, few people knew about it, even less developed for it. However, everything was changed in 2005 when Google acquired Android company to give birth to Android that we know now.

The History of Google Android

Since its initial release of 1.0 version in September 23^rd, 2008, Android has enjoyed a remarkable growth. All versions until 3.0 (code name Honeycomb) are available under Apache free open source license. According toAndroid official blog, the source code of the Honeycomb series was not released to the public because it was specifically created for tablet computers, and Google did not want external developers to mess up the user interface by putting on smartphones a version of Android intended for tablets.

However, Google later announced that in the next 4.0 version, planned for the second half of this year (code nameIce Cream Sandwitch), Android will combine both the smartphone version (code name Gingerbread) and the tablet version (HoneyComb).

The first version of Android, released back in 2008, had drawbacks in many aspects. Some users voiced their worries about its terrible UI and development tools, saying “who would want to use it”. However with its advantage of using Java as SDK language and its openness Android gradually solidified itself as the most rapidly developing mobile OS, and it seems as if no one dare to challenge the idea that Android will further expand and attract more users than iPhone and be loaded onto more various mobile devices.

The following are the key functions of each Android version

• 2.0, 2.1 (Eclair): Web browser supporting new UI and HTML5. Advanced camera app, new launcher app and weather app. Supports moving background.
• 2.2 (Froyo) : Improved performance by optimizing JIT. Chrome V8 Javascript engine. Support Wi-fi hot spot tethering and Flash.
• 2.3 (Gingerbread) : Optimized UI for minimum battery use. Enhanced soft keyboard, COPY/PASTE function, NFC (Near Field Communication, local network).
• 3.0 (Honeycomb) : Support tablet screen. New UI and multi-core processor.
• 3.2 (Honeycomb) : Compatible mode for zoom-to-fill screen. Read media file directly from SD card. Additional screen API. Controls UI more delicately.
• 4.0 (Ice Cream Sandwich) : Combined version of Gingerbread and Honeycomb.

The Structure of Android

Android basically runs on DVM (Dalvik Virtual Machine) which is Android runtime loaded on Linux kernel. The figure below illustrates all the components. Further I will explain each of them from top to down.

ANDROID STRUCTURE

Applications

This domain is for applications developed with Java. Generally, email client, SMS program, calendar, browser and contacts applications are loaded.

Application Framework

Application framework provides all kinds of APIs needed to develop a full-featured application for Android platform. Using APIs in this domain allows to express buttons and text on the screen or use data of other applications such as images and character strings. Also it is possible to manage application’s lifecycle.

Libraries

This domain provides various C/C++ libraries which can be used on Android. All libraries in this domain can be used by developers via application framework. Standard C system libraries based on BSD (Berkeley Software Distribution) are revised to be suitable for Linux-based devices. Media library based on OpenCore of PacketVideo supports MPEG4, H.264, MP3, AAC, AMR, JPG and PNG files. Surface manager supports 2D and 3D graphics whileWebKit supports browser functions. SQLite is also available as a database engine which can be used by applications.

Android Runtime

Although Android is developed in Java language, DVM (Dalvik Virtual Machine) is used instead of JVM (Java Virtual Machine). Thus, a source code (Java) is compiled to a class file (.class) with Java compiler which is later transformed to Dalvik Executable by DX tools. The reason for this is that Dalvik is optimized to run on small sized devices with limited memory.

Linux Kernel

Android is developed based on Linux kernel 2.6 which provides high security, memory and process management, network stack and various drivers.

Android Project Structure

Android applications are developed in Java. However, unlike the desktop version of Java applications, the class files in Android applications are not loaded directly. Instead Android's development tools combine the resource files and DEX file transformed from class file to make APK. And this APK gets loaded.

Below is the basic directory structure for developing the Android application.

You need to make sure that files are stored in the right directory, particularly for resources. Since directories are already reserved, changing at your will would make directories difficult to recognize.

Resource Directory

Resource directory is composed of drawable directory for placing images and layout directory for putting the XML files to the layout. Raw and xml directories are not automatically created in the default project, so should be created manually.

The vertical and horizontal mode differs in size on the phone screen. Therefore, if you need both of them to be shown identically, you need to create screen layout file on /res/layout/ directory. To show differently, create the layout in the layout-port (portrait) and layout-land (landscape) directories. Also, often you may need to create a layout and layout-land only. Then files in layout will be applied to vertical mode layout.

Result Directory

The structure of bin (result) directory is as follows.

Files that end with “.apk” are the final outcome of Android project, thereby they are ready to be registered in the Android market. The compressed format of this file is zip, which allow us to easily open with compressing programs. However opening this file will not let you decode the meaning since the source files are already compiled.

AndroidManifest.xml

AndroidManifest.xml file is a very important file holding information about the overall structure of the application. Through this file, the Android system finds out component kinds and intent filter and later checks the authority.

AndroidManifest.xml can declare the following elements.

Elements and Structure of Android Application

Android has 4 key components:

• Activity
• Service
• Broadcast Receiver
• And Content Provider

These four components can be closely connected using Intent. Below are the key terms and 4 components for Android. Knowing below words will help you understand better Android.

Android Key Terminology

There are various kinds of components for Android, but development using these elements can be described in MVC structure.

VIEW

View is for screen composition and this domain can be composed by using the class which inherits View class provided by Android.

CONTROL

This domain is for connecting and controlling VIEW and MODEL. Activity, Service and BrodacastReceiver are included.

MODEL

This domain is for storing application’s data. SQLite, an embedded database system in Android, File and Content Provider, which can be used for sharing data among applications, are included.

Conclusion

Java SDK is, perhaps, one of the most important advantages of Android which provides various powerful functions of Java programming language. However, many developers complain about Android's fast development pace that they have difficulties to catch up. But if we think more positively, this fast-paced transformation seems to have materialized Android to consolidate itself as the most popular mobile platform. On top of that this popularity has led to many patent suits with Microsoft, thus pressuring many carrier providers to inevitably pay patent fees who initially used the technology thinking it was free. Also Oracle’s patent suit against Google citing that Java cannot be used on mobile devices just because Android uses Java. For above reasons and more, the future path of Android might not seem as smooth as before, but we can’t say it will be gloomy either. I hope all these issues will be settled down, and Android will continue to prosper.

By Dong Ho Han, Web Platform Development Lab, NHN Corporation.

To mention, a few months ago we have developed the CUBRID driver for CI, submitted to their new repository at Github, and now we can see the fruits. From now on we will help the CI team to maintain the code.

Feel free to download CUBRID and give it a try together with CodeIgniter. If you find any DB related bugs, please report them to our CUBRID Issue Tracker (JIRA) or CI Issue Tracker (Github).

If you have questions about CUBRID, ask on CUBRID Q&A site.

DataCleaner is an open source data quality analyzer which can work with CUBRID, Oracle, MS SQL Server, MySQL, PostgreSQL, CSV, and other datasources to help you quickly analyze, clean and profile you data. Its core consists of a strong data profiling engine, which is also extensible, and thereby adds data cleansing, transformations, enrichment, duplicate finding, matching and merging functionalities.

It is very exciting that DataCleaner now provides support for CUBRID RDBMS in their latest 2.5.1 release. Kasper Sørensen, the founder and developer of DataCleaner, says he is "very pleased to see DataCleaner finally supporting the CUBRID database." He added: "With it's focus on web applications I see a tremendous potential for the CUBRID database in conjunction with DataCleaner. And so far the CUBRID community has been a joy serving."

Here is how Kasper explains why DataCleaner can be a useful tool for Web developers:

Every self-respecting web application takes user input in one form or another. But as it turns out, we as webapp developers do not always foresee every possible type of user input. So over time data quality issues will occur! Typically what we see happening in web applications is in the form of missing values, fake/spam/made-up values, wrongly formatted database statements leading to values being placed in the wrong fields, duplicated records and so on. All typically because web applications are rapidly being developed and maintained in a quite agile way.

Here is what you can do with DataCleaner:

• Profile and analyze your database within minutes!
• Access almost any datastore - Oracle, MySQL, PostgreSQL, MS SQL Server, MongoDB, CUBRID, CSV files, Excel spreadsheets, DBase and more.
• Discover patterns in your textual data with the Pattern Finder.
• Find out which values occur the most with the Value Distribution profile.
• Cleanse your contact details with name and address validations.
• Detect duplicates using fuzzy logic and configurable weights and thresholds.
• Merge your duplicates and create a single version of the truth.
• Write data back to relational databases, CSV files, Excel spreadsheets or MongoDB databases.

Frankly speaking, DataCleaner is so sophisticated that when I ran DataCleaner for the first time, I was stunned by the number of features it provided. For you not to feel lost in this ocean of data analyzing possibilities, I recommend you first to watch some of the DataCleaner screencasts available on their project website. Then, based on the video instructions and your needs, you can launch DataCleaner and get started with data cleansing and analysis. We have also created a short tutorial which shows you how to quickly connect DataCleaner to CUBRID Database.

CUBRID community members will certainly benefit much from the whole set of powerful features that DataCleaner provides. On behalf of our community I would like to thank Kasper for his valuable contribution.

Information for our readers!

If you develop an open source application and would like to become a CUBRID Partner by supporting CUBRID Database in your project, contact us by email affiliates@cubrid.org. In your letter please provide an overview of your software, project links, and your statement on behalf of your project. We will be very glad to have you onboard!

We have created the installation instructions for you which provide three ways to install the driver. Please choose the one you feel more comfortable with.

1. Download the compiled library (DLL).
2. Download the installer (EXE).
3. Compile from source code.

Right now the beta version supports Win32. The stable version, which should come out in two weeks, will have 64-bit support as well.

If you plan to use the OLEDB driver for .NET projects via System.Data.OleDb namespace, we encourage you to use the fully managed CUBRID ADO.NET driver instead.

What's new in this version

• The provider naming is set to CUBRID.OLEDBProvider.xxx where xxx versioning will be in synch with CUBRID Engine releases.
• In this version we have focused on supporting OLE DB Consumer templates. Direct OLE DB interfaces work as well even though this technology is already deprecated.
• Added BLOB/CLOB support.
• Added support for localization (messages, errors content).
• Added Installer for the provider library.
• Enhanced schema rowsets support.
• Added test cases coverage. There are now 60+ test cases available!

Improvements in the coming stable release

• Add BLOB/BLOB Dynamic Accessors support in the stable release.
• Add schema support for CUBRID Stored Procedures.
• Improve support for Auto-Commit mode. Currently the Update() support is restricted to Auto-Commit mode = Off, which must be explicitly stated via session transaction.
• Add support for cci_connect_with_url. Improve UTF-8 support.
• Add support for the new cci_prepare_and_execute function.
• Add more test cases coverage, in particular for CUBRID specific data types and for .NET OleDb access.
• Remove the data provider distribution from the CUBRID standard installer.
• Release 64-bit version.
• Publish Doxygen documentation.
• SVN cleanup.

More information about this beta release you can find in the OLEDB Release Note.

In conclusion, I would like to invite all users to help us verify this beta version. If you find any bug, please report to our Issue Tracker.

And one more time, we will release the stable version in about two weeks from now, so stay tuned!

Statement Pooling allows to improve the performance of an application by caching SQL statements that are used repeatedly. Such caching mechanism allows to prepare frequently used statements only once and reuse them multiple times, thus reducing the overall number of times the database server has to parse, plan, and optimize these queries. A well-configured number of statements (maxStatements) to be cached can be as good as tuning the Garbage Collection. Now let's see how Statement Pooling can affect the Garbage Collection.

Why Check the Number of Statement in the Pool?

Often the size of the JDBC statement pool is set to the default value. Using the default value, of course does not usually lead to any special issue. But a well-configured maxStatements value can be as effective as GC tuning. If you are using the default maxStatements value and would like to optimize the use of memory, let's think about the correct statement pool value before attempting GC tuning.

As was discussed in Understanding Java Garbage Collection, a weak generational hypothesis (most objects quickly become unreachable and a reference from an old object to a new object is rare) was used as the precondition when creating garbage collector in Java. For the majority of NHN web services there should be a response within 300ms at the latest, unless it is a special case. Therefore, NHN web services are more applicable to the above situations than the general stand-alone type applications.

The GC Process between HTTP Request and Response

When developing a web service using web containers like Tomcat and other frameworks, the lifespan of objects created by a developer tend to be either very short or very long. Web developers usually write codes like Interceptor, Action, BO, or DAO (BO and DAO are generated and used as singletons from applicationContex in Spring, and are not the target of GC). The objects generated from these codes stay alive for a very brief time that exists between the time HTTP is requested and the time it has responded. For this reason, such objects are usually collected during Young GC.

There are also objects, such as singleton objects, that stay alive long enough to exist for the lifecycle of Tomcat. Such objects will be promoted to the old area soon after Tomcat starts running. Yet, when continuously monitoring web applications through jstat and the like, there are always some objects promoted to the old area during Young GC. These objects are usually used after being stored in the cache used for improving the performance of frameworks in most of the containers and projects. Whether the cached objects become the target of GC or not is determined by their cache hit ratio, not their age, so unless the hit ratio is 100%, they cannot avoid being promoted to the old area, even when the Young GC cycle is set to be long.

Among these caches, statement pooling affects the memory usage the most. If you are using iBatis, as iBatis processes all SQLs as preparedStatment, you will be using statement pooling. If the size of statement pooling is smaller than the number of SQLs being used, the cache hit ratio will decrease and result in cache maintenance cost. Objects that are reachable in the old area become the target of GC and will be retrieved, then will be regenerated during the HTTP request process, only to be cached and promoted to the old area. The full GC cycles are affected by this process.

Size of the Statement Objects

It would be safe to say that the size of a single statement object is proportional to the length of the SQL code processed by the same statement. Even for a long and complex SQL, the size of the object should be around 500 bytes. The object's small size would seem to have little effect on the full GC cycles, but such an assumption would be incorrect. 

When you look at the JDBC specifications, each connection has its own statement pool (maxStatementsPerConnection), as described in Figure 1 below. So, although a statement object is as small as 500 bytes, if there are many connections, the statements cache may occupy the proportional amount of the heap. 

Figure 1: Relationship between the Connection and the Statement.
(Though the statement has the ResultSet, it should be clarified that ResultSet is not an object for caching. ResultSet is allocated as null when rs.close() is called by iBatis, then retrieved in the young area during young GC.)

The Effect of Statement Pool's Cache Hit Ratio on the Full GC

A simple test program was created to assess the effect of cache hit ratio on the full GC. One cache hit ratio was set to 100% while the other was set to 50%. When the same amount of load was applied, the results presented in Table 1 and 2 were obtained.

In both cases, the occurrences of young GC were very similar but the results for the full GC was different. When the cache hit ratio was 100%, full GC occurred only once, because the number of objects promoted to the old area during young GC was small. When the ratio was 50%, full GC occurred 4 times because the number of statement objects promoted to the old area during young GC was high, as the objects were cached in the statement pool, then removed from the pool in LRU way, then cached again at the next request.

Table 1. Cache hit ratio = 100%.

Table 2. Cache hit ratio = 50%.

I would like to add one more thing. When the cache hit ratio is 50%, it violates the 2nd category of weak generational hypothesis I introduced previously. When low cache hit ratio causes frequent pool registration and subsequent removal, it means the statement object generated in the young area is being referenced in the pool from the old area, which leads to additional strain during GC because the card marking technique is used to manage the references separately.

In Conclusion

In Lucy (NHN's internal Java Framework), the maxStatements value for statement pooling in Oracle and MySQL is 500. In most cases, 500 should be enough. However, when more SQL is being used, increasing the default value to meet such demand would be a way to improve the system efficiency (when using $(String replacement) for query on iBatis for the reason of table partitioning and the like, the number of queries must be multiplied by the number of partitioned tables).

However, when the default value is higher than necessary, this leads to a different problem. A higher value means more memory usage and higher likelihood of an Out Of Memory (OOME) occurrence.

In a situation where the number of SQLs are 10,000 and the number of connections are 50, then the total size of statement objects is about 250 MB. (500 byte * 50 * 10,000 = 250 MB). It should be easy to determine the likelihood of OOME occurrence by checking the Xmx configuration for the service in use.

What strategy do you follow to determine the correct number of statements to be pooled? Share your experience in the comments below.

By Dongsun Choi, Senior Engineer at Game Service Solution Team, NHN Corporation.

Let’s look at what Nginx is and how it differs from Apache Web Server.

In the beginning of 21^st century as the use of internet became more active, people began to shed more interest on web servers capable of processing more requests. Later the C10k problem emerged in this era, in other words to enable each web server to be capable of processing 10,000 clients at the same time. Therefore a need to develop a better Network I/O and Thread management technology was emerged. During this period, far less-developed technology was applied to Linux than previous ones such as epoll and NPTL.

epoll was designed for better process of IO events on Linux. kqueue (FreeBSD) and IOCP (Windows) are similar. NPTL is a thread library of Linux loaded on Kernel 206 and up (based on safe version). It provided enormous improvement in thread performance.

It is fair to say that the emergence of several technology was not the result of an effort to tackle C10k problem (as a common problem) but "The C10k problem" succeeded in putting together efforts to enhance performance of network server development. As a result various web servers such as Nginx, Lightpad and Cherokee made with new technology were in the market.

Nginx was first developed by a Russian guy called Igor Sysoev in 2002. After two years in 2004 it made its debut in the forms of HTTP server and reverse proxy/IMAP/POP3 server. To respond to C10k problem, Nginx adopted event-driven (asynchronous) structure which is not a conventional way (as in Apache) of processing single client in a single thread. Other than Nginx, various newly developed web servers such as Lightpad, Tornado, Magnum and Aleph have adopted Event-driven structure too.

Event-driven Architecture (EDA)

EDA method is capable of handling more number of clients with less threads compared to conventional way (one thread for one client). In Apache each thread is dedicated to handle one client. So, there will be many cases when a certain thread needs to be on hold due to I/O problems until certain data is read from persistence layer in "accept" and delivered to the client. There should be the same number of threads as that of clients.

However, with EDA a method enables event (i.e. when event needs to be created or response result is completed) to be processed every time it takes place by designating each state. In other words, with less thread (or even without any thread) CPU can be used more efficiently.

Nginx provides almost all the functions Apache web server does. For example:

• handling of static files
• reverse proxy
• load balancing
• SSL support
• FastCGI
• Virtual Host
• FLV Streaming
• MP4 streaming
• Web page access authentication
• gzip
• URL Rewriting
• Custom Logging
• SSI
• WebDAV

Nginx Use

According to a research done in April, 2011 by W3Techs.com, Nginx takes up 6.9% in web server market and is on a steady increase. The graph below shows the market share of Nginx in recent 1 year.

We can see that many websites process massive traffic using Nginx. Alexa.com is well known for releasing website ranking according to its traffic. The following list, released on April 18, 2011, illustrates websites from Top 500 which use Nginx.

Characteristics of Nginx

Non-blocking event-driven method

The most notable characteristic of Nginx is that it uses non-blocking event-driven method. All the network connection operates in non-blocking way. Some socket interfaces return results immediately, while some are blocked in certain cases. Socket() and setsockopt() returns results without blocking, whereas connect(), send(), recv() and close() might experience some blocking in certain cases. Nginx calls above mentioned functions only after confirming there will be no lags. In order to prevent blocking, as for connect(), socket is changed in advance to non-blocking using ioctl().

As for send() and recv(), epoll is used to confirm blocklessness. Codes for calling send() or recv() are composed in event-driven format. Each event is composed of a socket, a socket state and an operating function. Almost all of internal Nginx is operated in event-driven way since web server processes request from web browser through network.

Single thread

Nginx is operated by pre-set number of worker processes. Each process operates as a single thread. Also, the need for multi-thread is relatively low since with the use of Non-blocking event-driven allows a single worker process to handle requests by multiple clients.

Advantage of Nginx

High Performance

Generally Nginx is known to record higher performance than Apache. However, we cannot simply jump to conclusion in terms of better performance since it varies greatly according to workload. Below chart indicates a system function that is called to serve a single favicon.ico using starce. As you can see in the table below Apache called 31 system functions whereas Nginx called 16. Although the system function used here cannot accurately indicate the performance, we can still expect relative performance.

Handling many clients with less process

Disadvantage of Nginx

Difficult to create module

Does not support HTTP/1.1 in communication with Backend

Things to note while using Nginx

• Nginx has a non-blocking event-driven structure. Such structure is desirable for network transmission and reception. However, it could generate blocking when file is being sent and received. But the latest Linux versions support Asynchronous Input-Output (AIO), and Nginx began to support AIO since version 0.8.11.
• Nginx uses AIO function through eventfd() system function of Linux. However it only works when AIO of Linux is set as DirectIO. Thus, using AIO on Linux disables buffer cache of OS.
• As in the case of reading the same file over and over again, it reads from disk in the first place however from the second time it can rapidly read stored file on buffer cache. No matter how many times the file is being read, using DirectIO makes it to be read from disk consistently.
• If a DBMS uses independent buffer, DirectIO might be efficient. However in other cases DirectIO could decrease file input/output performance.
• If most of the files that Nginx read exist in OS buffer cache, most of file I/O become non-blocking. However if Nginx gets to serve files which do not existent in the buffer cache, the worker process becomes blocked due to file I/O. In this case, you need to increase the number of worker processes in order to prevent the worker process from becoming blocking.

In this article, I will explain how JVM is actually running Garbage Collection in the real time.

What is GC Monitoring? 

Garbage Collection Monitoring refers to the process of figuring out how JVM is running GC. For example, we can find out:

1. when an object in young has moved to old and by how much,
2. or when stop-the-world has occurred and for how long.

GC monitoring is carried out to see if JVM is running GC efficiently, and to check if additional GC tuning is necessary. Based on this information, the application can be edited or GC method can be changed (GC tuning).

How to Monitor GC?

There are different ways to monitor GC, but the only difference is how the GC operation information is shown. GC is done by JVM, and since the GC monitoring tools disclose the GC information provided by JVM, you will get the same results no matter how you monitor GC. Therefore, you do not need to learn all methods to monitor GC, but since it only requires a little amount of time to learn each GC monitoring method, knowing a few of them can help you use the right one for different situations and environments.

The tools or JVM options listed below cannot be used universally regardless of the HVM vendor. This is because there is no need for a "standard" for disclosing GC information. In this example we will use HotSpot JVM (Oracle JVM). Since NHN is using Oracle (Sun) JVM, there should be no difficulties in applying the tools or JVM options that we are explaining here.

First, the GC monitoring methods can be separated into CUI and GUI depending on the access interface. The typical CUI GC monitoring method involves using a separate CUI application called "jstat", or selecting a JVM option called "verbosegc" when running JVM.

GUI GC monitoring is done by using a separate GUI application, and three most commonly used applications would be "jconsole", "jvisualvm" and "Visual GC".

Let's learn more about each method.

jstat

jstat is a monitoring tool in HotSpot JVM. Other monitoring tools for HotSpot JVM are jps and jstatd. Sometimes, you need all three tools to monitor a Java application.

jstat does not provide only the GC operation information display. It also provides class loader operation information or Just-in-Time compiler operation information. Among all the information jstat can provide, in this article we will only cover its functionality to monitor GC operating information.

jstat is located in $JDK_HOME/bin, so if java or javac can run without setting a separate directory from the command line, so can jstat.

You can try running the following in the command line.

Just like in the example, the real type data will be output along with the following columns: S0C    S1C     S0U     S1U    EC     EU     OC     OU     PC.

vmid (Virtual Machine ID), as its name implies, is the ID for the VM. Java applications running either on a local machine or on a remote machine can be specified using vmid. The vmid for Java application running on a local machine is called lvmid (Local vmid), and usually is PID. To find out the lvmid, you can write the PID value using a ps command or Windows task manager, but we suggest jps because PID and lvmid does not always match. jps stands for Java PS. jps shows vmids and main method information. Just like ps shows PIDs and process names.

Find out the vmid of the Java application that you want to monitor by using jps, then use it as a parameter in jstat. If you use jps alone, only bootstrap information will show when several WAS instances are running in one equipment. We suggest that you use ps -ef | grep java command along with jps.

GC performance data needs constant observation, therefore when running jstat, try to output the GC monitoring information on a regular basis. 

For example, running "jstat –gc <vmid> 1000" (or 1s) will display the GC monitoring data on the console every 1 second. "jstat –gc <vmid> 1000 10" will display the GC monitoring information once every 1 second for 10 times in total.

There are many options other than -gc, among which GC related ones are listed below.

Only looking at frequency, you will probably use -gcutil (or -gccause), -gc and -gccapacity the most in that order.

• -gcutil is used to check the usage of heap areas, the number of GC performed, and the total accumulated time for GC operations,
• while -gccapacity option and others can be used to check the actual size allocated.

You can see the following output by using the -gc option:

Different jstat options show different types of columns, which are listed below. Each column information will be displayed when you use the "jstat option" listed on the right.

The advantage of jstat is that it can always monitor the GC operation data of Java applications running on local/remote machine, as long as a console can be used. From these items, the following result is output when –gcutil is used. At the time of GC tuning, pay careful attention to YGC, YGCT, FGC, FGCT and GCT.

These items are important because they show how much time was spent in running GC.

In this example, YGC is 217 and YGCT is 0.928. So, after calculating the arithmetical average, you can see that it required about 4 ms (0.004 seconds) for each young GC. Likewise, the average full GC time us 33ms.

But the arithmetical average often does not help analyzing the actual GC problem. This is due to the severe deviations in GC operation time. (In other words, if the average time is 0.067 seconds for a full GC, one GC may have lasted 1 ms while the other one lasted 57 ms.) In order to check the individual GC time instead of the arithmetical average time, it is better to use -verbosegc.

-verbosegc

-verbosegc is one of the JVM options specified when running a Java application. While jstat can monitor any JVM application that has not specified any options, -verbosegc needs to be specified in the beginning, so it could be seen as an unnecessary option (since jstat can be used instead). However, as -verbosegc displays easy to understand output results whenever a GC occurs, it is very helpful for monitoring rough GC information.

The followings are other options that can be used with -verbosegc.

• -XX:+PrintGCDetails
• -XX:+PrintGCTimeStamps
• -XX:+PrintHeapAtGC 
• -XX:+PrintGCDateStamps (from JDK 6 update 4)

If only -verbosegc is used, then -XX:+PrintGCDetails is applied by default. Additional options for –verbosgc are not exclusive and can be mixed and used together.

When using -verbosegc, you can see the results in the following format whenever a minor GC occurs.

This is an example of -verbosegc output for minor GC:

This is the example of output results after an Full GC occurred.

If a CMS collector is used, then the following CMS information can be provided as well.

As -verbosegc option outputs a log every time a GC event occurs, it is easy to see the changes of the heap usage rates caused by GC operation.

(Java) VisualVM  + Visual GC

Java Visual VM is a GUI profiling/monitoring tool provided by Oracle JDK.

Figure 1: VisualVM Screenshot.

Instead of the version that is included with JDK, you can download Visual VM directly from its website. For the sake of convenience, the version included with JDK will be referred to as Java VisualVM (jvisualvm), and the version available from the website will be referred to as Visual VM (visualvm). The features of the two are not exactly identical, as there are slight differences, such as when installing plug-ins. Personally, I prefer the Visual VM version, which can be downloaded from the website.

Figure 4: HPJMeter.

What is the Next Article About?

In this article I focused on how to monitor GC operation information, as the preparation stage for GC tuning. From my personal experience, I suggest using jstat to monitor GC operation, and if you feel that it takes too lmuch time to execute GC, then try -verbosegc option to analyze GC. The general GC tuning process is to analyze the results after applying the changed GC options after the -verbosegc option has been applied based on the analysis. In the next article, we will see the best options for executing GC tuning by using real cases as our examples.

By Sangmin Lee, Senior Engineer at Performance Engineering Lab, NHN Corporation.

We have already created CUBRID Facebook page as well as @CUBRID Twitter account to communicate with our users by answering to their questions and posting very detailed tutorials, announcing new CUBRID releases as well as praising our distinguished users.

We hope you will add CUBRID to your Google+ Circles because we would love to connect with you there. You can talk directly to CUBRID core developers, learn about the latest improvements in CUBRID Database, and stay up to date with the important news from our open source database community.

Join us at Google+!

The story behind Hibernate integration is that at NHN we have a large service running on Hibernate. However, unlike other cases, in this service we use CUBRID as a back-end database engine. So for several years we have been successfully using this combination in production.

Last month, though, we decided to allocate some resources and contact the Hibernate team and commit the patch. Fortunately, there were not serious reasons on Hibernate side which would keep our code away from their master branch thanks to the way Hibernate works.

Since the framework is implemented in Java, it connects to the database server through the JDBC driver, which provides standard Java methods to manage a database. So, ideally Hibernate will work with any database which provides a JDBC driver with a small exception. There has to be a way to tell Hibernate if certain database features, implemented in the framework, are supported by the DBMS. In other words DB vendors need to introduce the Dialect which the database speaks in.

Cutting to the chase, we have committed the CUBRIDDialect class to Hibernate master and 3.6 repositories on GitHub. Yesterday, the Hibernate team released the second Code Release of its new 4.0.0 branch. I have just checked its source code and can confirm that CUBRID dialect is included. Congratulations!

I do not know how many more CRs are planned, but if you wish to use Hibernate with CUBRID now, feel free to download the latest version of its 4.0.0 line from http://hibernate.org/downloads.

If you notice any weird behavior when working with CUBRID, please report your issues to our http://jira.cubrid.org issue tracker. Any other questions can be posted at our CUBRID Forum.

In this article, I will explain why these details affect an UPDATE performance, and what happens inside CUBRID when an UPDATE query is executed. My story will be based on the latest version of CUBRID as of today 8.4.1.

UPDATE in RDBMS

Let's make an analogy between the process that implements an UPDATE query in RDBMS and the process of selling products in a shopping mall.

At a shopping mall the employee keep frequently bought products on the shelves and less frequently purchased products in the main storage. When customers come, they can buy the products available on shelves quickly without waiting. For products not available on the shelves, they have to wait while the shopping mall employees go and retrieve them from the storage. It is the same in RDBMS. To have fast access to the data, you need to put frequently requested data in a memory buffer. How and what products are displayed on the shelves will have a profound impact on the shopping mall's sales. Likewise, depending on which data is in the memory buffer, it will have a decisive effect on RDBMS.

In order to improve performance, RDBMS handles data input/output by page (the default value in CUBRID is 16KB). Operations for the memory buffer used for cache are also processed by pages.

There could be a shortage of space if too many products are displayed on the shelf. Suppose you want to remove slow-selling items off the shelves and display other more frequently sold products. However, the latter ones are too big to be displayed. What should you do? You will have to move products around on the shelf to allocate more space for big ones.

Which method should you use if you do not want to rearrange a page when the new data to be UPDATED from RDBMS is bigger than the old record?

Maybe the employees at the shopping mall could display products on the shelf all together. It would be effective to divide areas but it is impossible in RDBMS. In RDBMS, reading and writing can occur in one record at the same time, or there could be a race in limited buffer space. Therefore, you should be able to access a record exclusively by using record locking.

How do we actually make this work?

The above SQL implementation process can be summarized in the flowchart below:

Figure 1: Implementation process flowchart for UPDATE query.

To be precise, we should also express the process of SQL parsing, SQL optimization, transaction process, data locking, data correction, data storing, operation logging, and operation restoration. However, this article will focus only on what happens in memory buffer and disk while updating.

Index Navigation

The first thing to be done is to find the record we need to update. If the conditions used in the corresponding query can use index, then we can use index navigation.

If the details of the our record are in the memory buffer, use them as they are. If the record is not found in the memory buffer, we need to read it from the disk and transfer it to the memory buffer. This process is called FIX.

In CUBRID, index is composed of a B+ tree (for details refer to CUBRID Query Tuning Techniques and What is Covering Index in CUBRID 8.4.0?). Generally speaking, a B+ tree stores the location (VPID) of previous and next key nodes in a non-leaf node while the disk location of the actual data which corresponds to the key is stored in a leaf node. Therefore data search process is affected by the type of a B+ tree configuration.

Figure 2: Storage structure of a B+ tree index.

Sometimes the size of a column (like text) can be bigger than a page (16KB). In this case you need a separate storage other than the node of a B+ tree. Therefore, you might have to allocate a separate storage space except the B+ tree when searching for data.

Further I will explain about which index configuration is likely to cause the overflow key file or overflow OID issues when UPDATE-ing a record.

Overflow Key

First, let's look at a very large node that configures index.

If history field data used by key is roughly 100KB, it will be greater than a page (16KB), a unit that stores regular data. This is called an overflow key, and you cannot save the key value to a page. In a node, you should store ID (OFID, Overflow File ID), which refers to a file that stores the key value.

As CUBRID manages data by page, it is burdensome if data is divided into several pages, as many memory copies (memcpy) are needed to configure a single piece of data. Hence, it is recommended to consider the size of a key so that it can be stored in one page if possible.

Overflow OID

Now let's look at a situation in which keys are overlapped.

Assume gender column is indexed. Low-selectivity of the gender field (only two values "M" and "F", or DISTINCT(column) is low) may result in many OIDs being stored in the same node in B+ tree for the same index key. If space necessary for storing OID is bigger than a page by 1/4, it will be stored in a separate Overflow OID (OOID) page. If the number of separate overflow pages increases, it would affect the performance as there would be a burden to navigate between these additional links.

The following is a structure of a B+ tree in case overflow files and overflow OID pages exist.

Figure 3: A storage structure of OFID and OOID.

Modifying the Column Value

This process finds the record to be updated and modifies the necessary column data of the record. The following flowchart summarizes this process:

Figure 4: Data update flowchart.

Since RDBMS implements read and write operations in pages, you will need to read or write more than one pages. Data stored in the memory can be read/written faster than the data stored on a disk, thus you should minimize disk I/O operations by maximizing the memory.

To do this, you can use a memory buffer, which is an intermediate storage space.  As shown in the above Figure 4, first, check if the necessary data is in the memory buffer. If there is no data, load it to the memory buffer from the disk (DB Volume). Then update the data in the memory. When the data is updated, the data to be modified is recorded in logs, and, finally, it is recorded to the database volume (disk) at the checkpoint.

Memory and Disk Management

The following figure summarizes the policy and main tasks used for memory and disk management.

Figure 5: Work that occurs when storing updated data.

Page Management Policy

LRU (Least-Recently Used)

In order to effectively use the limited buffer size and maintain recently used data, remove the data which has not been used for a long time and free up the space. This is a memory method used by the STEAL policy.

Tasks of Buffer Administrator

FIX

The task acquires a latch for a buffer in competition with other transactions when reading a page from a disk using a memory buffer. Therefore, you cannot use different transactions while acquiring a latch for a buffer to be used in your transaction. If there is a corresponding page in a buffer, a page is not loaded from a disk (database volume).

UNFIX

This is a different concept from FIX in a way that the task is for a page that is no longer used in a transaction. It is not always the case that an UNFIXed page is flushed to a disk.

FLUSH

In this task you will write a DIRTY page in a buffer to a disk. As you have to write a log file (WAL - Write-Ahead Log) to a disk, just before executing the task a race can occur in which two or more files write in the same log file.

Logging Policy to Restore Failure or Rollback

WAL (Write-Ahead Logging)

Write-Ahead Logging is a policy to enable restoration upon system failure, i.e. always write UNDO/REDO logs to a disk before writing the data page. UNDO includes the data before UPDATE and REDO includes the new data after UPDATE.

You can use STEAL/NO STEAL, FORCE/NO FORCE policies to specify when to reflect the page stored in a memory buffer to a disk.

STEAL/NO STEAL

When a transaction attempts to use a memory buffer, the STEAL policy flushes the Least Recently Used (LRU) dirty pages and frees up the memory buffer. To use the STEAL policy, you need UNDO logging in order to recover old data when you rollback a transaction.

In contrast, the NO STEAL policy keeps a DIRTY page in a buffer until a transaction is completed. Therefore, there should be enough buffer space to keep all pages that have been changed by all transactions in progress.

FORCE/NO FORCE

The FORCE policy reflects all pages updated by a transaction to a database volume when commit is called. Using the FORCE policy affects performance, as disk write has to be carried out every time a transaction commits. If a page has to be corrected 20 times by several transactions in a brief period of time, the disk also needs to be written to 20 times.

The NO FORCE policy does not necessarily reflect all pages renewed by a transaction during a commit process. The costs for rewriting a page could be reduced, if other transactions are renewing the same page while commit is not reflected to a database volume. However, to ensure data changes by a successfully committed transaction, REDO logging is necessary in the event of a system failure.

The NO STEAL and FORCE policies will be the easiest implementation, but their performance is the worst. As the STEAL and NO FORCE policies are the best, most databases including CUBRID are implemented via these policies. For this reason both UNDO logging and REDO logging are necessary, and the entire record should be logged not just the column to UPDATE. When UNDO logging, the record before the change is stored as-is and the record after the change is compressed and stored when REDO logging.

You must note that a big record size will affect logging, even if the size of column to UPDATE is small. If there are a lot of records that are very big and the number of columns to UPDATE frequently is small, it is preferable to gather these UPDATE-able columns and create a separate table for them. This is recommended only considering the size of logging records.

Page Storage Structure

Now let's learn about how a page is stored depending on the size of data to UPDATE. The following figure shows the basic structure where a page is stored in the record:

Figure 6: Record storage structure inside a page.

When a new value is entered, records are assigned from the front of a page while slots are assigned from the end of a page in the opposite direction. One record consists of a header (header), fixed-length columns (Fix Col), and variable-length columns (Var Col). The following explains each component.

• page header: Stores the following information about total slots within a page
• a number of slots and records
• the size of total extra space and contiguous extra space
• the initial offset of contiguous space
• transaction ID
• etc.
• heap page header: Stores initial record location and schema information location.
• record: Stores a record value.
• slot: Stores information about record location (offset, length).

If there is a variable-length column in a record, the size of the record to UPDATE may be changed. If the record to UPDATE is smaller than the existing size, data can be UPDATEd in the same location. However, if the size is bigger, assign the record to UPDATE in the same page, and the existing slot indicates the record to UPDATE. If there is no space in the same page to UPDATE a new record, a separate page will be assigned as shown in the following figure, and the existing record indicates a new record.

Figure 7: The structure of record storage when space to UPDATE in the existing page is insufficient.

If the size of a record to UPDATE is bigger than a page, data is divided and stored by page as shown in the following figure, and the page that stored the previous data indicates the page that stored renewed data.

Figure 8: The structure of a record storage when the record to UPDATE is bigger than a page.

When writing data in this case, data should be split into several pages, and to read, the split pages should be loaded in a series of memory, which may negatively affect the performance.

Exclusive Record Lock

To this point we have explored how CUBRID uses system memory and disk when UPDATE query is requested. Now I will explain how to manage records within the corresponding range when executing an UPDATE query.

Generally, a shared lock is used to search for a specific record to allow other transactions to see [SELECT] the same record you are currently navigating. For INSERT, UPDATE, or DELETE, an exclusive lock is used to block access from other transactions.

In the searching process using WHERE condition, however, you must use an update lock instead of shared lock when acquiring the lock for the record in the target range of UPDATE. You also need to acquire the key of the record in operation and the lock for the next key to block your operating range from other transactions.

We will describe why an additional update lock is necessary besides exclusive lock during UPDATE, and why key locking is needed.

Update Lock

The following explains lock acquisition/release for UPDATE execution.

• Request UPDATE query with the WHERE condition to the database server.
• Acquire update lock while searching for the record with the corresponding range condition. 
• In this case, acquire the key lock for the following corresponding key in the WHERE condition range.
• While executing actual data UPDATE, convert update lock to exclusive lock.
• Commit transactions and release all acquired locks.

The action of an update lock can be explained as:

as this is the only transaction authorized to UPDATE until the UPDATE transaction is completed, other transactions cannot not UPDATE.

Another transaction can acquire an update lock on a transaction that has a record with shared lock. However, if a transaction has acquired update lock, another transaction cannot obtain shared lock.

Concurrency may decrease if an exclusive lock of the record to UPDATE can be obtained immediately, and thus using shared lock seems to be a better option while searching for a target by condition. Why do we have to use an update lock instead of shared lock? The reason is to minimize deadlock. For better understanding, let's look at the following table, which has no update lock.

The situation above is the one in which Transactions 1 and 2 of the same Record A acquired the shared lock, and they are about to upgrade the lock to exclusive lock. However, they both are waiting for the other transaction to free shared lock; that is, they are in deadlock. What would happen if there is an update lock?

Transaction 1 has acquired an update lock and Transaction 2 is in the standby mode to obtain an update lock. Transaction 1 changes the update lock to exclusive lock, UPDATEs Record A and releases the lock. Transaction 2 acquires the update lock2, which was in a standby mode, and thus it can execute UPDATE. When the update lock is introduced, you can avoid some of the deadlock effects caused by UPDATE.

The following deals with specific examples of the situations above. Automatic commit mode is released and isolation level is 4 (REPEATABLE READ SCHEMA, READ COMMITTED INSTANCES).

When a transaction is acquiring an update lock, other transactions cannot UPDATE or DELETE the corresponding record. Therefore, if you use an update lock and access the same record above, you can avoid deadlock resulting from UPDATE. An update lock is used not only for UPDATE, but also for DELETE tasks with the WHERE condition that uses index.

Key Lock

An update lock can be obtained even when the record to UPDATE is in the shared lock with other transactions; however, two or more transactions cannot be acquired at the same time.

For data consistency, update lock alone is insufficient. Let's consider the following example.

There are many models that can be used to handle the request above, and approach methods vary depending on RDBMS.

CUBRID 2008 R4.1 places the INSERT task of Transaction 2 in a standby mode. However, Transaction 1 cannot put Transaction 2's tasks in the standby mode by using an update lock as the data of the corresponding key will be deleted from the B+tree when executing DELETE. This problem can be solved by using a key lock.

When performing DML tasks, a key lock acquires lock authority of the record indicated by the next key of the record for processing. If table locking is executed without obtaining a key lock, it will deteriorate concurrency; thus, a lock to a certain area should be obtained to ensure maximum concurrency.

Typical example of using key locks is a unique index. When performing DML tasks, you should obtain a key lock for the record indicated by the next key of the corresponding record and the target record, to ensure there are no DML tasks by other transactions within the range.

Let's learn about the operating process of key locks with the following example: Automatic commit mode is released.

• When Transaction 1 executes UPDATE of the id=40 record, the transaction will acquire the key lock for the id=40 key and id=80, the next key. In this case, other transactions within the range cannot INSERT, UPDATE, or DELETE new records.
• For Transaction 2 to INSERT the id=30 record, it should acquire the key lock for id=40; as Transaction 1 already acquired the key lock for id=40, the work is in standby mode.
• As Transaction 1 already acquired the key lock for id=40 and id=80, all other tasks, which require the work with these keys, are all in standby mode. 

It is likely that other transactions can be in standby mode because of key lock. The more indexes you have on a column you plan to UPDATE, the occurrence of key lock may increase which results in a greater negative effect on UPDATE performance. Thus, we recommend that you remove index for the corresponding columns, considering both SELECT and UPDATE performance.

Figure 9: Key lock related scenario about records being updated.

Considering the occurrence of deadlock between indexes due to key lock, we recommend that you retry the corresponding task. In the following scenario, we will learn about deadlocks resulting from key lock. Deadlock may occur under the assumption that Transaction 1 and 2 are implemented at the same time.

Figure 10: Deadlock scenario caused by key lock.

Transaction 1 attempts to UPDATE the column num of id=1 record and obtains the key lock for id=1 and id=5 as well as the exclusive lock for record (1,9). In this case, Transaction 2 attempts to UPDATE the column id, which is num = 7, and acquires the key lock for num=7 and num=9 as well as the exclusive lock for record (5 and 7).

Transaction 1 needs the key lock for index idx 9 to UPDATE the column num of id=1 record but Transaction 2 has already acquired the corresponding lock. While Transaction 2 needs the key lock for index pk 5 to UPDATE the column id of num=9 record, Transaction 1 has already acquired the corresponding lock. That is, deadlock has occurred.

Key lock has key shared lock and key exclusive lock. The lock that the INSERT task acquires for the next key is key shared lock, and the lock that UPDATEd or DELETEd tasks will acquire for the next key is key exclusive lock. When UPDATE or DELETE is working on the record of a specific range, other transactions must not change the record within the corresponding range; this is to allow INSERT by other transactions into the key lock range of the next location during INSERT operation. Let's examine the following figure:

Figure 11: Key lock related scenario for record being INSERTed.

When Transaction 1 INSERTs id = 90 record, it acquires the key shared lock for id = 100, the next key. When Transaction 2 INSERTs id = 95 record, it acquires the key shard lock for id = 100, which is also the next key. Lock sharing by transactions that are different from one another on the same key id = 100 is possible because key lock is acquired in key shared lock mode.

In Conclusion

In this article we have observed the events that occur in the memory and disk storage structure during UPDATE of data in CUBRID and how to obtain locks so as to block other transactions from the record within the UPDATE range.

It is a mistake to think that the processing costs will be small if just one record or a small column is UPDATEd. When searching for the WHERE condition to find the UPDATE target, yes, the use of index will affect the searching time. But also you must remember that the size of a record also affects UNDO/REDO logging, not the size of UPDATE target column. We can assume that the smaller the size of record to be UPDATEd and the longer the fixed-length compared to the existing size, the more advantageous it is for UPDATE performance.

Under the assumption that UPDATE of a variable-length column occurs always, you can consider adjusting the system parameter unfill_factor (the default value 0.1) value, which controls the disk space of a pre-allocated page for data UPDATE. For example, suppose the value of unfill_factor is 0.1, there will be about 10% of extra space in a page when data INSERT occurs continuously; thus, the remained 10% of extra space could be used for INSERT for the same page or for UPDATE.

UPDATE requires update lock and key lock. You should understand that these could lead to standby or deadlock of competing transactions. If an unexpected delay or deadlock occurs, you should check if an index configuration for UPDATE column is appropriate, or if the number of indexes on the column could be minimized.

I hope that this article and the recommendations I gave to reduce the burden of UPDATE execution provide you an opportunity to understand the locking and storage structure usage methods in CUBRID database when executing UPDATE operations.

By Lee Donghyun, CUBRID Manual and Release Notes writer, CUBRID DBMS Development Lab, NHN Corporation.

So far it has been very successful that I am very excited to announce our first partner jOOQ (Java Object Oriented Querying), an open source SQL query builder with Active Record modeling which provides easy 1:1 mapping between tables/views in your database and classes in your Java application.

According to Lukas Eder, the core developer, jOOQ is a type safe SQL query builder which allows to model standard and vendor-specific SQL constructs directly in a Java fluent API. "Using its code generator, its intuitive API and your favourite IDE for syntax-autocompletion, you can greatly increase your productivity writing and executing SQL in your Java application. Use jOOQ for your migration from MySQL to CUBRID and let SQL be fun again!" says Lukas.

We have already tested jOOQ with CUBRID and created a tutorial for you to quickly get started. Here are the major set of features which make jOOQ very useful for database based Java application developers:

• Code Generation: jOOQ generates a simple Java representation of your database schema. Every table, view, stored procedure, enum, UDT is a class.
• Active records: jOOQ implements an easy-to-use active record pattern. It is NOT an OR-mapper, but provides a 1:1 mapping between tables/views and classes. Between columns and members.
• Typesafe SQL: jOOQ allows for writing compile-time typesafe querying using its built-in fluent API.
• SQL standard: jOOQ supports all standard SQL language features including the more complex UNION's, nested SELECTs, joins, aliasing
• Vendor-specific feature support: jOOQ encourages the use of vendor-specific extensions such as stored procedures, UDT's and ARRAY's, recursive queries, and many more.

When we have asked Lukas about what it was like to add CUBRID Database support to an open source application used by thousands of users, he replied: "At first sight, I was thrilled by the amount of functionality, documentation, and tooling offered by this new emerging open source database. Its integration into jOOQ was straight-forward thanks to the detailed documentation and compatibility with MySQL."

On behalf of CUBRID, I would like to thank Lukas for his valuable contribution to CUBRID open source community. I am sure CUBRID users will greatly benefit from jOOQ's rich functionality.

Stay tuned! We will be announcing more of such interesting open source projects in the coming week.

Information for our readers!

If you develop an open source application and would like to become a CUBRID Partner by supporting CUBRID Database in your project, contact us by email affiliates@cubrid.org. In your letter please provide an overview of your software, project links, and your statement on behalf of your project. We will be very glad to have you onboard!

In this article we will reveal the characteristics of Cassandra, HBase and MongoDB identified through multiple workloads.

Why NoSQL?

The interest in NoSQL continues to rise because the amount of data to process continues to increase. International internet companies, including Google and Facebook, have their own NoSQL solution designed to process the exploding amount of data.

Why are they using NoSQL instead of RDBMS?

Twitter is still using MySQL. Although smaller than Facebook, the international SNS company MySpace uses MS SQL Server as their main storage.

RDBMS is known to experience burden when processing tera or peta unit large sized data. However, this can be resolved through sharding. For instance, NHN’s Café, Blog and News services use RDBMS through sharding.

There is no single correct answer in processing bulk data. Since every situation is different, the company must select the solution that best fits their situation and apply that for seamless service.

Out of the RDBMSs, Oracle is an exception since Oracle’s performance and functions, such as mass data processing or data synchronization, are far more superior to other RDBMS. However, the high cost can be a problem. Depending on the size, it may be more economical to develop a NoSQL than purchasing an Oracle license.

Even if you have enough expertise on how to process bulk data with RDBMS, we will need to have continuous interest and training in NoSQL. NoSQL is gaining popularity because of its ability to process mass data, but it still has many technical (or functional) limitations compared to RDBMS. However, this will be resolved as time passes.

NoSQL provides a non-relational, and in the long run, schema-free data model, allowing the horizontal extension of the system. Instead, it is less structured than RDBMS and does not guarantee ACID. Therefore, after the INSERT operation is completed and SELECT conducted, a different value can be acquired. And after the NoSQL demo has failed and restored, the stored value may be different than before. Operations such as transactions or join cannot be conducted.

While ACID has been abandoned, the flexibility, scalability and usability of data storage have increased. Therefore, it is more suitable for explosive amounts of data processing.

Now let’s see if the NoSQL products really play the expected part for the internet services you are to develop. Why don’t we take a look at the characteristics of the most widely used NoSQL products (Cassandra, HBase, MongoDB) by looking into each of its architectures and benchmark tests that were conducted.

Benchmarking Tests using YCSB

YCSB (Yahoo Cloud Servicing Benchmark) is a test framework developed by Yahoo. It allows to conduct benchmark tests on storages by creating "work loads". Through this benchmark, the storage most suitable for the service that is to be developed can be selected.

Basic operations are Insert, Update, Read, and Scan. There are basic workload sets that combine the basic operations, but new additional workloads can also be created.

YCBS currently supports Cassandra, HBase, MongoDB, Voldemort and JDBC. If tests on other storages are needed, then use YCBS interfaces to test during the development process.

This article contains tests conducted on the following products and versions.

• Cassandra-0.7.4
  Although Cassandra’s latest version is 0.8.0, we have decided to use the previous version known to be stable. Because when testing with the 0.8 version, the gossip protocol between nodes malfunctioned and the node up/down information was incorrect.
• HBase-0.90.2 (Hadoop-0.20-append)
  The HBase-0.90.2 (Hadoop-0.20-append) was selected because, if not the Hadoop-append version, there may be problems on decreased durability in HDFS.
• MongoDB-1.8.1

The test workload is as follows.

• Insert Only
  Enter 50 million 1K-sized records to the empty DB.
• Read Only
  Search the key in the Zipfian Distribution for a one hour period on the DB that contains 50 million 1K-sized records.
• Read & Update
  Conduct read and update one-on-one instead of read under the identical conditions of Read Only.

There are three testing equipments with the same specifications:

• Nehalem 6 Core x 2 CPU, 16GB Memory.

Each conduct replicates and distributes three copies. However, with MongoDB the performance result was abnormally high when both replications and distribution compositions were organized, so the test was conducted only with the replica set. The parameter for each product is shown in the following listings. Other items were set to default.

• Cassandra

• HBase

• MongoDB

The results are as shown in the following figure.

1. All three products showed better throughput in INSERT.
2. Cassandra showed outstanding throughput in INSERT-only with 20,000 ops.
3. HBase also presented relatively good performance in INSERT-only.
4. The products did not show large differences with READ-only than INSERT-only.
5. HBase’s performance was better than Cassandra in READ-only.
6. Cassandra’s performance in READ-and-UPDATE was better than the other two products. Cassandra’s READ-and-UPDATE might have been higher than READ-only due to Cassandra’s excellent insert throughout.
7. MongoDB’s throughout in all three conditions was the lowest of the three products. MongoDB, which uses the Memory Mapped File (mmap), probably showed poor performance because the large data size exceeded the physical memory size.

Cassandra and HBase Architecture

Both Cassandra and HBase are influenced by Google’s BigTable. (Although Cassandra was directly influenced by Amazon’s Dynamo.) BigTable is a Column-Oriented DB that stores data in a Multidimensional Sorted Map format and has the <row key, column family, column key> data structure. Column family is a basic unit that stores data with column groups that are related to each other.

In BigTable, data is written basically with the append method. In other words, when modifying data, the updates are appended to a file, rather than an in-place update in the stored file. The figure below shows BigTable’s data read/insert path.

When a write operation is inserted, it is first placed in a memory space called memtable. If the memtable is full, then the whole data is stored in a file called SSTable (Sorted String Table). Data may be lost if the server collapses before the memtable data is moved to SSTable, so to provide durability it is necessary to save the history of commit logs every time before writing to the memtable. When conducting the read operation, first find the pertaining key in the memtable. If it is not in the memtable, search for it in the SSTable. You may have to search multiple SSTables.

There are advantages in the writing operation if this architecture is used. This is because the ‘writing’ operation is only recorded in the memory and moved to the actual disk only after a certain amount has been accumulated. Therefore, concurrent I/O can be avoided. However, when "reading", if "reading" is done in the SSTable and not in the memtable, then the performance will relatively decrease. Both Cassandra and HBase use bloom filter to quickly judge whether the key exists in the SSTable and creates an index for use. However, if there are many SSTables, then a lot of I/O will be created during the reading operation. Therefore, compaction is done to improve the reading performance. Compaction is where two SSTables merge and sort to become one SSTable, which decreases the number of SSTables. The reading and writing performance improved as more compactions are done.

For these reasons READ and READ-and-UPDATE are much slower than INSERT operations in these NoSQL solutions.

Cassandra

Additionally, Cassandra uses consistent hashing. The HBase and Casandra may have their similarities, there are differences as well. Cassandra prefers AP (Availability & Partition tolerance) while HBase prefers CP (Consistency & Partition tolerance). CAP theorem is a theory in Distributed Computing.

The theory claims that there is no system that provides all three Consistency, Availability and Partition tolerance. For example, replications must be made in multiple nodes to increase usability, and the adjustability between replicated data must be met. However, in order to make operation possible, even during network partitioning, it becomes difficult to guarantee the adjustability between replications or data. Therefore, only a part of the CAP can be provided.

As mentioned before, Amazon’s Dynamo has a direct influence on Cassandra. Dynamo uses consistent hashing to disperse data to the key-value store, and provides high adjustability. Consistent hashing sequences the value (slot) that hashed the key and placed it in a ring format. The multiple nodes that create the cluster processes certain ranges of the ring. Therefore, every time a node in the cluster falls out or comes in, the closed node on the ring can take over the concerned range or distribute the range without rehashing.

Cassandra also raises the usability level with the concept of consistency level. This concept is related to replication. Confirmation of the number of replications and the completion of replication can be adjusted with the system parameter. For example, if three replications are maintained and a write operation is inserted, then the operation is only considered to be successful if the three replications are completed.

However, Cassandra allows only N (under 3) number of executions to be checked and immediately returns to value. Therefore, write can be conducted successfully even with failures in the replication node, which raises usability. Histories of failed write operations are recorded on a different node, and the operation can be retried at a later date (this is called “hinted handoff”). Since the success of replicated writing is not guaranteed, the data suitability is checked in the reading stage. Generally, if there are multiple replications, it is collected into one when reading. However, Cassandra keeps in mind that not all replications match, and reads the data from all three replications, checks whether it is identical, and restores the latest data if it is not suitable (this is called “read repair”).

This is why Cassandra is slower in reading than writing.

HBase

Now let’s look into HBase. HBase has the same structure as BigTable. While BigTable operates on the tablet unit, HBase disperses and replicates with the region unit. Key is arranged according to ranges, and the location of the region, where the key is stored in, is saved as meta data (this is called meta region). Therefore, if a key is inserted, first find the region, and later the client will cache this information. HBase places the writing operation in a single region, without distributing and splits the region if it becomes too large. Adjustments, such as creating regions in advance, must be made for the action above.

HDFS is in charge of data storage and replication. When writing a file with Hadoop’s distribution file system, HDFS writes multiple replications (synchronous replication) and only reads one of them when reading. This guarantees data integrity and lowers reading overhead. However, since HBase and HDFS operates separately, the location of the node that contains memtable and SSTable may change, which may cause additional network I/O.

What happens if there is a failure in the node? If the region server collapses, other nodes take over the tasks which comprised the region. In other words, the commit log that pertains to the region is replayed and memtable is created. If the failed node contains HDFS’s data, then chunk that pertains to the node is migrated to a different node.

Thus, we have looked at the similarities and differences of Cassandra and HBase.

Then why did the two products show a difference in writing during the comparison test?

1. First of all, there is a difference in commit log. Cassandra writes the commit log once in the local file system. However, HBase enter the commit log in the HDFS and HDFS conducts the replication. Data locality is not put into consideration, so the file write request will probably be sent to a different physical node in the network.
2. Second, Cassandra receives the write request on all three nodes, but HBase only ‘writes’ on a single region in the beginning, and receives requests on only one node.

Then why is there a difference in reading performance?

There are various differences, but for one READ request Cassandra reads the data three times while HBase only reads the data once. This places an influence on the throughput.

MongoDB Architecture

MongoDB is unlike Cassandra or HBase eplained earlier. There are too many functional groups that can be classified as NoSQL, which makes it difficult to compare with Cassandra and HBase on the same level.

Cassandra and HBase is for processing full-scale large amounts of data, while MongoDB can be used quickly, schema-free when using a certain amount of data.

MongoDB adopts a documented-oriented format, so it is more similar to RDBMS than a key-value or column oriented format.

MongoDB operates on a memory base and places high performance above data scalability. If reading and writing is conducted within the usable memory, then high-performance is possible. However, performance is not guaranteed if operations exceed the given memory. Therefore, MongoDB should be used as a medium or small-sized storage system.

Configure a test condition similar to the original test on a lower specification server and test 300 thousand records instead of 50 million, and MongoDB will record greater performance than Cassandra or HBase.

Mongo DB uses BSON for data storage. (As known as Binary JSON, but is similar to Hessian than JSON in data encoding format and expressive datatype.)

The logical unit that is expressed using BSON is called document. (When using NPC as an example, it is the same as when the NPC data is recorded on the disk and searched.) Document is a concept that corresponds to a row in RDBMS. The difference is that a document in MongoDB does not have a schema.

Accumulated documents are called collections. A collection corresponds to a table in RDBMS.

Basic query method is to search the collection to find documents that meet the conditions. Each document has a field called _id, which acts as the OID (Object Identifier), and all query results are returned with this _id.

Index can also be created for collections. An index can be prepared for the fields in the document. This index is implemented in B-Tree, similar to that of RDBM’s index. Each collection creates an index for _id by default. One of the strengths of MongoDB is that its functions are similar to RDBMS, in that it supports indexing and JOIN.

Further, we will now look at the structure of MongoDB.

MongoDB Structure

MongoDB is configurable in replication and distribution configuration. The system configuration recommended by MongoDB is three shard servers and three replica sets for each server (in other words, nine nodes). Replications are made possible by Master/Slave or a replica set. Replica set supports automatic failover. It has one master and multiple slaves, and write can only be sent to the master. If the master is terminated, service is resumed after one of the slaves are appointed as a master.

The distribution method provided by MongoDB is sharding. Each collection can shard, and a field in the Collection can designate a shard key. Like HBase, sharding is operated with the Order-preserving partitioning method. Chunk is included in the <collection, minkey, maxkey>, a continuous data range within shard, and can be as big as 64M. Chunks that have exceeded the maximum size can split into two chunks, and the divided chunks can migrate to different shard servers. The following figure contains a picture of MongoDB, which is composed of shards and replica sets.

The client first contacts a process called mongos, which is in charge of routing and coordination. Coordination is done over multiple nodes through requests and the results are collected and returned. The config server retains meta information on shards and chunks.

Now let’s look into MongoDB's internal structure. MongoDB uses the asynchronous method for replication. When a “write” operation is inserted, master stores the history in Oplog. Oplog is a fixed sized, FlfO method collection called capped collection. Logs accumulated in the Oplog are regularly polled by the slave, read, then implemented. If the slave failed to replay the log while the Oplog fills up, then the slave must suspend log-based replications and synchronizes by resyncing the master’s data. One of Oplog’s other distinctive characteristics is that it can conduct the saved operations idempotently when replaying the log. Operation logs are saved in the order it was executed in, so logs can be replayed after that period. Ops such as increment are changed to set ops when logged for this purpose.

MongoDB’s replication and failover method and query power is similar to RDBMS. The most significant difference is that flexibility in data storage is provided to the application while schema-free data is stored.

RDBMS Architecture

Unlike NoSQL, RDBMS is "read" operation oriented. The detailed method differs for each product, but generally NoSQL products are implemented so that mass data can be ‘written’ quickly. On the other hand, RDBMS, which is ‘read’ oriented, not only enables quick reading, but also provides many functions and inquiry features. DBMS can quickly inquire values using indexes, but at the same time indexes slow down ‘writing’. Since an index must be configured for every ‘writing’ operation, the more indexes there are, the more time is needed to configure the indexes.

The difference of RDBMS and NoSQL can be compared with ACID and BASE, but comparisons can also be made with reading and writing.

If the business you are to build requires more reading than writing, and needs complex reading methods, as well as basic aggregate functions such as SUM() or AVG(), then RDBMS might be a better choice than NoSQL.

Summary

We have looked at benchmark tests done on Cassandra, HBase, MongoDB and the differences between the products. Cassandra and HBase show excellent writing abilities, but its reading performance did not meet expectations. This was because unlike existing RDBMSs, the two products were optimized for writing, and this resulted in a lot of concurrent I/O when reading.

MongoDB has similar structures to that of RDBMS and shows great flexibility in data modeling especially for medium and small-sized businesses.

By Lee Hye Jeong, Storage System Development Team, NHN Corporation.

This is a very personal and subjective opinion of mine, but I believe that a person well versed in GC tends to be a better Java developer. If you are interested in the GC process, that means you have experience in developing applications of certain size. If you have thought carefully about choosing the right GC algorithm, that means you completely understand the features of the application you have developed. Of course, this may not be common standards for a good developer. However, few would object when I say that understanding GC is a requirement for being a great Java developer. 

This is the first of a series of "Become a Java GC Expert" articles. I will cover the GC introduction this time, and in the next article, I will talk about analyzing GC status and GC tuning examples from NHN.

The purpose of this article is to introduce GC to you in an easy way. I hope this article proves to be very helpful. Actually, my colleagues have already published a few great articles on Java Internals which became quite popular on Twitter. You may refer to them as well.

Returning back to Garbage Collection, there is a term that you should know before learning about GC. The term is "stop-the-world." Stop-the-world will occur no matter which GC algorithm you choose. Stop-the-world means that the JVM is stopping the application from running to execute a GC. When stop-the-world occurs, every thread except for the threads needed for the GC will stop their tasks. The interrupted tasks will resume only after the GC task has completed. GC tuning often means reducing this stop-the-world time.

Generational Garbage Collection 

Java does not explicitly specify a memory and remove it in the program code. Some people sets the relevant object to null or use System.gc() method to remove the memory explicitly. Setting it to null is not a big deal, but calling System.gc() method will affect the system performance drastically, and must not be carried out. (Thankfully, I have not yet seen any developer in NHN calling this method.)

In Java, as the developer does not explicitly remove the memory in the program code, the garbage collector finds the unnecessary (garbage) objects and removes them. This garbage collector was created based on the following two hypotheses. (It is more correct to call them suppositions or preconditions, rather than hypotheses.) 

• Most objects soon become unreachable.
• References from old objects to young objects only exist in small numbers.

These hypotheses are called the weak generational hypothesis. So in order to preserve the strengths of this hypothesis, it is physically divided into two - young generation and old generation - in HotSpot VM.

Young generation: Most of the newly created objects are located here. Since most objects soon become unreachable, many objects are created in the young generation, then disappear. When objects disappear from this area, we say a "minor GC" has occurred. 

Old generation: The objects that did not become unreachable and survived from the young generation are copied here. It is generally larger than the young generation. As it is bigger in size, the GC occurs less frequently than in the young generation. When objects disappear from the old generation, we say a "major GC" (or a "full GC") has occurred. 

Let's look at this in a chart.

Figure 1: GC Area & Data Flow.

The permanent generation from the chart above is also called the "method area," and it stores classes or interned character strings. So, this area is definitely not for objects that survived from the old generation to stay permanently. A GC may occur in this area. The GC that took place here is still counted as a major GC. 

Some people may wonder:

What if an object in the old generation need to reference an object in the young generation?

To handle these cases, there is something called the a "card table" in the old generation, which is a 512 byte chunk. Whenever an object in the old generation references an object in the young generation, it is recorded in this table. When a GC is executed for the young generation, only this card table is searched to determine whether or not it is subject for GC, instead of checking the reference of all the objects in the old generation. This card table is managed with write barrier. This write barrier is a device that allows a faster performance for minor GC. Though a bit of overhead occurs because of this, the overall GC time is reduced. 

Figure 2: Card Table Structure.

Composition of the Young Generation

In order to understand GC, let's learn about the young generation, where the objects are created for the first time. The young generation is divided into 3 spaces. 

• One Eden space
• Two Survivor spaces

There are 3 spaces in total, two of which are Survivor spaces. The order of execution process of each space is as below:

1. The majority of newly created objects are located in the Eden space.
2. After one GC in the Eden space, the surviving objects are moved to one of the Survivor spaces. 
3. After a GC in the Eden space, the objects are piled up into the Survivor space, where other surviving objects already exist.
4. Once a Survivor space is full, surviving objects are moved to the other Survivor space. Then, the Survivor space that is full will be changed to a state where there is no data at all.
5. The objects that survived these steps that have been repeated a number of times are moved to the old generation.

As you can see by checking these steps, one of the Survivor spaces must remain empty. If data exists in both Survivor spaces, or the usage is 0 for both spaces, then take that as a sign that something is wrong with your system.

The process of data piling up into the old generation through minor GCs can be shown as in the below chart:

Figure 3: Before & After a GC.

Note that in HotSpot VM, two techniques are used for faster memory allocations. One is called "bump-the-pointer," and the other is called "TLABs (Thread-Local Allocation Buffers)." 

Bump-the-pointer technique tracks the last object allocated to the Eden space. That object will be located on top of the Eden space. And if there is an object created afterwards, it checks only if the size of the object is suitable for the Eden space. If the said object seems right, it will be placed in the Eden space, and the new object goes on top. So, when new objects are created, only the lastly added object needs to be checked, which allows much faster memory allocations. However, it is a different story if we consider a multithreaded environment. To save objects used by multiple threads in the Eden space for Thread-Safe, an inevitable lock will occur and the performance will drop due to the lock-contention. TLABs is the solution to this problem in HotSpot VM. This allows each thread to have a small portion of its Eden space that corresponds to its own share. As each thread can only access to their own TLAB, even the bump-the-pointer technique will allow memory allocations without a lock. 

This has been a quick overview of the GC in the young generation. You do not necessarily have to remember the two techniques that I have just mentioned. You will not go to jail for not knowing them. But please remember that after the objects are first created in the Eden space, and the long-surviving objects are moved to the old generation through the Survivor space.

GC for the Old Generation

The old generation basically performs a GC when the data is full. The execution procedure varies by the GC type, so it would be easier to understand if you know different types of GC. 

According to JDK 7, there are 5 GC types. 

1. Serial GC
2. Parallel GC
3. Parallel Old GC (Parallel Compacting GC)
4. Concurrent Mark & Sweep GC  (or "CMS")
5. Garbage First (G1) GC

Among these, the serial GC must not be used on an operating server. This GC type was created when there was only one CPU core on desktop computers. Using this serial GC will drop the application performance significantly. 

Now let's learn about each GC type.

Serial GC (-XX:+UseSerialGC)

The GC in the young generation uses the type we explained in the previous paragraph. The GC in the old generation uses an algorithm called "mark-sweep-compact."

1. The first step of this algorithm is to mark the surviving objects in the old generation.
2. Then, it checks the heap from the front and leaves only the surviving ones behind (sweep).
3. In the last step, it fills up the heap from the front with the objects so that the objects are piled up consecutively, and divides the heap into two parts: one with objects and one without objects (compact).

The serial GC is suitable for a small memory and a small number of CPU cores.

Parallel GC (-XX:+UseParallelGC)

Figure 4: Difference between the Serial GC and Parallel GC.

From the picture, you can easily see the difference between the serial GC and parallel GC. While the serial GC uses only one thread to process a GC, the parallel GC uses several threads to process a GC, and therefore, faster. This GC is useful when there is enough memory and a large number of cores. It is also called the "throughput GC." 

Parallel Old GC(-XX:+UseParallelOldGC)

Parallel Old GC was supported since JDK 5 update. Compared to the parallel GC, the only difference is the GC algorithm for the old generation. It goes through three steps: mark – summary – compaction. The summary step identifies the surviving objects separately for the areas that the GC have previously performed, and thus different from the sweep step of the mark-sweep-compact algorithm. It goes through a little more complicated steps.

CMS GC (-XX:+UseConcMarkSweepGC)

Figure 5: Serial GC & CMS GC.

As you can see from the picture, the Concurrent Mark-Sweep GC is much more complicated than any other GC types that I have explained so far. The early initial mark step is simple. The surviving objects among the objects the closest to the classloader are searched. So, the pausing time is very short. In the concurrent mark step, the objects referenced by the surviving objects that have just been confirmed are tracked and checked. The difference of this step is that it proceeds while other threads are processed at the same time. In the remark step, the objects that were newly added or stopped being referenced in the concurrent mark step are checked. Lastly, in the concurrent sweep step, the garbage collection procedure takes place. The garbage collection is carried out while other threads are still being processed. Since this GC type is performed in this manner, the pausing time for GC is very short. The CMS GC is also called the low latency GC, and is used when the response time from all applications is crucial. 

While this GC type has the advantage of short stop-the-world time, it also has the following disadvantages.

• It uses more memory and CPU than other GC types.
• The compaction step is not provided by default.

You need to carefully review before using this type. Also, if the compaction task needs to be carried out because of the many memory fragments, the stop-the-world time can be longer than any other GC types. You need to check how often and how long the compaction task is carried out.

G1 GC

Finally, let's learn about the garbage first (G1) GC. 

Figure 6: Layout of G1 GC.

If you want to understand G1 GC, forget everything you know about the young generation and the old generation. As you can see in the picture, one object is allocated to each grid, and then a GC is executed. Then, once one area is full, the objects are allocated to another area, and then a GC is executed. The steps where the data moves from the three spaces of the young generation to the old generation cannot be found in this GC type. This type was created to replace the CMS GC, which has causes a lot of issues and complaints in the long term.

The biggest advantage of the G1 GC is its performance. It is faster than any other GC types that we have discussed so far. But in JDK 6, this is called an early access and can be used only for a test. It is officially included in JDK 7. In my personal opinion, we need to go through a long test period (at least 1 year) before NHN can use JDK7 in actual services, so you probably should wait a while. Also, I heard a few times that a JVM crash occurred after applying the G1 in JDK 6. Please wait until it is more stable.

I will talk about the GC tuning in the next issue, but I would like to ask you one thing in advance. If the size and the type of all objects created in the application are identical, all the GC options for WAS used in our company can be the same. But the size and the lifespan of the objects created by WAS vary depending on the service, and the type of equipment varies as well. In other words, just because a certain service uses the GC option "A," it does not mean that the same option will bring the best results for a different service. It is necessary to find the best values for the WAS threads, WAS instances for each equipment and each GC option by constant tuning and monitoring. This did not come from my personal experience, but from the discussion of the engineers making Oracle JVM for JavaOne 2010.

In this issue, we have only glanced at the GC for Java. Please look forward to our next issue, where I will talk about how to monitor the Java GC status and tune GC. 

I would like to note that I referred to a new book released in December 2011 called "Java Performance" (Amazon, it can also be viewed from safari online, if the company provides an account), as well as “Memory Management in the Java HotSpotTM Virtual Machine,” a white paper provided by the Oracle website. (The book is different from "Java Performance Tuning.")

By Sangmin Lee, Senior Engineer at Performance Engineering Lab, NHN Corporation.

Database Expansion

To store and search a volume of data which is so big that it cannot be handled by one database, you must find a way to use multiple databases. Although there are some databases made for distributed environments, such as Cassandra or Dynamo, these have many functional constraints, such as weakness in terms of search range or inability to use the JOIN operations. In order to expand data while using a relatively feature-rich functionality, it is recommended to use RDBMS by sharding the databases.

In the past, the Sharding logic was implemented directly on the application layer, but now there are an increasing number of examples introducing Sharding platforms which allow to move the Sharding logic from the application layer to database or middleware layers. At a core level, Sharding platforms must respond effectively to ever-increasing data without failure, and handle different data characteristics and models depending on services.

In this article I will compare Spock Proxy, the Sharding platform based on MySQL Proxy, Gizzard, created by Twitter, and CUBRID SHARD, native database sharding feature in CUBRID which is set to launch in the first half of 2012. A table of other solutions was previous posted at Database Sharding with CUBRID.

Horizontal Partitioning and Sharding

Horizontal partitioning is a design that divides and stores data, such as a schema, into two or more tables within one database. For example, to handle large data of user messages, such schema may be created so that messages by users from city A are stored in one table, while messages by users from city B are stored in another table. This allows to reduce the size of the index and increases the concurrency of operations. The key point in this approach is that the data is partitioned between tables within a single database.

Sharding, on the other hand, is the distributed approach where data is horizontally partitioned between tables created in physically different databases.  Thus, Sharding is a method to store the messages by users from city A in database A and messages by users from city B in database B. Here each database is called a Shard.

As you have to work on multiple databases, there could be limits to functionality according to circumstances and also a drawback in terms of consistency and replication, including JOIN operations. In many cases, Sharding used to be implemented at the application server level. There have been many attempts to provide this at the platform level. These can be classified into a pattern that operates in an application server, such as Hibernate Shards, a middle tier pattern as CUBRID SHARD, Spock Proxy, and Gizzard, and a pattern that provides the Sharding functionality from a database itself, e.g. nStore or MongoDB.

Middle Tier Sharding Platform

By default, a Sharding platform should consider the following items:

• Database location abstraction
• Scalability
• Monitoring/Ease of operations

Database location abstraction and scalability are different from each other but connected. Database location abstraction ensures that on the application layer you do not need to know which data (which Row) is located in which database. The application is connected only with the Sharding platform. Connecting to a database is what the Sharding platform should do. In addition, Sharding platform should carry out the task to add a replicated storage to a specific Shard (one of the partitioned databases) in order to migrate database for replacement without restarting or changing the application code. When the Sharding platform comes to a stop, it is obvious that an application server will not be able to access the database. For this reason, the Sharding platform should provide redundancy.

For monitoring, the Sharding platform should be able to provide a number of requests and error information according to Shard keys (a standard to determine which data is stored in which database).

Comparison between Spock Proxy and CUBRID SHARD

The key function of the Sharding platform is the CRUD operation (CREATE, READ, UPDATE, DELETE) which chooses one database among many according to 'a standard set by developer' in other words the Sharding strategy. We will compare Spock Proxy, a typical Sharding platform of MySQL, with the CUBRID SHARD platform developed by NHN.

Spock Proxy is a Sharding platform designed based on MySQL Proxy. In MySQL Proxy, you can execute the Lau scrip code, which is written by a developer before and after performing SQL. The primary purpose of using MySQL Proxy is to analyze and modify SQL. To use Spock Proxy, you need to create a MySQL database to manage the information about shards and how the data should be distributed. The row is a rule for Sharding.

Figure 1: Specifying Sharding rules in Spock Proxy.

CUBRID SHARD is the Sharding platform for CUBRID. The uniqueness of CUBRID SHARD is that it can also be used with MySQL as well as Oracle. It is set to be launched in the first half of 2012, and is planned to be used for processing the meta information database system of NDrive service, the cloud storage system developed by NHN.

The following table shows a summary of a comparison between Spock Proxy and CUBRID SHARD.

Table 1 Comparison between Spock Proxy and CUBRID SHARD

Spock Proxy stores Sharding rules in the table in MySQL database (the rest 'universal_db'). The SQL received from an application server is parsed, and checks whether the query has shard keys. If shard keys are provided, the MySQL instance will be identified according to the standard recorded in universal_db, then SQL will be relayed to that MySQL instance. When using this method, you do not need to describe information related to shard keys in SQL, unlike in CUBRID SHARD. Therefore, if you did not use Sharding before for your coding, but recently had to use due to the data increase, you may use Spock Proxy which will work without requiring you to change the SQL in your application. Note that this is limited to cases where there is no need to change the schema for Sharding, or when Sharding can be applied without changing the SQL that you use. However, the method used to find Shard after parsing SQL, as used by Spock Proxy, has weaknesses in terms of performance. It could lead to unnecessary work, as SQL should be parsed twice: once by Spock Proxy to determine the MySQL instance, then by MySQL itself.

In CUBRID SHARD, HINT is used which allows to avoid parsing SQL twice. Suppose there is a table as follows:

To use the student_no column data as a shard key, an application server sends the following prepared SQL to CUBRID SHARD.

CUBRID SHARD shows that if a HINT displays /*+ SHARD_KEY */ in SQL, the column data with the corresponding HINT can be used as a Shard key. It then reads the student_no value, which follows the hint, and identifies RDBMS based on the configuration file and transmits the corresponding query. As such, the benefit of using HINT is that you can improve processing efficiency by avoiding parsing SQL twice, and react to various RDBMS without violating the database location abstract.

1. CUBRID SHARDING provides various HINTs, in addition to /*+ SHARD_KEY */.
2. Typically, there is /*+ SHARD_ID(__id__) */, a HINT that allows you to find a special shard.
3. The /*+ SHARD_VAL(__shard_key_val__) */ HINT can also play a role in finding a special shard for tables which have no shard keys. While this HINT is the same in that it searches for tables that have no specific shard keys, it configures a value for shard key column directly without choosing a shard, and selects the shard according to the internal rules of the middleware.

Unlike Spock Proxy where Sharding rules are inserted into a database table, in CUBRID SHARD Sharding rules are specified in the configuration file. If there are three access addresses for actual RDBMS storage, you should specify DB addresses to CUBRID SHARD as follows.

If you want to use the Modulo method, write the corresponding value in the configuration file as below.

In addition, specify [MIN..MAX] according to the value generated from SHARD_KEY_MODULAR, and describe which shard to send the corresponding query to.

If you need a more subtle method than Modulo, you can create program code of your own that calculates the Shard ID for Shard keys.

As shown in the above example, you can register your own library which will calculate the Shard ID logic.

The common weakness of both Spock Proxy and CUBRID SHARD is that they both require additional network IO time for each additional hop because they are implemented as a middle tier.

The following figure displays a general process of internal execution in CUBRID SHARD, which is performed when a developer executes a query. When a developer executes a query, the query is analyzed by the DB shard middleware, which determines to which shard it will be sent. Then the query is transmitted to the selected shard, and finally the middleware delivers the response to the client.

Figure 2: CUBRID SHARD process.

Gizzard

Gizzard is a Sharding platform developed by Twitter. It is also a middle tier just like Spock Proxy and CUBRID SHARD. However, its usage and architecture are quite different from Spock Proxy or CUBRID SHARD.

The following figure shows how the data can be shared between several databases deploying two units of Gizzard.

Figure 3: Gizzard deployment diagram.

In this case, the interface being used by an application server is Thrift, an RPC protocol, not JDBC. Therefore, it is similar to DBGW platform developed by NHN. When the schema of a storage changes, Gizzard may also need modifications. While it can be a constraint, it could also open up access not only to RDBMS, but also to various other databases (e.g. Lucene).

Gizzard is written based on Scala, and you can expand its functions by adding Scala codes as needed. Instead of viewing it as a complete product, we recommend that you download the source and modify it to suit your needs.

The biggest advantage of Gizzard is that you can perform hotspot response and database migration in the middle tier platform level. For example, Gizzard provides the direct replication feature, as shown in the following figure.

Figure 4: The replication feature of Gizzard.

Spock Proxy or CUBRID SHARD do not offer this feature, as they do not need it. CUBRID, MySQL, and Oracle have their own replication feature, thus they do not need to offer the feature from the middle tier.

The replication feature provided by Gizzard is very simple - send each request to several replicas - but cannot handle variety of requests that may occur while running the replication feature. However, it can be useful when using a database without the replication feature.

Figure 5: The migration feature of Gizzard.

As shown in the figure above, it offers the replication feature which does not require a service halt. When migrating a certain database, the feature replicates the details of a database storage before replication configuration, after allowing the new data to be written in two locations by configuring the replication. This approach, of course, cannot be used on RDBMS due to consistency issue.

Limitations of Sharding

Sharding, on the other hand, also has its own limitations. Typical constraints of Sharding are as follows:

• It cannot perform JOIN operations for two or more Shardings.
• auto increment (serial) values can vary depending on Sharding.
• last_insert_id() value is no more valid.
• shard key column value must not be updated; requires delete, then insert.
• does not allow to access two or more Shards from one transaction.

When using Sharding, therefore, it is important that you right perform data modeling and schema design to prevent the above constraint issues.

To learn more about Sharding and other features of CUBRID I suggest you to see the Slideshare presentation at http://www.slideshare.net/cubrid/growing-in-the-wild-the-story-by-cubrid-database-developers.

By Jeon Won Hee, Senior Software Engineer at CUBRID DBMS Development Lab, NHN Corporation.

When there is an obstacle, or when a Java based Web application is running much slower than expected, we need to use thread dumps. If thread dumps feel like very complicated to you, this article may help you very much. Here I will explain what threads are in Java, their types, how they are created, how to manage them, how you can dump threads from a running application, and finally how you can analyze them and determine the bottleneck or blocking threads. This article is a result of long experience in Java application debugging.

Java and Thread

A web server uses tens to hundreds of threads to process a large number of concurrent users. If two or more threads utilize the same resources, a contention between the threads is inevitable, and sometimes deadlock occurs.

Thread contention is a status in which one thread is waiting for a lock, held by another thread, to be lifted. Different threads frequently access shared resources on a web application. For example, to record a log, the thread trying to record the log must obtain a lock and access the shared resources.

Deadlock is a special type of thread contention, in which two or more threads are waiting for the other threads to complete their tasks in order to complete their own tasks.

Different issues can arise from thread contention. To analyze such issues, you need to use the thread dump. A thread dump will give you the information on the exact status of each thread.

Background Information for Java Threads

Thread Synchronization

A thread can be processed with other threads at the same time. In order to ensure compatibility when multiple threads are trying to use shared resources, one thread at a time should be allowed to access the shared resources by using thread synchronization.

Thread synchronization on Java can be done using monitor. Every Java object has a single monitor. The monitor can be owned by only one thread. For a thread to own a monitor that is owned by a different thread, it needs to wait in the wait queue until the other thread releases its monitor.

Thread Status

In order to analyze a thread dump, you need to know the status of threads. The statuses of threads are stated on java.lang.Thread.State.

Figure 1: Thread Status.

• NEW: The thread is created but has not been processed yet.
• RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITING status due to the OS's resource distribution.)
• BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock.
• WAITING: The thread is waiting by using a wait, join or park method.
• TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.) 

Thread Types

Java threads can be divided into two:

1. daemon threads;
2. and non-daemon threads.

Daemon threads stop working when there are no other non-daemon threads. Even if you do not create any threads, the Java application will create several threads by default. Most of them are daemon threads, mainly for processing tasks such as garbage collection or JMX.

A thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.)

Getting a Thread Dump

We will introduce the three most commonly used methods. Note that there are many other ways to get a thread dump. A thread dump can only show the thread status at the time of measurement, so in order to see the change in thread status, it is recommended to extract them from 5 to 10 times with 5-second intervals.

Getting a Thread Dump Using jstack

In JDK 1.6 and higher, it is possible to get a thread dump on MS Windows using jstack.

Use PID via jps to check the PID of the currently running Java application process.

Use the extracted PID as the parameter of jstack to obtain a thread dump.

A Thread Dump Using jVisualVM

Generate a thread dump by using a program such as jVisualVM.

Figure 2:  A Thread Dump Using visualvm.

The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file.

Generating in a Linux Terminal

Obtain the process pid by using ps -ef command to check the pid of the currently running Java process.

Use the extracted pid as the parameter of kill –SIGQUIT(3) to obtain a thread dump.

Thread Information from the Thread Dump File

"pool-1-thread-13" prio=6 tid=0x000000000729a000 nid=0x2fb4 runnable [0x0000000007f0f000] java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)

              at java.net.SocketInputStream.read(SocketInputStream.java:129)

              at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

              at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

              at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

              - locked <0x0000000780b7e688> (a java.io.InputStreamReader)

              at java.io.InputStreamReader.read(InputStreamReader.java:167)

              at java.io.BufferedReader.fill(BufferedReader.java:136)

              at java.io.BufferedReader.readLine(BufferedReader.java:299)

              - locked <0x0000000780b7e688> (a java.io.InputStreamReader)

              at java.io.BufferedReader.readLine(BufferedReader.java:362)

• Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number).
• Priority: Represents the priority of the threads.
• Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.)
• Thread status: Represents the status of the threads.
• Thread callstack: Represents the call stack information of the threads. 

Thread Dump Patterns by Type

When Unable to Obtain a Lock (BLOCKED)

This is when the overall performance of the application slows down because a thread is occupying the lock and prevents other threads from obtaining it. In the following example, BLOCKED_TEST pool-1-thread-1 thread is running with <0x0000000780a000b0> lock, while BLOCKED_TEST pool-1-thread-2 and BLOCKED_TEST pool-1-thread-3 threads are waiting to obtain <0x0000000780a000b0> lock.

Figure 3: A thread blocking other threads. 

"BLOCKED_TEST pool-1-thread-1" prio=6 tid=0x0000000006904800 nid=0x28f4 runnable [0x000000000785f000]
   java.lang.Thread.State: RUNNABLE
                at java.io.FileOutputStream.writeBytes(Native Method)
                at java.io.FileOutputStream.write(FileOutputStream.java:282)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                - locked <0x0000000780a31778> (a java.io.BufferedOutputStream)
                at java.io.PrintStream.write(PrintStream.java:432)
                - locked <0x0000000780a04118> (a java.io.PrintStream)
                at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
                at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
                at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
                - locked <0x0000000780a040c0> (a java.io.OutputStreamWriter)
                at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
                at java.io.PrintStream.newLine(PrintStream.java:496)
                - locked <0x0000000780a04118> (a java.io.PrintStream)
                at java.io.PrintStream.println(PrintStream.java:687)
                - locked <0x0000000780a04118> (a java.io.PrintStream)
                at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:44)
                - locked <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState)
                at com.nbp.theplatform.threaddump.ThreadBlockedState$1.run(ThreadBlockedState.java:7)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - <0x0000000780a31758> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"BLOCKED_TEST pool-1-thread-2" prio=6 tid=0x0000000007673800 nid=0x260c waiting for monitor entry [0x0000000008abf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:43)
                - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState)
                at com.nbp.theplatform.threaddump.ThreadBlockedState$2.run(ThreadBlockedState.java:26)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - <0x0000000780b0c6a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"BLOCKED_TEST pool-1-thread-3" prio=6 tid=0x00000000074f5800 nid=0x1994 waiting for monitor entry [0x0000000008bbf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:42)
                - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState)
                at com.nbp.theplatform.threaddump.ThreadBlockedState$3.run(ThreadBlockedState.java:34)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - <0x0000000780b0e1b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

When in Deadlock Status

This is when thread A needs to obtain thread B's lock to continue its task, while thread B needs to obtain thread A's lock to continue its task. In the thread dump, you can see that DEADLOCK_TEST-1 thread has 0x00000007d58f5e48 lock, and is trying to obtain 0x00000007d58f5e60 lock. You can also see that DEADLOCK_TEST-2 thread has 0x00000007d58f5e60 lock, and is trying to obtain 0x00000007d58f5e78 lock. Also, DEADLOCK_TEST-3 thread has 0x00000007d58f5e78 lock, and is trying to obtain 0x00000007d58f5e48 lock. As you can see, each thread is waiting to obtain another thread's lock, and this status will not change until one thread discards its lock.

Figure 4: Threads in a Deadlock status.

"DEADLOCK_TEST-1" daemon prio=6 tid=0x000000000690f800 nid=0x1820 waiting for monitor entry [0x000000000805f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
                - waiting to lock <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
                - locked <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135)

   Locked ownable synchronizers:
                - None

"DEADLOCK_TEST-2" daemon prio=6 tid=0x0000000006858800 nid=0x17b8 waiting for monitor entry [0x000000000815f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
                - waiting to lock <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
                - locked <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135)

   Locked ownable synchronizers:
                - None

"DEADLOCK_TEST-3" daemon prio=6 tid=0x0000000006859000 nid=0x25dc waiting for monitor entry [0x000000000825f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
                - waiting to lock <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
                - locked <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135)

   Locked ownable synchronizers:
                - None

When Continuously Waiting to Receive Messages from a Remote Server

The thread appears to be normal, since its state keeps showing as RUNNABLE. However, when you align the thread dumps chronologically, you can see that socketReadThread thread is waiting infinitely to read the socket.

Figure 5: Continuous Waiting Status.

"socketReadThread" prio=6 tid=0x0000000006a0d800 nid=0x1b40 runnable [0x00000000089ef000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
                at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
                - locked <0x00000007d78a2230> (a java.io.InputStreamReader)
                at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107)
                - locked <0x00000007d78a2230> (a java.io.InputStreamReader)
                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93)
                at java.io.InputStreamReader.read(InputStreamReader.java:151)
                at com.nbp.theplatform.threaddump.ThreadSocketReadState$1.run(ThreadSocketReadState.java:27)
                at java.lang.Thread.run(Thread.java:662)

When Waiting

The thread is maintaining WAIT status. In the thread dump, IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue, then the thread status will not change.

Figure 6: Waiting status.

"IoWaitThread" prio=6 tid=0x0000000007334800 nid=0x2b3c waiting on condition [0x000000000893f000]
   java.lang.Thread.State: WAITING (parking)
                at sun.misc.Unsafe.park(Native Method)
                - parking to wait for  <0x00000007d5c45850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:440)
                at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:629)
                at com.nbp.theplatform.threaddump.ThreadIoWaitState$IoWaitHandler2.run(ThreadIoWaitState.java:89)
                at java.lang.Thread.run(Thread.java:662)

When Thread Resources Cannot be Organized Normally

Unnecessary threads will pile up when thread resources cannot be organized normally. If this occurs, it is recommended to monitor the thread organization process or check the conditions for thread termination.

Figure 7: Unorganized Threads.

How to Solve Problems by Using Thread Dump

Example 1: When the CPU Usage is Abnormally High

1. Extract the thread that has the highest CPU usage.
   
   [user@linux ~]$ ps -mo pid.lwp.stime.time.cpu -C java
   
        PID         LWP          STIME                  TIME        %CPU
   10029               -         Dec07          00:02:02           99.5
            -       10039        Dec07          00:00:00              0.1
            -       10040        Dec07          00:00:00           95.5

   From the application, find out which thread is using the CPU the most.

   Acquire the Light Weight Process (LWP) that uses the CPU the most and convert its unique number (10039) into a hexadecimal number (0x2737).

2. After acquiring the thread dump, check the thread's action.

   Extract the thread dump of an application with a PID of 10029, then find the thread with an nid of 0x2737.

   "NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable [0x49aa5000]
   java.lang.Thread.State: RUNNABLE
                   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
                   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
                   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
                   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
                   - locked <0x74c52678> (a sun.nio.ch.Util$1)
                   - locked <0x74c52668> (a java.util.Collections$UnmodifiableSet)
                   - locked <0x74c501b0> (a sun.nio.ch.EPollSelectorImpl)
                   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
                   at external.org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
                   at external.org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:708)
                   at external.org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
                   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                   at java.lang.Thread.run(Thread.java:662)
   

   Extract thread dumps several times every hour, and check the status change of the threads to determine the problem.

Example 2: When the Processing Performance is Abnormally Slow  

After acquiring thread dumps several times, find the list of threads with BLOCKED status.

" DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000]
java.lang.Thread.State: BLOCKED (on object monitor)
                at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
                - waiting to lock <0xe0375410> (a beans.ConnectionPool)
                at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
                at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)

"DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020]
java.lang.Thread.State: BLOCKED (on object monitor)
                at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
                - waiting to lock <0xe0375410> (a beans.ConnectionPool)
                at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
                at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)

" DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080]
java.lang.Thread.State: RUNNABLE
                at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570)
                - waiting to lock <0xe03ba2e0> (a oracle.jdbc.driver.OracleConnection)
                at beans.ConnectionPool.getConnection(ConnectionPool.java:112)
                - locked <0xe0386580> (a java.util.Vector)
                - locked <0xe0375410> (a beans.ConnectionPool)
                at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66)
                at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120)

Acquire the list of threads with BLOCKED status after getting the thread dumps several times.

If the threads are BLOCKED, extract the threads related to the lock that the threads are trying to obtain.

Through the thread dump, you can confirm that the thread status stays BLOCKED because <0xe0375410> lock could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock.

There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working, they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them, you will often see that some of the threads that were BLOCKED previously are in a different state.

The second reason is the abnormal connection. When the connection with DBMS stays abnormal, the threads wait until the time is out. In this case, even after extracting the thread dumps several times and comparing them, you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values, such as the timeout value, you can shorten the time in which the problem occurs.

Coding for Easy Thread Dump

Naming Threads

When a thread is created using java.lang.Thread object, the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object, the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application, if all the threads still have their default names, analyzing them becomes very difficult, because it is difficult to distinguish the threads to be analyzed.

Therefore, you are recommended to develop the habit of naming the threads whenever a new thread is created. 

When you create a thread using java.lang.Thread, you can give the thread a custom name by using the creator parameter.

When you create a thread using java.util.concurrent.ThreadFactory, you can name it by generating your own ThreadFactory. If you do not need special functionalities, then you can use MyThreadFactory as described below:

Obtaining More Detailed Information by Using MBean

You can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps, by using ThreadInfo.

You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo, and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time.

In Conclusion

In this article I was concerned that for developers with a lot of experience in multi-thread programming, this material may be common knowledge, whereas for less experienced developers, I felt that I was skipping straight to thread dumps, without providing enough background information about the thread activities. This was because of my lack of knowledge, as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers.

By Tae Jin Gu, Software Engineer at Web Platform Development Lab, NHN Corporation.