CUBRID Blog

PostgreSQL at a glance

2013-05-13T23:33:08-07:00

PostgreSQL shows excellent functionalities and performance. Considering its high quality, it may seem strange that PostgreSQL is not more popular. However, PostgreSQL continues to make progress. This article will discuss this database.

Why You Should Know about PostgreSQL

PostgreSQL is an RDBMS, which is popular mainly in North America and Japan. It is not used much in Korea yet, but as it is a very excellent RDBMS in terms of functionality and performance, it is worth learning about what kind of database PostgreSQL is.

PostgreSQL (pronounced as [Post-Gres-Q-L]) is an object-relational database system (ORDBMS), and is an open-source DBMS that provides the enterprise-level DBMS functionalities and many other functionalities you can find only in advanced DBMS. PostgreSQL is also known as an open-source DBMS that Oracle users can adapt themselves to the most easily, as it has many functionalities similar to those of Oracle.

History

There were many ancestors of PostgreSQL, and of them, Ingres (INteractive Graphics REtrieval System) can be said to be the progenitor of PostgreSQL. Ingres was a project launched by Michael Stonebraker (Picture 1), a great master in the area of databases who is still working hard even today.

Picture 1: Michael Stonebraker started Ingres project.

The Ingres project was launched at Berkeley University in the US in 1977. After Ingres Michael Stonebraker had started another project called Postgres (Post-Ingres). As Postgres version 3 was released in 1991, its user base grew to be quite large. But as the burden of providing support to users became too high, the project was terminated in 1993 (Postgres is known to have had a huge influence on the current Informix product, even after the end of the project. Illustra, a commercial version of POSTGRES, was taken over by Informix in 1997, and then by IBM in 2001.).

Figure 1: Product History.

Despite the project having ended, Postgres users and students continued its development and finally created Postgres95, which achieved 40% better performance than Postgres by supporting SQL and improving its structure.

When Postgre95 became an open-source system in 1996, it was given the name PostgreSQL, its current name, to reflect the fact that it succeeded Postgres and supports SQL (Postgres supported a language called QUEL instead of SQL). In 1997, PostgreSQL was finally released after determining its first version as 6.0.

Since then, PostgreSQL has been actively developed to this day through an open-source community, and the latest release is 9.2, as of May 2013. In addition, due to its open license (like the BSD or MIT license, PostgreSQL allows commercial use and modification, but it also clarifies that the original developers are not liable for any problem that may occur in its use), there have been more than 20 various forks, some of which have had an influence on PostgreSQL and some of which have disappeared.

PostgreSQL's logo is an elephant named 'Slonik' (a baby elephant in Russian language). The true reason why an elephant was used for the logo is not known, but it has been said that just after it became an open-source system, one of its users was inspired by Agatha Christie's novel "Elephants Can Remember" and suggested it. Since then, the elephant logo has been visible at every official PostgreSQL event.

As elephants are thought of as large, strong, reliable and have a good memory, Hadoop and Evernote also use an elephant as their official logo.

Functionalities and Limitations

PostgreSQL supports transaction and ACID, which are the basic functionalities of a relational DBMS. Moreover, PostgreSQL also has many progressive functionalities or expanded functionalities for academic research as well as for basic reliability and stability. Even a general list of PostgreSQL functionalities includes a large number of functionalities.

Nested transactions (savepoints)
Point in time recovery
Online/hot backups, Parallel restore
Rules system (query rewrite system)
B-tree, R-tree, hash, GiST method indexes
Multi-Version Concurrency Control (MVCC)
Tablespaces
Procedural Language
Information Schema
I18N, L10N
Database & Column level collation
Array, XML, UUID type
Auto-increment (sequences),
Asynchronous replication
LIMIT/OFFSET
Full text search
SSL, IPv6
Key/Value storage
Table inheritance

In addition to these, it features a variety of functionalities and new functionalities of enterprise-level DBMS.

In general, PostgreSQL has the following limits:

Table 1: Basic Limits of PostgreSQL.
Limit	Value
Max. Database Size	Unlimited
Max. Table Size	32 TB
Max. Row Size	1.6 TB
Max. Field Size	1 GB
Max. Rows per Table	Unlimited
Max. Columns per Table	250~1600
Max. Indexes per Table	Unlimited

Roadmap

As of May 2013, the latest release is 9.2. Figure 2 provides some brief information on the progress of PostgreSQL by year.

Figure 2: Progress of PostgreSQL by Year.

The main functionalities of each version are as follows:

Table 2: Main Functionalities by Version.
Version	Release Year	Main Functionalities
0.01	1995	Postgres95 release
1.0	1995	Copyright change, open source
6.0~6.5	1997~1999	Renamed PostgreSQL Index, VIEWs and RULEs Sequences, Triggers Genetic Query Optimizer Constraints, Subselect MVCC, JDBC interface,
7.0~7.4	2000~2010	Foreign keys SQL92 syntax JOINs Write-Ahead Log Information Schema, Internationalization
8.0~8.4	2005~2012	Native Support for MS Windows Savepoint, Point-in-time recovery Two-phase commit Table spaces, Partitioning Full text search Common table expressions (CTE) SQL/XML, ENUM, UUID Type Window functions Per-database collation Replication, Warm standby
9.0	2010-09	Streaming replication, Hot standby Support for 64bit MS Windows Per-column conditional trigger
9.1	2011-09	Functionality differentiation Synchronous replication Per-column collations Unlogged tables K-nearest-neighbor indexing Serializable isolation level Writeable CTE (WITH) SQL/MED External Data SE-Linux integration
9.2	2012-09	Performance optimization linear scalability to 64 cores Reduction in CPU power consumption Cascade streaming replication JSON, Range Type Improved lock management Space-partitioned GiST index Index-only scans (covering)

The next PostgreSQL release under development is PostgreSQL 9.3, which is due to be released in the third quarter of 2013. This release features many functionalities, including an enhanced management functionality, parallel query, MERGE/UPSERT, multi-master replication, materialized view, and enhanced multi-language support.

Internal Structure

The following shows the process structure:

Figure 3: Process Structure.

If the client requests connection with the server through the (1) interface library (variety of Interfaces Including libpg, JDBC and ODBC), the Postmaster process relays connection with the server (2). Then, the client executes a query through connection with the allocated server (Figure 3).

The following shows the process of query execution in the server:

Figure 4: Query Execution Procedure.

If it receives a query request from the client, the system creates a parse tree through the syntax analytics process (1), starts a new transaction through the semantic checking process (2) and creates a query tree.

Next, a query tree is re-generated according to the rules defined in the server (3), and of the many available execution plans, the most optimized plan tree is created (4). The server executes this (5) and sends the result of the requested query to the client.

While the server executes a query, a system catalog in the database is frequently used. In the system catalog, users can directly define the type of functions and data, as well as index access methods and rules. In PostgreSQL, therefore, a system catalog is utilized as an important point in adding or expanding its functionalities.

A file that stores data consists of multiple pages, and a single page has a scalable slotted page structure (Figures 5 and 6).

Figure 5: Data Page Structure.

Figure 6: Index Page Structure.

Development Process

The development process model of PostgreSQL can be explained by the following sentence:

‘A community-based open-source project led by a few.’

Like the Linux, Apache and Eclipse projects, the PostgreSQL project is also composed only of a few administrators, a variety of developers and a large number of users. The small administrator group (Core Team) collects requests and feedback (the group sometimes takes a vote to determine priorities at http://postgresql.uservoice.com) from a large number of users, determines the direction of the product, has final approval right for the code and exerts its right for release. This is a different model from corporate management development processes such as MySQL and JBoss.

The developer group consists of code committers and code developers/contributors. They are located in many countries, including the U.S., Japan and Europe.

Figure 7: Distribution of PostgreSQL Developers by Region.

Codes developed by a variety of developers go through a variety of review processes (Submission Review, Usability Review, Feature Test, Performance Review, Coding Review, Architecture Review, Review Review), and are reflected in the product after approval by the Core Team. The mailing list that has been used by the community for a long time is usually used, and a variety of documents, including manuals, are well maintained through the official website.

Products in Competition

PostgreSQL wants to be compared with enterprise-level commercial DBs, but it has been compared mainly with popular open-source DBMSs. The following are the catchphrases of these open-source DBMSs, each of which reflects its features:

PostgreSQL: The world's most advanced open source database
MySQL: The world's most popular open source database
CUBRID: Open Source Database Highly Optimized for Web Applications
Firebird: The true open source database
SQLite: self-contained library, serverless, zero-configuration, transactional SQL database engine

It is not easy to compare these products using their catchphrases alone, but you can see that PostgreSQL seeks progressiveness and openness.

The following is brief comparison of PostgreSQL and its competitiors:

Table 3: Comparison of Products in Competition.
Oracle	An enormous amount of long-proven code and a variety of references. High cost
DB2, MS SQL	Similar to Oracle
MySQL	A variety of applications and references. Corporate development model And the burden of licensing
CUBRID	An alternative to MySQL Built-in HA and database sharding Dual licensing
Other commercial DBs	Other commercial DBs show a downtrend due to open-source DBMSs
Other open source DBs	Struggle to attract developers

For a long time, the PostgreSQL community has made attempts to enter the enterprise DBMS market. In 2004, EnterpriseDB, a company using PostgreSQL, was established, and it is striving to strengthen its position in the enterprise DBMS market. One of the company's main products is Postgres Plus Advanced Server. Postgres Plus Advanced Server was developed by adding Oracle-compatible functionalities (PL/SQL, SQL statements, functions, DB Links, OCI library, etc.) to the open-source PostgreSQL, featuring easy data and application migration and a cost reduction of 20% compared to Oracle (Figure 7).

Figure 8: Cost Reduction Compared to Oracle.

In addition, Postgres Plus Advanced Server provides differentiated services, including a training, consulting and migration, and technical support service from PostgreSQL experts. Through approximately 300 reference sites in a variety of areas, the product is promoted as a database for all industries, with a growing base of users across the world.

Present Status and Trend

As you can see from most posts on PostgreSQL, most PostgreSQL users have a developer-like tendency, and are very loyal to the product.

In fact, they have a good reason for their loyalty. PostgreSQL provides sufficient functionalities and conservative performance compared to other products, and one of its advantages is that it has good enough conditions for beginners to attract new developers.

These good conditions include a well-written manual on the project page, related documents, over 300 reference publications, and over 10 seminars and conferences held in a variety of countries every year. More recently, a PostgreSQL magazine has even appeared. And these are the results of the active PostgreSQL community.

The representative features that PostgreSQL users identify as being important are as follows:

Reliability is the top priority of the product
ACID and transaction
A variety of indexing techniques
Flexible full-text search
MVCC for better concurrency performance
Diverse and flexible replication methods
A variety of procedures (PL/pgSQL, Perl, Python, Ruby, TCL, etc.)/Interface (JDBC, ODBC, C/C++, .Net, Perl, Python, etc.) languages
Excellent community and commercial support
Well-made documents and a thorough manual

A variety of expansion functionalities and ease of development of such functionalities are also advantages of PostgreSQL. The following are the differentiated expansion functionalities of PostgreSQL:

GIS add-on (PostGIS)
Key-Value store expansion (HStore)
DBLink
Support for a variety of functions and types, including Crypto and UUID

There are many other practical and experimental expansion functionalities as well.

Of these, you will see a brief account of GIS (Geographic Information System), which has recently become a hot topic. PostGIS is a middleware expansion functionality that enables PostgreSQL to conform to the OpenGIS standard and support geographic objects (Figure 9).

Figure 9: PostGIS Structure.

PostGIS began to be developed from 2001, and with many functionality and performance improvements, it currently has the most users among the open-source products. There are some commercial products, such as Oracle Spatial, DB2 and MS SQL Server, but the commercial products have not been as well-received in terms of price-performance ratio.

In addition, you can easily find benchmark data that shows that the functionalities and performance of PostGIS/PostgreSQL are worthy of comparison to Oracle.

According to the recent trend, PostgreSQL is also much talked about in relation to cloud as well as GIS. With the recent increase in the number of companies providing DBaaS (Database as a Service), the demand for PostgreSQL, which has advantages in terms of costs and license, has increased, and as such EnterpriseDB has released Postgres Plus Cloud Database in the cloud market, with the following features:

Simple setup & web-based management
Automatic scaling, load balancing and failover
Automated online backup
Database Cloning

It is used in many web services, including Amazon EC2, Eucalyptus cloud, and Red Hat Openshift development platform cloud. Other cloud service providers such as Heroku and dotCloud also provide services using PostgreSQL.

Conclusion

As Sun, which had acquired MySQL, was acquired by Oracle in 2009, MySQL began to be developed as a more closed corporate project, and many MySQL developers left the community around the same time. Afraid of this change, MySQL users are paying attention not only to the forks (MariaDB, Drizzle, Percona, etc.) of MySQL to which they can easily migrate, but also to the migration to PostgreSQL.

Looking at the trend of help-wanted ads related to PostgreSQL and MySQL in the most popular job finding portal http://www.indeed.com (Figure 9), we can see the increase in help-wanted ads related to MySQL is slowing down, while help-wanted ads related to PostgreSQL continue to increase.

Figure 10: Trend of Help-wanted Ads.

According to the trend of search frequency in search sites (Figure 10), MySQL shows a continued downtrend, while PostgreSQL seems to have almost no change. In Korea, however, the search frequency for PostgreSQL has shown an upward trend since mid 2010.

Figure 11: Search Frequency Trend (source).

Of course, the popularity and usage of MySQL is still much higher than PostgreSQL. Although you may not be able to determine the true status or prospects of these products from the above data alone, you could infer that if the popularity of MySQL declines, the popularity of PostgreSQL will increase.

PostgreSQL is not yet powerful enough to surpass MySQL in popularity, but the PostgreSQL open source project community continues to make the following efforts:

Improvement of the reliability of basic DBMS functionalities
Provision of progressive and differentiated functionality expansion
Continuous attraction of more open source developers

In addition, EnterpriseDB, which has stronger business purposes, is also striving to achieve the following objectives:

Expansion of its share in the enterprise market
Expansion of its share in the cloud market
Efforts to replace Oracle and MySQL

By Kim Sung Kyu, Senior Software Engineer at CUBRID DBMS Lab, NHN Corporation.

My impressions after giving talks at RIT++ and Percona conferences

2013-05-12T22:58:31-07:00

Three weeks ago on behalf of the CUBRID team a few of my colleagues and me have attended and given talks at two international conferences. Today I would like to share my impressions of these events. I will write a separate post about various sharding solutions introduced at these conferences. So, stay tuned!

RIT++

The first presentation at RIT++ (Russian Internet Technologies) was held on Monday April 22nd, 2013, in Moscow, Russia. The second one at Percona MySQL Conference & Expo was held on Wednesday the same week on Wednesday April 24th, 2013, in Santa Clara, CA, US. At both conferences the agenda was the same: "Easy MySQL Database Sharding with CUBRID SHARD". At RIT++, though, the presentation was given in Russian language. Very exciting! The following is a list of resouces related to the talks.

This was the third time we, the CUBRID team, have attended the conferences organized by Russian Ontico company. Previously we have attended to RIT++ 2012 and HighLoad++ 2012 conferences. This year at RIT++ 2013 there were over 800 attendees, and 13 categories of talks ranging from client-side development to server-side, to database scalability, to project management, to analytics, and so on. Annually after the conference is over the organizers conduct after-event survey and assess the past experience. I think because of users' past feedback this year RIT++ organizers have accepted more talks related to client-side development than usually.

Besides us from Korea, there were presenters from the States, representing Facebook, and Brasil, representing PUC-Rio University of Brasil. My personal impression was that this year there were fewer foreign speakers than last year at RIT++ or HighLoad++.

At my session about MySQL database sharding with CUBRID SHARD there were over 100, close to 150, I guess attendees. The audience welcomed my speech in Russian language very well. Next time I should talk in Russian again. They like it! When the presentation was over, there were slew of questions. I think CUBRID SHARD as an easy sharding middleware for MySQL was received very well.

To my surprise there were many questions unrelated to CUBRID SHARD. The audience asked a lot about CUBRID itself and its HA feature. Later I learned that many attendees listened to my talks about CUBRID open source relational database system from the last year. One from the audience said that he'd been looking into CUBRID for a while already and was considering to use it in production. His most favorite feature in CUBRID was its built-in support for HA and very clever 3-tier architecture. Overall the unofficial Q&A session lasted for over 1 hour 30 minutes.

It was a great experience for me to present CUBRID SHARD at RIT++ this year and a great opportunity to our CUBRID team. The conference lasted two days, but I could not attend the second day as I had to head to Santa Clara, CA, to give a talk at Percona MySQL Conference & Expo.

Percona

It was the first time I have talked at Percona. Previously we have spoken at OSCON 2011 about CUBRID HA, and 2010 MySQL Conference & Expo about CUBRID Database. When compared to OSCON, Percona MySQL conference was a lot more specific (obviously about MySQL). There were more quality talks about scalability and performance tuning. If I was to choose where to go next year, I would definitly select Percona. That interesting it was!

Unlike at RIT++, our session at Percona conference had attracted only about 20 attendees. The presentation went well, but I should accept that the number of listeners plays a big role. There were fewer questions, less enthusiasm. On the other hand, Facebook, two Percona, Continuent, and Tokutek presentations, which were held at the same time at 3:30 PM, attracted hundreds of listeners each. After realizing this I came to a conclusion that it is the brand recognition that plays a significant role in attracting listeners. Even though NHN is very popular in Korea and Asia in general, it is almost unknown in Western countries. In fact, when I asked the audience at Percona if they had ever heard about NHN, their answer was negative. Very pitty. I think NHN has to seriously reconsider its strategy on increasing its worldwide brand recognition. Nevertheless, I am very glad we had this chance to present our open source sharding middleware at a well-known conference like Percona.

Like I mentioned at the beginning of this post, I will write another post covering various sharding solutions presented at Percona conference. It was very interesting to learn about different techniques used by large scale service providers who have developed their own sharding solutions.

After my presentation was over and I had answered all the questions, I headed to one of the lounge rooms where I had made an oppointment to meet with Ryan Walsh, a Corporate Account Executive at Percona. We have discussed about various opportunities for cooperation between Percona and NHN, the company behind CUBRID development.

Persona is a widely-known and reputable MySQL support and consulting company. It is known to be the oldest and largest independent company which provides not only MySQL support, consulting and training but also develops a custom MySQL server, i.e. provides patches, "backport changes to older MySQL versions to obtain a key patch without a full version upgrade".

During our conversation Rayn had introduced his company and told about large scale cases their company has worked so far. One that I would like to mention today is that some of the services at Amazon Web Services have been actually developed by Percona. Amazon RDS was said to have been developed by Percona team. Percona database tools seem to work with RDS natively.

Also Percona is cooperating with HP to build RedDwarf DaaS as part of the OpenStack open source cloud project. At Percona conference HP engineers have presented how to use RedDwarf APIs to use and administer the features of Percona Server. Such vast knowledge and experience of Percona in bulding cloud database services may be quite benefitial to NHN to develop and provide its own cloud computing service.

Overall, both presentations went well. I have talked to many attendees and answered to quite a lof of their questions about CUBRID SHARD and CUBRID Database. One thing which requires more attention from NHN is its global brand recognition. The more developers will recognize NHN and its services, the more will be eager to listen to and learn from NHN enginneers.

If you have any feedback or suggestions, feel free to comment below. Also you should follow us on twitter here.

NoSQL Benchmarking

2011-09-07T09:27:43-07:00

NoSQL is the talk of the town. And we have already covered what it is for in one of our previous blogs. Today I would like to share the NoSQL benchmark test results we have recently conducted. It will help you to understand if the soon to develop system is compatible to NoSQL, and which NoSQL product to select.

In this article we will reveal the characteristics of Cassandra, HBase and MongoDB identified through multiple workloads.

Why NoSQL?

The interest in NoSQL continues to rise because the amount of data to process continues to increase. International internet companies, including Google and Facebook, have their own NoSQL solution designed to process the exploding amount of data.

Why are they using NoSQL instead of RDBMS?

Twitter is still using MySQL. Although smaller than Facebook, the international SNS company MySpace uses MS SQL Server as their main storage.

RDBMS is known to experience burden when processing tera or peta unit large sized data. However, this can be resolved through sharding. For instance, NHN’s Café, Blog and News services use RDBMS through sharding.

There is no single correct answer in processing bulk data. Since every situation is different, the company must select the solution that best fits their situation and apply that for seamless service.

Out of the RDBMSs, Oracle is an exception since Oracle’s performance and functions, such as mass data processing or data synchronization, are far more superior to other RDBMS. However, the high cost can be a problem. Depending on the size, it may be more economical to develop a NoSQL than purchasing an Oracle license.

Even if you have enough expertise on how to process bulk data with RDBMS, we will need to have continuous interest and training in NoSQL. NoSQL is gaining popularity because of its ability to process mass data, but it still has many technical (or functional) limitations compared to RDBMS. However, this will be resolved as time passes.

NoSQL provides a non-relational, and in the long run, schema-free data model, allowing the horizontal extension of the system. Instead, it is less structured than RDBMS and does not guarantee ACID. Therefore, after the INSERT operation is completed and SELECT conducted, a different value can be acquired. And after the NoSQL demo has failed and restored, the stored value may be different than before. Operations such as transactions or join cannot be conducted.

While ACID has been abandoned, the flexibility, scalability and usability of data storage have increased. Therefore, it is more suitable for explosive amounts of data processing.

Now let’s see if the NoSQL products really play the expected part for the internet services you are to develop. Why don’t we take a look at the characteristics of the most widely used NoSQL products (Cassandra, HBase, MongoDB) by looking into each of its architectures and benchmark tests that were conducted.

Benchmarking Tests using YCSB

YCSB (Yahoo Cloud Servicing Benchmark) is a test framework developed by Yahoo. It allows to conduct benchmark tests on storages by creating "work loads". Through this benchmark, the storage most suitable for the service that is to be developed can be selected.

Basic operations are Insert, Update, Read, and Scan. There are basic workload sets that combine the basic operations, but new additional workloads can also be created.

YCBS currently supports Cassandra, HBase, MongoDB, Voldemort and JDBC. If tests on other storages are needed, then use YCBS interfaces to test during the development process.

This article contains tests conducted on the following products and versions.

Cassandra-0.7.4
Although Cassandra’s latest version is 0.8.0, we have decided to use the previous version known to be stable. Because when testing with the 0.8 version, the gossip protocol between nodes malfunctioned and the node up/down information was incorrect.
HBase-0.90.2 (Hadoop-0.20-append)
The HBase-0.90.2 (Hadoop-0.20-append) was selected because, if not the Hadoop-append version, there may be problems on decreased durability in HDFS.
MongoDB-1.8.1

The test workload is as follows.

Insert Only
Enter 50 million 1K-sized records to the empty DB.
Read Only
Search the key in the Zipfian Distribution for a one hour period on the DB that contains 50 million 1K-sized records.
Read & Update
Conduct read and update one-on-one instead of read under the identical conditions of Read Only.

There are three testing equipments with the same specifications:

Nehalem 6 Core x 2 CPU, 16GB Memory.

Each conduct replicates and distributes three copies. However, with MongoDB the performance result was abnormally high when both replications and distribution compositions were organized, so the test was conducted only with the replica set. The parameter for each product is shown in the following listings. Other items were set to default.

Cassandra

# If a minimum of one succeeds while replicating, then it is returned to application.
Consistency Level, Read=ONE, Write=ONE

# Can be set to periodic(default) or batch.
# Periodically writes (fsync) the commit log on the disk,
# and batch executes regularly (1ms) collecting those to fsync every time.
commitlog_sync: batch, commitlog_sync_batch_window_in_ms: 1

# Degree the key location is cached. When 1.0 - everything is cached.
key_cached=1.0

HBase

HeapSize, HBase=8G, HDFS=2G

MongoDB

# Size of Oplog. Oplog is the log the master accumulated for replication.
--oplogSize=100G

The results are as shown in the following figure.

All three products showed better throughput in INSERT.
Cassandra showed outstanding throughput in INSERT-only with 20,000 ops.
HBase also presented relatively good performance in INSERT-only.
The products did not show large differences with READ-only than INSERT-only.
HBase’s performance was better than Cassandra in READ-only.
Cassandra’s performance in READ-and-UPDATE was better than the other two products. Cassandra’s READ-and-UPDATE might have been higher than READ-only due to Cassandra’s excellent insert throughout.
MongoDB’s throughout in all three conditions was the lowest of the three products. MongoDB, which uses the Memory Mapped File (mmap), probably showed poor performance because the large data size exceeded the physical memory size.

Let’s take a look at why each product showed such difference in performance.

Cassandra and HBase Architecture

Both Cassandra and HBase are influenced by Google’s BigTable. (Although Cassandra was directly influenced by Amazon’s Dynamo.) BigTable is a Column-Oriented DB that stores data in a Multidimensional Sorted Map format and has the <row key, column family, column key> data structure. Column family is a basic unit that stores data with column groups that are related to each other.

In BigTable, data is written basically with the append method. In other words, when modifying data, the updates are appended to a file, rather than an in-place update in the stored file. The figure below shows BigTable’s data read/insert path.

When a write operation is inserted, it is first placed in a memory space called memtable. If the memtable is full, then the whole data is stored in a file called SSTable (Sorted String Table). Data may be lost if the server collapses before the memtable data is moved to SSTable, so to provide durability it is necessary to save the history of commit logs every time before writing to the memtable. When conducting the read operation, first find the pertaining key in the memtable. If it is not in the memtable, search for it in the SSTable. You may have to search multiple SSTables.

There are advantages in the writing operation if this architecture is used. This is because the ‘writing’ operation is only recorded in the memory and moved to the actual disk only after a certain amount has been accumulated. Therefore, concurrent I/O can be avoided. However, when "reading", if "reading" is done in the SSTable and not in the memtable, then the performance will relatively decrease. Both Cassandra and HBase use bloom filter to quickly judge whether the key exists in the SSTable and creates an index for use. However, if there are many SSTables, then a lot of I/O will be created during the reading operation. Therefore, compaction is done to improve the reading performance. Compaction is where two SSTables merge and sort to become one SSTable, which decreases the number of SSTables. The reading and writing performance improved as more compactions are done.

For these reasons READ and READ-and-UPDATE are much slower than INSERT operations in these NoSQL solutions.

Cassandra

Additionally, Cassandra uses consistent hashing. The HBase and Casandra may have their similarities, there are differences as well. Cassandra prefers AP (Availability & Partition tolerance) while HBase prefers CP (Consistency & Partition tolerance). CAP theorem is a theory in Distributed Computing.

The theory claims that there is no system that provides all three Consistency, Availability and Partition tolerance. For example, replications must be made in multiple nodes to increase usability, and the adjustability between replicated data must be met. However, in order to make operation possible, even during network partitioning, it becomes difficult to guarantee the adjustability between replications or data. Therefore, only a part of the CAP can be provided.

As mentioned before, Amazon’s Dynamo has a direct influence on Cassandra. Dynamo uses consistent hashing to disperse data to the key-value store, and provides high adjustability. Consistent hashing sequences the value (slot) that hashed the key and placed it in a ring format. The multiple nodes that create the cluster processes certain ranges of the ring. Therefore, every time a node in the cluster falls out or comes in, the closed node on the ring can take over the concerned range or distribute the range without rehashing.

Cassandra also raises the usability level with the concept of consistency level. This concept is related to replication. Confirmation of the number of replications and the completion of replication can be adjusted with the system parameter. For example, if three replications are maintained and a write operation is inserted, then the operation is only considered to be successful if the three replications are completed.

However, Cassandra allows only N (under 3) number of executions to be checked and immediately returns to value. Therefore, write can be conducted successfully even with failures in the replication node, which raises usability. Histories of failed write operations are recorded on a different node, and the operation can be retried at a later date (this is called “hinted handoff”). Since the success of replicated writing is not guaranteed, the data suitability is checked in the reading stage. Generally, if there are multiple replications, it is collected into one when reading. However, Cassandra keeps in mind that not all replications match, and reads the data from all three replications, checks whether it is identical, and restores the latest data if it is not suitable (this is called “read repair”).

This is why Cassandra is slower in reading than writing.

HBase

Now let’s look into HBase. HBase has the same structure as BigTable. While BigTable operates on the tablet unit, HBase disperses and replicates with the region unit. Key is arranged according to ranges, and the location of the region, where the key is stored in, is saved as meta data (this is called meta region). Therefore, if a key is inserted, first find the region, and later the client will cache this information. HBase places the writing operation in a single region, without distributing and splits the region if it becomes too large. Adjustments, such as creating regions in advance, must be made for the action above.

HDFS is in charge of data storage and replication. When writing a file with Hadoop’s distribution file system, HDFS writes multiple replications (synchronous replication) and only reads one of them when reading. This guarantees data integrity and lowers reading overhead. However, since HBase and HDFS operates separately, the location of the node that contains memtable and SSTable may change, which may cause additional network I/O.

What happens if there is a failure in the node? If the region server collapses, other nodes take over the tasks which comprised the region. In other words, the commit log that pertains to the region is replayed and memtable is created. If the failed node contains HDFS’s data, then chunk that pertains to the node is migrated to a different node.

Thus, we have looked at the similarities and differences of Cassandra and HBase.

Then why did the two products show a difference in writing during the comparison test?

First of all, there is a difference in commit log. Cassandra writes the commit log once in the local file system. However, HBase enter the commit log in the HDFS and HDFS conducts the replication. Data locality is not put into consideration, so the file write request will probably be sent to a different physical node in the network.
Second, Cassandra receives the write request on all three nodes, but HBase only ‘writes’ on a single region in the beginning, and receives requests on only one node.

Then why is there a difference in reading performance?

There are various differences, but for one READ request Cassandra reads the data three times while HBase only reads the data once. This places an influence on the throughput.

MongoDB Architecture

MongoDB is unlike Cassandra or HBase eplained earlier. There are too many functional groups that can be classified as NoSQL, which makes it difficult to compare with Cassandra and HBase on the same level.

Cassandra and HBase is for processing full-scale large amounts of data, while MongoDB can be used quickly, schema-free when using a certain amount of data.

MongoDB adopts a documented-oriented format, so it is more similar to RDBMS than a key-value or column oriented format.

MongoDB operates on a memory base and places high performance above data scalability. If reading and writing is conducted within the usable memory, then high-performance is possible. However, performance is not guaranteed if operations exceed the given memory. Therefore, MongoDB should be used as a medium or small-sized storage system.

Configure a test condition similar to the original test on a lower specification server and test 300 thousand records instead of 50 million, and MongoDB will record greater performance than Cassandra or HBase.

Mongo DB uses BSON for data storage. (As known as Binary JSON, but is similar to Hessian than JSON in data encoding format and expressive datatype.)

The logical unit that is expressed using BSON is called document. (When using NPC as an example, it is the same as when the NPC data is recorded on the disk and searched.) Document is a concept that corresponds to a row in RDBMS. The difference is that a document in MongoDB does not have a schema.

Accumulated documents are called collections. A collection corresponds to a table in RDBMS.

Basic query method is to search the collection to find documents that meet the conditions. Each document has a field called _id, which acts as the OID (Object Identifier), and all query results are returned with this _id.

Index can also be created for collections. An index can be prepared for the fields in the document. This index is implemented in B-Tree, similar to that of RDBM’s index. Each collection creates an index for _id by default. One of the strengths of MongoDB is that its functions are similar to RDBMS, in that it supports indexing and JOIN.

Further, we will now look at the structure of MongoDB.

MongoDB Structure

MongoDB is configurable in replication and distribution configuration. The system configuration recommended by MongoDB is three shard servers and three replica sets for each server (in other words, nine nodes). Replications are made possible by Master/Slave or a replica set. Replica set supports automatic failover. It has one master and multiple slaves, and write can only be sent to the master. If the master is terminated, service is resumed after one of the slaves are appointed as a master.

The distribution method provided by MongoDB is sharding. Each collection can shard, and a field in the Collection can designate a shard key. Like HBase, sharding is operated with the Order-preserving partitioning method. Chunk is included in the <collection, minkey, maxkey>, a continuous data range within shard, and can be as big as 64M. Chunks that have exceeded the maximum size can split into two chunks, and the divided chunks can migrate to different shard servers. The following figure contains a picture of MongoDB, which is composed of shards and replica sets.

The client first contacts a process called mongos, which is in charge of routing and coordination. Coordination is done over multiple nodes through requests and the results are collected and returned. The config server retains meta information on shards and chunks.

Now let’s look into MongoDB's internal structure. MongoDB uses the asynchronous method for replication. When a “write” operation is inserted, master stores the history in Oplog. Oplog is a fixed sized, FlfO method collection called capped collection. Logs accumulated in the Oplog are regularly polled by the slave, read, then implemented. If the slave failed to replay the log while the Oplog fills up, then the slave must suspend log-based replications and synchronizes by resyncing the master’s data. One of Oplog’s other distinctive characteristics is that it can conduct the saved operations idempotently when replaying the log. Operation logs are saved in the order it was executed in, so logs can be replayed after that period. Ops such as increment are changed to set ops when logged for this purpose.

MongoDB’s replication and failover method and query power is similar to RDBMS. The most significant difference is that flexibility in data storage is provided to the application while schema-free data is stored.

RDBMS Architecture

Unlike NoSQL, RDBMS is "read" operation oriented. The detailed method differs for each product, but generally NoSQL products are implemented so that mass data can be ‘written’ quickly. On the other hand, RDBMS, which is ‘read’ oriented, not only enables quick reading, but also provides many functions and inquiry features. DBMS can quickly inquire values using indexes, but at the same time indexes slow down ‘writing’. Since an index must be configured for every ‘writing’ operation, the more indexes there are, the more time is needed to configure the indexes.

The difference of RDBMS and NoSQL can be compared with ACID and BASE, but comparisons can also be made with reading and writing.

If the business you are to build requires more reading than writing, and needs complex reading methods, as well as basic aggregate functions such as SUM() or AVG(), then RDBMS might be a better choice than NoSQL.

Summary

We have looked at benchmark tests done on Cassandra, HBase, MongoDB and the differences between the products. Cassandra and HBase show excellent writing abilities, but its reading performance did not meet expectations. This was because unlike existing RDBMSs, the two products were optimized for writing, and this resulted in a lot of concurrent I/O when reading.

MongoDB has similar structures to that of RDBMS and shows great flexibility in data modeling especially for medium and small-sized businesses.

By Lee Hye Jeong, Storage System Development Team, NHN Corporation.

Comprehensive Overview of Top 14 Content Management Systems

2011-01-20T18:22:43-08:00

Update: added the Distinguished clients for DotNetNuke.

These days many websites (in fact, millions) are implemented by the use of popular open source content management systems (CMS). I would say, for everything you would imagine your site doing, there is one or another CMS which can do that. Thus, the reason for choosing a particular one depends on the tasks you want to accomplish. For blogs there is a common sense of using WordPress blog management system, for large scale websites Drupal CMS is perceived to be a better fit, while Joomla CMS is easier to learn for newbies. .NET developers prefer alternative DonNetNuke or umbraco CMS. Besides these, CMS Made Simple and Liferay are amongst the most popular content management systems based on the number of the number of downloads. Elgg and MODx are amongst the rising stars.

I will not tell you what a CMS is. If you want to learn, see the CMS Wikipedia page. In this blog I will give a comprehensive overview of top 14 content management systems based on the number of weekly downloads, installations, and brand familiarity. The full statistical data can be found in the 2010 OPEN SOURCE CMS MARKET SHARE REPORT by water&stone (2010). The report is distributed under the Creative Commons Attribution-Noncommercial License (3.0).

Today we will cover Drupal, Joomla, MODx, WordPress, DotNetNuke, umbraco, Liferay, TYPO3, CMS Made Simple, MOVABLE TYPE, Plone, eZ Publish, concrete5, and Alfresco content management systems. You might already be familiar with some of these, while some can be new to you. I bet you will be surprised to learn that some unknown for you systems suit your requirements much better than those you have got to use so far (my personal experience). So, let's learn about each of them, and do not forget to leave your comments below on what other CMS should have been included in this list and why.

Note: the order of the CMS does not represent its quality, performance, or superiority over the other. They are numbered to give a better perception to readers. The users feedback and opinions are retrieved from Stack Overflow, thus can be subjective as they are based on personal views and preferences.

# 1: Drupal

Official site: http://drupal.org/

1% of all websites on the Internet are based on this platform. An estimated 7.2 million sites were powered by Drupal as of July 2010. Drupal started out in 2001. In one year from May 2007 to April 2008, Drupal was downloaded from their official site more than 1.4 million times, an increase of approximately 125% from the previous year.

Weekly downloads: 33,671 (ranked #3 after Joomla and before DotNetNuke).

Installations: 575 according to the survey (#3 after WordPres and before DotNetNuke), but 1.4% (#3 after Joomla and before Typo3) of Alexa Top 1 million sites.

Brand Familiarity: Drupal is known to be the #3 most familiar content management system.

Major Features:

You can manage multiple sites with Drupal in multiple languages. You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine.
You can easily manage your site users, providing standard registration, including OpenID support. You can set various access control rules to limit the activity of your site users.
Provides multiple-level menu system, template customization, advanced search, RSS feed aggregator.
Officially Drupal supports several databases including MySQL, PostgreSQL, MariaDB, and SQLite.
To increase its performance, you can use caching. At the same time it provides high security with notifications about the new update releases.
Provides Search Engine Friendly descriptive URLs.
Powered by jQuery JavaScript framework.

Extensions: over 7,000 free community-contributed addons, known as contrib modules.

Distinguished Clients:

What users say:

More difficult to master especially for newbies. It's more for advanced users. Though the new Drupal 7 claims to provide significantly improved usability (maybe more toward WordPress style). To achieve this, they hired web designers to specifically address the UX problems it had in previous versions.
New major releases are not quite backward compatible. More focus on new features and functionality. External module developers should take care of compatibility themselves, except for data representation, which Drupal is intended to keep same.
Drupal is often compared with Joomla and is perceived to be a bit slower. Though this depends on the type of website you develop. However, leveraging its caching and gzipping technologies, it is possible to achieve quite impressive results.
jQuery, the default JavaScript framework in Drupal, allows to use alternative Mootools framework at the same time.

# 2: Joomla

Official site: http://www.joomla.org/

Joomla is the result of a fork of Mambo CMS on August 17, 2005. Within its first year of release, Joomla had been downloaded 2.5 million times.

Weekly downloads: 113,836 (ranked #2 after WordPress and before Drupal).

Installations: 1,297 according to the survey (#1 before WordPress), but 2.5% (#2 after WordPress and before Drupal) of Alexa Top 1 million sites.

Brand Familiarity: Joomla is known to be the #1 most familiar content management system.

Major Features:

You can manage multiple site with Joomla in multiple languages natively (since Joomla 1.6). You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine.
You can easily manage your site users, providing standard registration, including Google OpenID support. Full support for Access Control List.
Provides Multiple-level menu and content category system, template customization, advanced search, RSS feed aggregator.
Officially supports only MySQL.
Page cashing for increased performance.
Provides moderate descriptive URLs (still not fully customizable as you can do in WordPress).
Powered by MooTools JavaScript framework.

Extensions: There are over 6,000 free and commercial plugins available from the official site.

Distinguished Clients:

http://www.linux.com/
http://www.itwire.com/
http://www.quizilla.com
http://www.ihop.com
http://gsas.harvard.edu
Citibank (Financial institution intranet) - Not publicly accessible
http://www.greenmaven.com
http://www.outdoorphotographer.com
http://www.playshakespeare.com
http://www.sensointeriors.co.za

What users say:

More intuitive and easy to use than Drupal, though still not like WordPress.
Powerful. Fully-fledged content management system, so you can create whatever site you want.
Really strong security. If security problems found, immediately fixed.
The new Joomla 1.6 release is expected to be faster, more convenient, with more features.
Its strong dependency on Mootools JavaScript framework sometimes bothers users as Joomla does not give easy workaround to disable it and use jQuery instead.
Support of only one database, limits Joomla a lot in terms of the number of users. However, this is a compromise for high optimizations for MySQL, thus increased overall performance.
Does not allow to fully customize URLs - a must feature for CMS.

#3: modx

Official site: http://www.modxcms.com/

modx is not just an open source CMS but also a web application framework. Raymond Irving and Ryan Thrash began the MODx CMS project in 2004 as a fork of Etomite. In 2008 MODx users created a new logo and branding for the project. Now MODx allows for full segregation of content (plain HTML), appearance and behavior (standards compliant CSS and JavaScript) and logic (PHP, snippets).

Weekly downloads: 4,500 (ranked #11 after umbraco and before Tiki).

Installations: 58 according to the survey (#12 after eZ Publish and before umbraco), has less than 0.1% among the Alexa Top 1 million sites.

Brand Familiarity: #14 (before Liferay and after eZ Publish).

Major Features:

As with Joomla, modx officially supports only MySQL database.
Not just CMS but a PHP framework for Web.
Freedom to choose jQuery, Mootools, ExtJS, Prototype or any other JavaScript library.
Supports PHP 4.3.11 and above.
Complete control of all metadata and URL structure for SEO (Search Engine Optimization).
Unlimited hierarchical page depth.
Can create custom fields and widgets for templates.
Role-based permissions for the Manager.
Ability to customize the Manager on a per-deployment basis.
Ecommerce integration via Foxy Cart.

Extensions: 622, also known as add-ons.

Distinguished Clients:

What users say:

Good to have a choice for favorite JavaScript framework.
Light CMS solution (but not necessarily the fastest).
PHP 4 support for developers mean that a lot of compromises had to be made in terms of OOP (Object Oriented Programming) in order to offer PHP 4 support .
Nice to have freedom to set custom URL.

#4: WordPress

Official site: http://wordpress.org/

WordPress was first released on May 27, 2003, by Matt Mullenweg as a fork of b2/cafelog. As of August 2010, version 3.0 had been downloaded over 12.5 million times. Nowadays, known as the #1 CMS for blogging.

Weekly downloads: 983,625 (ranked #1 before Joomla).

Installations: 1,012 according to the survey (#2 after Joomla and before Drupal), but 12.9% of the Alexa Top 1 million sites (#1 before Joomla).

Brand Familiarity: #2 (after Joomla and before Drupal).

Major Features:

Highly optimized for blogging.
Custom and easy to switch themes.
Users can re-arrange widgets without editing PHP or HTML code.
Official support for only MySQL.
Custom URL, clean permalink structure, excellent for SEO.
Nested, multiple categories to articles.
Support for tagging. Advanced search by tags.
Highly intuitive UI (User Interface).
jQuery JavaScript framework.
Supports the Trackback and Pingback standards for displaying links to other sites that have themselves linked to a post or article.
Rich plugin architecture which allows users and developers to extend its functionality beyond the features that come as part of the base install.

Native applications exist for Android, iPhone/iPod Touch, and BlackBerry which provide access to some of the features in the WordPress Admin panel and work with WordPress.com and many WordPress.org blogs.

Extensions: 12,780 plugins and 1,315 themes.

Distinguished Clients:

What users say:

Perhaps, the most convenient, easy to use and intuitive CMS (or BMS) in the world. Perfect for blog sites. But if you need to develop a dynamic site with various components, perhaps, other CMS would fit better, though, WordPress provides enough plugins to accomplish almost all tasks.
I would avoid Wordpress as a CMS in a professional environment. As stated earlier, it's a great blogging platform, but doesn't generally offer the robustness that most professional environments require.
Availability of jQuery makes the plugin development a lot easier for external developers and site owners.
Endless themes - no need to worry about the new design for your site, unless you really need something special.

#5: DotNetNuke

Official site: http://www.dotnetnuke.com/

DotNetNuke is an open source platform for building web sites based on Microsoft .NET technology. It is written in VB.NET and distributed under both a Community Edition BSD-style license and a commercial proprietary license. The Community Edition is a popular web content management (WCM) system and application development framework for ASP.NET, with over 6 million downloads and 600,000 production web sites as of October 2010. More than 8,000 DotNetNuke apps are available for purchase on Snowcovered.com. DotNetNuke.com has over 800,000 registered members as of October 2010.

Weekly downloads: 13,000 (ranked #4 after Drupal and before CMS Made Simple).

Installations: 402 according to the survey (#4 after Drupal and before Liferay), but 0.2% of the Alexa Top 1 million sites (#4 after Typo3 and before MOVABLE TYPE).

Brand Familiarity: #4 (after Drupal and before Typo3).

Major Features:

Distinguishes between community (common features) and enterprise (full set of features) editions.
Various modules, and data providers.
Provides language packs for about 60 languages.
Customizable through skins and templates.

Distinguished clients:

What users say:

A little bit difficult to create modules.
It is perceived as a little bit bulky CMS.
Does not provide extensive documentation and user guides.
Similar to most CMS, does not provide full backward compatibility in its new major releases.
Unlike its enterprise edition, the community edition is not tested and certified by the DotNetNuke Corporation.
Even if it provides the language packs, the sites cannot be created to support multiple languages. Third party module should be used to enable this feature.
Auto-upgrade, Advanced Site Search, Page Cashing, and many must-have features are not included in the Community edition, only in Professional or Enterprise editions.

#6: Umbraco

Official site: http://umbraco.org/

Umbraco is also an open source content management system. It was developed by Niels Hartvig in 2000 and released as open source software in 2004. It is written in C# and can be deployed on Microsoft based infrastructure. In 2010, with about 1,000 downloads a day, Umbraco was in the Top 5 most popular downloads via the Microsoft Web Platform Installer, two places below its main rival DotNetNuke.

Weekly downloads: 5,420 (ranked #10 after Alfresco and before MODx).

Installations: 57 according to the survey (#13 after MODx and before e107), but less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #16 (after Liferay and before e107).

Major Features:

Can be deployed with several databases, including MySQL, SQL Server, and VistaDB.
SEO-friendly URLs.

Extensions: 310 add-on modules.

What users say:

Limited number of extensions.
Official support for Windows OS only.
Umbraco is oriented to small low-cost sites.
Video trainings have to be purchased.
Many must-have features have to purchased.

#7: Liferay

Official site: http://www.liferay.com/

Liferay Portal is a free and open source enterprise portal written in Java and distributed under the GNU Lesser General Public License. It allows users to set up features common to websites. It is fundamentally constructed of functional units called portlets. Liferay is sometimes described as a content management framework or a web application framework. It comes with certain portlets preinstalled. These comprise the core functionality of the portal system.

Weekly downloads: 9,435 (ranked #6 after CMS Made Simple and before TYPO3).

Installations: 154 according to the survey (#5 after DotNetNuke and before TYPO3), but less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #15 (after MODx and before Umbraco).

Major Features/ Portlets:

Can tag and categorize contents.
Document Library Manager, Recent Documents.
Alfresco, Documentum, and other document library integration.
User management based on various roles and groups (ACL).
WebDAV Integration (Web-based Distributed Authoring and Versioning which allows users to collaboratively edit and manage files on remote web servers).
Nested Portlets
User Directory
LDAP Integration
Microsoft Office Integration
Calendar/Chat/Mail/Message Boards/Polls
Wiki (supports Creole as well as MediaWiki syntax)
Alerts and Announcements
Knowledge Base
Social Equity
Can create multi-language sites.
Asset Publisher to publish many contents, tagged by a specific term, at once.

Extensions: 27 official plugins and 208 community developed plugins.

Distinguished Clients: It is primarily used to power corporate business sites.

What users say:

It is mostly used by enterprise companies rather than for powering personal or community sites, though it has social and collaboration features.
More professional developers driven project (backed by Liferay Inc.) rather than the community driven.
Provide paid Enterprise edition.
All features are available in Community edition (except for support and customization related).

#8: Typo3

Official site: http://typo3.org/

TYPO3 is a free and open source CMS released under the GNU General Public License oriented to small to mid size enterprise-class users. TemplaVoila is an alternative template engine extension for TYPO3. A graphical mapping tool for creating templates is included, an alternative page module, the ability to create flexible content elements and an API for developers. New content element types can be created without programming. TemplaVoila facilitates more flexibility for maintaining web pages than TYPO3's standard templating, while making it possible to enforce a strict corporate design and allowing editors to work with content more intuitively.

Weekly downloads: 7,461 (ranked #7 after Liferay and before eZ Publish).

Installations: 122 according to the survey (#6 after Liferay and before Tiki), 0.6% of the Alexa Top 1 million sites (#4 after Drupal and before DotNetNuke).

Brand Familiarity: #5 (after DotNetNuke and before OpenCMS).

Major Features:

Supports MySQL, Oracle, MS-SQL, PostgreSQL, ODBC, LDAP - virtually any external data source.
You can undo any change you make on the site.
Can create multiple sites with multiple domains for each.
Can have multiple template per site.
User management.
Able to switch from administrator user to general user to check permissions.
Sandbox: administrators can set up a section within the system to test new features without disturbing the main site.
Versioning of content pages.
Advanced caching: template, navigation or page level.
Link management.
Multi-language content.
Search Engine friendly URLs.

Extensions: more than 4,500 pluggable extensions are available for TYPO3.

Distinguished clients:

What users say:

The administrator's UI (User Interface) is not user-friendly, though intuitive and functional.
It is extremely powerful which provide more enterprise level features, but the learning curve is incredibly steep.
Great ease of multi-lingual site management.
Difficult to customize templates. Need to learn TypoScript and TemplaVoila, two TYPO3-specific systems.

#9: CMS Made Simple

Official site: http://www.cmsmadesimple.org/

CMS Made Simple is an open source cms built using PHP with support for MySQL and PostgreSQL. The template system is driven using the Smarty Template Engine.

Weekly downloads: 9,948 (ranked #5 after DotNetNuke and before Liferay).

Installations: 72 according to the survey (#7 after Tiki and before Alfresco), 0.1% of the Alexa Top 1 million sites (#8 after Xoops and before eZ Publish).

Brand Familiarity: #12 (after Xoop and before Ez Publish).

Major Features:

Officially supports MySQL and PostgreSQL.
Search Engine Friendly URLs.
Users management and group based permissions system.
Content tagging.
Site localization is available in 20 languages.

Extensions: unknown, but many.

What users say:

It is oriented more toward professional users.
It does not provide many templates, so coding knowledge is a must.

#10: Movable Type

Official site: http://movabletype.org/

Movable Type is a weblog publishing system, similar to WordPress, developed by the Six Apart company in Perl programming language. At various times, this company maintained three other publishing systems - TypePad, Vox, and LiveJournal. Movable Type was publicly announced on September 3, 2001. Version 1.0 was publicly released on October 8, 2001, thus it is a blogging system older than WordPress. On 12 December 2007, Movable Type was relicensed as free software under the GNU General Public License. Based on the list of its customers, Movable Type is quite credible CMS.

Weekly downloads: unavailable.

Installations: 30 according to the survey (#17 after Silverstripe and before OpenCMS), 0.1% of the Alexa Top 1 million sites (#6 after DotNetNuke and before Xoops).

Brand Familiarity: #8 (after Plone and before Alfresco).

Major Features:

Convenient blogging system with social community features.
Since version 5 officially supports only MySQL. PostgreSQL and SQLite can be used via plugins. Databases such as Oracle can be integrated with Movable Type Enterprise edition.
Manage user roles. OpenID support.
Multiple site hosting.
Easily customizable templates.
Revision history.
Can add custom fields.
Content tags and categories.
Feeds and trackback links.
Available localization and internationalization.
Can generate static pages (updated whenever the content of the site is changed).

Extensions: about 1,000 plugins.

Distinguished clients:

What users say:

Users often deploy Movable Type for their blog sites, newspaper or other type of publishing sites.
A powerful alternative for WordPress.

#11: Plone

Official site: http://plone.org/

Plone, a free and open source CMS, started in 1999 by Alexander Limi, Alan Runyan, and Vidar Andersen. It was made as a usability layer on top of the Zope content management framework, thus Plone is written in Python. The first version was released in 2001. In 2004, Plone 2.0 was released. This release brought more customizable features to Plone, and enhanced the add-on functions. In 2007, Plone 3 was released. This new release brought inline editing, an upgraded visual editor, and strengthened security, among many other enhancements. Recently in 2010, Plone 4 was released with major improvements in performance.

Weekly downloads: unavailabe.

Installations: 34 according to the survey (#15 after e107 and before Silverstripe), 0.1% of the Alexa Top 1 million sites (#10 after eZ Publish).

Brand Familiarity: #7 (after OpenCMS and before Movable Type).

Major Features:

Inline editing - no need to reload the page for editing.
Localized into 40 languages.
Plone can be integrated with Active Directory, Salesforce, LDAP, SQL, Web Services, and Oracle.
Working Copy support - keep the old version of a content published until you publish a new version.
Cut/copy/paste operations on content.
Link and reference integrity checking - no more broken links within your site.
Can create workflows - useful for organizations.
LiveSearch - instant site search powered by AJAX.
Full-text indexing of Word and PDF documents.
Wiki support
Collaboration and sharing
Automatic locking and unlocking
Versioning, history and reverting content
Authentication back-end
Collections
Multilingual content management
Automatic previous/next navigation
Human-readable URLs
Caching proxy integration
Drag and drop reordering of content
Adjustable templates on content
RSS feed support
Automatic image scaling and thumbnail generation
Comment capabilities on any content
WebDAV and FTP support
Hot -backup support

Extensions: 1490 add-ons.

Distinguished clients:

What users say:

It has a lot of features (several times more than listed above) built-in.
To become a Plone expert is long and expensive. It has a steep learning curve.
Plone is backed by Zope framework which is very powerful with support for caching, rollback, etc. - everything what your organization might need.
Plone is really complex to deeply tweak.
Plone is known for its high security.
Plone runs slow if you don't know how to optimize it.

#12: eZ Publish

Official site: http://ez.no/

eZ Publish is an open source enterprise CMS developed by the Norwegian company eZ Systems in 1999 using PHP programming language. eZ Publish is freely available under the GPL licence, as well as under proprietary licenses that include commercial support. eZ Publish supports the development of customized web applications. Typical applications range from a personal homepage to a multilingual corporate website, which include role-based multi-user access, e-commerce functions and online communities.

Weekly downloads: 7,031 (ranked #8 after TYPO3 and before Alfresco).

Installations: 60 according to the survey (#11 after Concrete5 and before MODx), 0.1% of the Alexa Top 1 million sites (#9 after CMS Made Simple and before Plone).

Brand Familiarity: #13 (after CMS Made Simple and before MODx).

Major Features:

Advanced Search feature is available in only Enterprise Edition.
Online image editor.
Can create building blocks for the site and later reuse them in several pages.

Distinguished clients:

What users say:

The documentation is a bit limited especially there is almost no documentation for module development.

#13: Concrete 5

Official site: http://www.concrete5.org/

Concrete5 is an open source CMS started in 2003 as a rapid-design approach to building the now-defunct LewisAndClark200.org, the official site for the Ad Council's National Council for the Lewis & Clark Bicentennial. Concrete5 is developed in PHP and is distributed under MIT software license.

Weekly downloads: unavailable.

Installations: 62 according to the survey (#10 after Alfresco and before eZ Publish), has less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #20 (after TextPattern).

Major Features:

Integrated server side caching.
Support for only MySQL.
Inline content editing.
Image editing tool.
Editable areas are defined in concrete5 templates which allow editors to insert 'blocks' of content. Additional blocks are available as add-ons.
Automatic upgrade is available.
Advanced Permissions to track content versions.

Distinguished clients:

What users say:

It's PHP based and quite new but it's quite a nice layout and it's really natural for new CMS users. You can go from a paper-based sitemap and PSD to a full site structure, ready for data entry, within a day, two at a push.
Concrete5 is simple, suitable for creating sites quickly.
Creating template is very easy with Concrete5.

#14: Alfresco

Official site: http://www.alfresco.com/

Alfresco is an open source enterprise content management system for Microsoft Windows and Unix-like operating systems. Alfresco includes a content repository, an out-of-the-box web portal framework for managing and using standard portal content, a CIFS interface that provides file system compatibility on Microsoft Windows and Unix-like operating systems, a web content management system capable of virtualizing web apps and static sites via Apache Tomcat, Lucene indexing, and jBPM workflow. The Alfresco system is developed using Java technology. John Newton (co-founder of Documentum) and John Powell (a former COO of Business Objects) founded Alfresco Software, Inc. in 2005.

Weekly downloads: 7,000 (ranked #9 after Ez Publish and before Umbraco).

Installations: 70 according to the survey (#9 after CMS Made Simple and before Concrete5) Alexa ranking is not available.

Brand Familiarity: #9 (after Movable Type and before Tiki).

Major Features:

Document Management.
Web Content Management (including full webapp & session virtualization).
Repository-level versioning (similar to Subversion).
Records Management, including 5015.2 certification.
Repository access via CIFS /SMB, FTP, Web DAV, NFS and CMIS.
j BPM workflow.
Advanced search with Lucene.
Multi-language support.
Officially runs on Windows, Linux and Solaris.
User Interface official supports Internet Explorer and Firefox.
Desktop integration with Microsoft Office and OpenOffice.org.
Clustering support.
Pluggable authentication: NTLM, LDAP, Kerberos, CAS.

Distinguished clients: no links are available but numerous case studies can be found on Alfresco home page.

France Air Force
Harvard Business School Publishing
Toyota
Sony Pictures
Fox
National Academy of Sciences
Cisco

What users say:

Alfresco is mostly for enterprises rather than for personal sites.
Simple to install and use, flexible and open-ended.
Alfresco is a solution with the broadest range of technical capabilities and the best feedback from users. In addition to demonstrating a promising roadmap for collaboration tools, Alfresco was highly attractive from a cost perspective, compared to the proprietary products offered by other ECM vendors.
All in one solution for enterprises.

Our Experience of Creating Large Scale Log Search System Using ElasticSearch

2013-05-06T02:06:36-07:00

At NHN we have a service called NELO (NHN Error Log System) to manage and search logs pushed to the system by various applications and other Web services. The search performance and functionality of NELO2, the second generation of the system, have significantly been improved through ElasticSearch. Today I would like to share our experience at NHN in deploying ElasticSearch in Log Search Systems.

ElasticSearch is a distributed search engine based on Lucene developed by Shay Banon. Shay and his team have recently released the long awaited version 0.90. Here is a link to a one-hour recorded webinar where Clinton Gormley, one of the core ElasticSearch developers, explains what's new in ElasticSearch 0.90.

If you are developing a system which requires a search functionality, I would recommend ElasticSearch as its installation and server expansion are very easy. Since it is a distributed system, ElasticSearch can easily cope with an increase in the volume of search targets. At NHN all logs coming into NELO2 are stored and indexed by ElasticSearch for faster near real-time search results.

Features of ElasticSearch

Let's get started with familiarizing ourselves with the terms widely used in ElasticSearch. For those who are familiar with relational database systems, the following table compares the terms used in relational databases with the terms used in ElasticSearch.

Table 1: Comparison of the terms of RDBMS and ElasticSearch.
Relational DB	ElasticSearch
Database	Index
Table	Type
Row	Document
Column	Field
Schema	Mapping
Index	Everything is indexed
SQL	Query DSL

JSON-based Schemaless Storage

ElasticSearch is a search engine but can be used like NoSQL. Since a data model is represented in JSON, both requests and responses are exchanged as JSON documents. Moreover, sources are also stored in JSON. Although schema is not defined in advance, JSON documents are automatically indexed when they are transferred. Number and date types are automatically mapped.

Multitenancy

ElasticSearch supports multitenancy. Multiple indexes can be stored in a single ElasticSearch server, and data of multiple indexes can be searched with a single query. NELO2 separates indexes by date and stores logs. When executing a search, NELO requests indexes of dates within the scope of search with a single query.

Code 1: Multitenancy Example Query.

# Store logs in the log-2012-12-26 index
curl -XPUT http://localhost:9200/log-2012-12-26/hadoop/1 -d '{
    "projectName": "hadoop",
    "logType": "hadoop-log",
    "logSource": "namenode",
    "logTime":"2012-12-26T14:12:12",
    "host": "host1.nelo2",
    "body": "org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile"
}'

# Store logs in the log-2012-12-27 index
curl -XPUT http://localhost:9200/log-2012-12-27/hadoop/1 -d '{
    "projectName": "hadoop",
    "logType": "hadoop-log",
    "logSource": "namenode",
    "logTime":"2012-12-27T02:02:02",
    "host": "host2.nelo2",
    "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
}'

# Request search to the nelo2-log-2012-12-26 and nelo2-log-2012-12-27 indexes at once
curl -XGET http://localhost:9200/nelo2-log-2012-12-26,nelo2-log-2012-12-27/_search

Scalability and Flexibility

ElasticSearch provides excellent scalability and flexibility. It enables the expansion of functionality through plug-ins, which was further improved in recent 0.90 release. For example, by using Thrift or Jetty plugin, you can change transfer protocol. If you install BigDesk or Head, which is a required plugin, you can use the functionality of ElasticSearch monitoring. As shown in the following Code 2, you can also adjust the number of replicas dynamically. The number of shards is not changeable as it is fixed for each index, so an appropriate number of shards should be allocated in the first time by taking the number of nodes and future server expansion into account.

Code 2: Dynamic Configuration Change Query.

$ curl -XPUT http://localhost:9200/log-2012-12-27/ -d '{
    "settings": {
        "number_of_shards": 10,
        "number_of_replicas": 1
    }
}'

Distributed Storage

ElasticSearch is a distributed search engine. It distributes data by configuring multiple shards according to keys. An index is configured for each shard. Each shard has 0 or more replicas. Moreover, ElasticSearch supports clustering, and when a cluster runs, one of many nodes is selected as the master node to manage metadata. If the master node fails, another node in the cluster automatically becomes the master. It is also very easy to add nodes. When a node is added to the same network, the added node will automatically find the cluster through multicast and add itself to the cluster. If the same network is not used, the master node address should be specified through unicast (see a related video: http://youtu.be/l4ReamjCxHo).

Installing

Quick Start

ElasticSearch supports zero configuration installation. As shown in the following code snippets, all you have to do for execution is download a file from the official homepage and unzip it.

Download

~$ wget http://download.ElasticSearch.org/ElasticSearch/ElasticSearch/ElasticSearch-0.20.1.tar.gz   
~$ tar xvzf ElasticSearch-0.20.1.tar.gz

Executing Server
```
~$ bin/ElasticSearch -f
```

Installing Plugins

You can easily expand the functionality of ElasticSearch through plugins. You can add management functionalities, change the analyzer of Lucene, and change the basic transfer module from Netty to Jetty. The following is a command we use to install plugins for NELO2. Head and bigdesk, which are found in the first and second lines, are the plugins required for ElasticSearch monitoring. It is strongly recommended to install them and check their functionalities. After installing them, visit http://localhost:9200/plugin/head/ and http://localhost:9200/plugin/bigdesk/, and you can see the status of ElasticSearch in your Web browser.

bin/plugin -install Aconex/ElasticSearch-head
bin/plugin -install lukas-vlcek/bigdesk
bin/plugin -install ElasticSearch/ElasticSearch-transport-thrift/1.4.0
bin/plugin -install sonian/ElasticSearch-jetty/0.19.9

Main Configurations

You don't need to change configurations when conducting a simple functionality test. When you carry out a performance test or apply it to production services, then you should change some default configurations. See the following snippet and try to find for yourself the configurations which should be changed from the initial configuration file.

Code 5: Main Configurations (config/ElasticSearch.yml).

# As it is a name used to identify clusters, use a name with uniqueness and a meaning.
cluster.name: ElasticSearch-nelo2

# A node name is automatically created but it is recommended to use a name that is discernible in a cluster like a host name.
node.name: "xElasticSearch01.nelo2"

# The default value of the following two is all true. node.master sets whether the node can be the master, while node.data is a configuration for whether it is a node to store data. Usually you need to set the two values as true, and if the size of a cluster is big, you should adjust this value by node to configure three types of node. More details will be explained in the account of topologies configuration later.
node.master: true
node.data: true

# You can change the number of shards and replicas. The following value is a default value:
index.number_of_shards: 5
index.number_of_replicas: 1

#To prevent jvm swapping, you should set the following value as true:
bootstrap.mlockall: true

# It is a timeout value for checking the status of each node in a cluster. You should set an appropriate value; if the value is too small, nodes may frequently get out of a cluster. The default value is 3 seconds.
discovery.zen.ping.timeout: 10s

# The default value is multicast, but in an actual environment, unicast should be employed due to the possibility of overlapping with other clusters. It is recommended to list servers that can be a master in the second setting.
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1", "host2:port", "host3[portX-portY]"]

Using REST API

ElasticSearch provides a REST API as shown below. It provides most of its functionalities through REST API, including the creation and deletion of indexes, mappings, as well as search and change of settings. In addition to REST API, it also provides various client APIs for Java, Python and Ruby.

Code 6: REST API Format in ES.

http://host:port/(index)/(type)/(action|id)

As mentioned earlier, NELO2 classifies indexes (databases in RDBMS terms) by date, and type (table) is separated by project. Code 7 below shows the process of creating logs that came into the hadoop project on December 27, 2012, in the unit of document by using a REST API.

Code 7: An Example of Using ElasticSearch REST API.

#Creating documents
curl -XPUT http://localhost:9200/log-2012-12-27/hadoop/1
curl -XGET http://localhost:9200/log-2012-12-27/hadoop/1
curl -XDELETE http://localhost:9200/log-2012-12-27/hadoop/1

#Search
curl -XGET http://localhost:9200/log-2012-12-27/hadoop/_search
curl -XGET http://localhost:9200/log-2012-12-27/_search
curl -XGET http://localhost:9200/_search

#Seeing the status of indexes
curl -XGET http://localhost:9200/log-2012-12-27/_status

Creating Documents and Indexes

As shown in the following Code 8, when the request is sent, ElasticSearch creates the log-2012-12-27 index and hadoop type automatically without any pre-defined index or type. If you want to create them specifically instead of using auto creation, you should specify the setting of action.auto_create_index and index.mapper.dynamic as false in the configuration file.

Code 8: Creating Documents.

# Request
curl -XPUT http://localhost:9200/log-2012-12-27/hadoop/1 -d '{
    "projectName": "hadoop",
    "logType": "hadoop-log",
    "logSource": "namenode",
    "logTime":"2012-12-27T02:02:02",
    "host": "host2.nelo2",
    "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
}'

# Result
{
    "ok": true,
    "_index": "log-2012-12-27",
    "_type": "hadoop",
    "_id": "1",
    "_version": 1
}

As shown in Code 9 below, you can make a request after including type in a document.

Code 9: A Query Including Type.

curl -XPUT http://localhost:9200/log-2012-12-27/hadoop/1 -d '{
    "hadoop": {
        "projectName": "hadoop",
        "logType": "hadoop-log",
        "logSource": "namenode",
        "logTime":"2012-12-27T02:02:02",
        "host": "host2.nelo2",
        "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
    }
}'

If an id value is omitted as in Code 10, an id will be created automatically when a document is created. Note that thePOST method was used instead of PUT when a request was made.

Code 10: A Query Creating a Document without an ID.

# Request
curl -XPOST http://localhost:9200/log-2012-12-27/hadoop/ -d '{
    "projectName": "hadoop",
    "logType": "hadoop-log",
    "logSource": "namenode",
    "logTime":"2012-12-27T02:02:02",
    "host": "host2.nelo2",
    "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
}'

# Result
{
    "ok": true,
    "_index": "log-2012-12-27",
    "_type": "hadoop",
    "_id": "kgfrarduRk2bKhzrtR-zhQ",
    "_version": 1
}

Deleting a Document

Code 11 below shows how to delete a document (a record in RDBMS terms) in type (a table). You can delete a hadoop type document with id=1 of the log-2012-12-27 index by using the DELETE method.

Code 11: A Query to Delete a Document.

# Request
$ curl -XDELETE 'http://localhost:9200/log-2012-12-27/hadoop/1'

# Result
{
    "ok": true,
    "_index": "log-2012-12-27",
    "_type": "hadoop",
    "_id": "1",
    "found": true
}

Getting a Document

You can get a hadoop type document with id=1 of the log-2012-12-27 index by using the GET method as shown in Code 12.

Code 12: A Query to Get a Document.

#Request
curl -XGET 'http://localhost:9200/log-2012-12-27/hadoop/1'

# Result
{
    "_index": "log-2012-12-27",
    "_type": "hadoop",
    "_id": "1",
    "_source": {
        "projectName": "hadoop",
        "logType": "hadoop-log",
        "logSource": "namenode",
        "logTime":"2012-12-27T02:02:02",
        "host": "host2.nelo2",
        "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
    }
}

Search

When the Search API is called, ElasticSearch executes the Search API and returns the search results that match the content of the query. Code 13 shows an example of using Search API.

Code 13: An Example Query of Using Search API.

# All types of a specific index
$ curl -XGET 'http://localhost:9200/log-2012-12-27/_search?q=host:host2.nelo2'

# A specific type of a specific index
$ curl -XGET 'http://localhost:9200/log-2012-12-27/hadoop,apache/_search?q=host:host2.nelo2'

# A specific type of all indexes
$ $ curl - XGET 'http://localhost:9200/_all/hadoop/_search?q=host:host2.nelo2'

# All indexes and types
$ curl -XGET 'http://localhost:9200/_search?q=host:host2.nelo2'

Search API by Using URI Request

Table 2: Main Parameters.
Name	Description
q	Query string.
default_operator	The operator used as a default (`AND` or `OR`). The default is `OR`.
fields	The field to get as a result. The default is the "_source" field.
sort	Sort method. Ex) fieldName:asc/fieldName:desc.
timeout	Search timeout value. The default is "unlimited".
size	The number of result values. The default is 10.

If you use URI, you can search easily by using parameters in Table 2 and a query string. As it does not provide all search options, it is useful when used for tests.

Code 14: Search Query by Using URI Request.

# Request
$ curl -XGET 'http://localhost:9200/log-2012-12-27/hadoop/_search?q=host:host2.nelo2'

# Result
{
    "_shards":{
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits":{
        "total": 1,
        "hits": [
            {
                "_index": "log-2012-12-27",
                "_type": "hadoop",
                "_id": "1", 
                "_source": {
                    "projectName": "hadoop",
                    "logType": "hadoop-log",
                    "logSource": "namenode",
                    "logTime":"2012-12-27T02:02:02",
                    "host": "host2.nelo2",
                    "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
                }
            }
        ]
    }
}

Search API by Using Request Body

When HTTP body is used, perform a search by using query DSL. As query DSL has a large amount of contents, you are advised to refer to a guide from the official website.

Code 15: Search by Using Query DSL.

# Request
$ curl -XPOST 'http://localhost:9200/log-2012-12-27/hadoop/_search' -d '{
    "query": {
        "term": { "host": "host2.nelo2" }
    }
}'

# Result
{
    "_shards":{
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits":{
        "total": 1,
        "hits": [
            {
                "_index": "log-2012-12-27",
                "_type": "hadoop",
                "_id": "1",
                "_source": {
                    "projectName": "hadoop",
                    "logType": "hadoop-log",
                    "logSource": "namenode",
                    "logTime":"2012-12-27T02:02:02",
                    "host": "host2.nelo2",
                    "body": "org.apache.hadoop.hdfs.server.namenode.FSNamesystem"
                }
            }
        ]
    }
}

Mapping

Put Mapping API

To add a mapping to a specific type, you can define a mapping in the form shown in Code 16.

Code 16: Query to Register a Mapping.

$ curl -XPUT 'http://localhost:9200/log-2012-12-27/hadoop/_mapping' -d '
{
    "hadoop": {
        "properties": {
            "projectName": {"type": "string", "index": "not_analyzed"},
            "logType": {"type": "string", "index": "not_analyzed"},
            "logSource": {"type": "string", "index": "not_analyzed"},
            "logTime": {"type": "date"},
            "host": {"type": "string", "index": "not_analyzed"},
            "body": {"type": "string"},
        }
    }
}'

Get Mapping API

To get defined mapping information, you can use a query in the form shown in Code 17.

Code 17: Query to Get a Mapping.

$ curl -XGET 'http://localhost:9200/log-2012-12-27/hadoop/_mapping'

Delete Mapping API

Code 18 shows an example of deleting a defined mapping.

Code 18: Query to Delete a Mapping.

$ curl -XDELETE 'http://localhost:9200/log-2012-12-27/hadoop'

How to Optimize Performance

Memory and the Number of Open Files

If the amount of data to search increases, you will need more memory. When you run ElasticSearch, you will encounter many problems due to the use of memory. In an operating method recommended by an ElasticSearch community, when you run a server exclusively for ElasticSearch, you are advised to allocate only half of the memory capacity to ElasticSearch, and to allow the OS to use the other half for system cache. You can set the memory size by setting the ES_HEAP_SIZE environmental variable or by using -Xms and -Xmx of JVM.

Code 19: Execution by Specifying Heap Size.

bin/ElasticSearch -Xmx=2G -Xms=2G

When using ElasticSearch, you will see OutOfMemory errors frequently. This error occurs when the field cache exceeds the maximum heap size. If you change the setting for index.cache.field.type from resident (default) to soft, soft reference will be used and the cache area will be preferentially GC, and this problem can be resolved.

Code 20: Configuring Field Cache Type.

index.cache.field.type: soft

If the amount of data increases, the number of index files also increases. This is because Lucene, which is used by ElasticSearch, manages indexes in the unit of segments. Sometimes the number will even exceed the number of MAX_OPEN files. For this reason, you need to change the maximum open file limit by using the ulimit command. The recommended value is 32000-64000, but sometimes you may need to set a larger value depending on the size of the system or data.

Index Optimization

NELO2 manages indexes by date. If indexes are managed by date, you can delete old logs that don't need to be managed easily and quickly, as shown in Code 21. In this case, the overhead imposed on the system is smaller than when deleting logs by specifying the TTL value for each document.

Code 21: Deleting an Index.

$ curl -XDELETE 'http://localhost:9200/log-2012-10-01/'

If index optimization is performed, segments are incorporated. Using this method, you can enhance search performance. As index optimization can impose a burden on the system, it is better to perform it when the system is being used less.

Code 22: Index Optimization.

$ curl -XPOST 'http://localhost:9200/log-2012-10-01/_optimize'

Shards and Replicas

You can't change the number of shards after setting it. For this reason, you need to decide this value carefully by taking the current number of nodes in the system and the number of nodes expected to be added in the future into account. For example, if there are 5 nodes and the number is expected to reach 10 in the future, it is recommended to set the number of shards as 10 from the beginning. If you set it as 5 in the beginning, you can add 5 more nodes later, but you won't be able to use the added 5 nodes. If you set the number of replicas to 1, of course, you can utilize the added 5 nodes as nodes exclusively for replication.

If the number of shards increases, it is more advantageous to process a large amount of data because queries are distributed as much as the number of shards. But you need to set this value appropriately, because the performance could be deteriorated due to increasing traffic if the value is too high.

Configuring Cluster Topologies

The content of the configuration file of ElasticSearch is shown in Code 23 below. There are three types of nodes:

data node
This does not act as the master, and only stores data. When it receives a request from a client, it searches data from shards or creates an index.
master node
It functions to maintain a cluster, and requests indexing or search to data nodes.
search balancer node
If it receives a search request, it requests data, gathers data and delivers the result.

You can have one node which will function both like a master and a data node. But if you use the three types of node separately, you can reduce the burden of the data node. In addition, if you configure the master node separately, you can improve the stability of a cluster. Also, you can reduce operation costs by using low-spec. server equipment for the master and search node.

Code 23: Settings Related to Topology.

# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
# node.master: false
# node.data: true
#
# 2. You want this node to only serve as a master, to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
# node.master: true
# node.data: false
#
# 3. You want this node to be neither a master nor a data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
# node.master: false
# node.data: false

Figure 1 below shows the configuration of NELO2 topologies that use ElasticSearch. The efficiency of equipment use and the stability of the entire cluster has been improved as follows: only ElasticSearch runs on the 20 data nodes (server) so that they can achieve sufficient performance, while other daemon server processes in addition to ElasticSearch run on the 4 master nodes and 3 search nodes.

Figure 1: NELO2 ElasticSearch Topologies.

Configuring Routing

When a large amount of data needs to be indexed, increasing the number of shards will improve the overall performance. On the other hand, if the number of shards increases, the traffic among nodes will also go up. For example, when there are 100 shards, if it receives a single search request, it sends the request to all the 100 shards and aggregates data, and this imposes a burden on the entire cluster.

If you use routing, data will be stored only in a specific shard. Even if the number of shards increases, the application will still send a request only to a single shard, and consequently the traffic can be reduced dramatically. Figure 2, 3, and 4 are excerpted from the slides Rafal Kuc presented at Berlin Buzzwords 2012. If you don't use routing, as shown in Figure 2, the application will send a request to all the shards. But if you use routing, it will send a request only to a specific shard, as shown in Figure 3.

According to the material cited, in Figure 4 when there are 200 shards, the response time is over 10 times faster with routing than without routing. If routing is applied, the number of threads will increase by 10 to 20 times compared to when it is not applied, but the CPU usage is much smaller. In some cases, however, the performance will be better when routing is not applied. For a search query whose result should be collected from multiple shards, it could be more advantageous in terms of performance to send the request to multiple shards. To complement this, NELO2 determines the use of routing depending on the log usage of the project.

Figure 2: Before Using Routing.

Figure 3: After Using Routing.

Figure 4: Performance Comparison before and after Using Routing.

Conclusion

The number of users of ElasticSearch is increasing rapidly, thanks to its easy installation and high scalability. It was several days only since the release of the latest ElasticSearch version 0.90. Its functionality is improving very quickly thanks to its active community. In addition, more and more companies are beginning to use ElasticSearch for their services. Recently, some committers, including the developer Shay Banon, gathered together and established ElasticSearch.com, which provides consulting and training services.

In this article I have explained the basic information on the installation of ElasticSearch, how to use it, and do performance tuning. We have started testing the latest 0.90 release and soon will migrate the current 0.20.1 ES deployment. In the next post I will continue this topic and tell you about our experience with 0.90 as well as the critical split-brain problem we have previously experienced. Due to the scarcity of solutions for this problem, I believe it will be very useful for our readers.

By Lee Jae Ik, Senior Software Engineer at Global Platform Development Lab, NHN Corporation.

References

Official guide: http://www.ElasticSearch.org/guide/
Introduction to ElasticSearch and comparison of the terms of ElasticSearch and RDB: http://www.slideshare.net/clintongormley/cool-bonsai-cool-an-introduction-to-ElasticSearch
About ElasticSearch: http://www.slideshare.net/dadoonet/ElasticSearch-devoxx-france-2012-english-version
Shay Banon's articles: http://2011.berlinbuzzwords.de/sites/2011.berlinbuzzwords.de/files/ElasticSearch-bbuzz2011.pdf
Using ElasticSearch for logs: http://www.ElasticSearch.org/tutorials/2012/05/19/ElasticSearch-for-logging.html
Concept of multitenancy: http://en.wikipedia.org/wiki/Multitenancy
Shay Banon's ElasticSearch optimization: https://github.com/logstash/logstash/wiki/ElasticSearch-Storage-Optimization
Rafal Kuc's article on performance tuning presented at Berlin Buzzwords 2012: http://www.slideshare.net/kucrafal/scaling-massive-elastic-search-clusters-rafa-ku-sematext

Announcing CUBRID ALL-IN-ONE Windows Downloader

2013-04-29T00:05:10-07:00

We are very glad to announce the immediate availability of CUBRID ALL-IN-ONE Windows Downloader version 1.0 beta. You can download CUBRID ALL-IN-ONE Windows Downloader from http://www.cubrid.org/wiki_tools/entry/cubrid-all-in-one-windows-downloader. The source code is available at http://svn.cubrid.org/cubridtools/cubrid-downloader/ which is open sourced under BSD license just like all other CUBRID Tools.

CUBRID ALL-IN-ONE Windows Downloader is an application that allows our users to easily download CUBRID components including the server engine, drivers and GUI tools. All you have to do is to select the components you want to download on your local Windows machine and the Downloader will download them for you, one by one, without any other actions required.

Application key features:

The application can auto-update itself, anytime a new version is available (it uses the ClickOnce technology).
Retrieves all the components information from a remote CUBRID online location, so it is always up-to- date with the latest application releases.
Detects local machine specifics - CUBRID version, OS architecture – and automatically selects the appropriate list of components.
Can handle software pre-prerequisites dependencies and download them as well.
Supports both HTTP and FTP protocols for downloads.
Provides additional information to users like links to online resources.
Handles download errors and auto-retries in case of failures.
Supports alternate download locations to try in case of failures.
Saves the user preferences and re-uses them next time.
Provides a comprehensive operations log information.
Supports UI localization.

Here is a mute video which shows how to use CUBRID ALL-IN-ONE Downloader.

If you have questions or suggestions, leave your comments below.

CUBRID Node.js Driver 1.1 is now available at NPM

2012-12-23T16:45:24-08:00

I am very glad to announce the immediate availability of CUBRID Node.js driver version 1.1. You can download node-cubrid from NPM. For more details, see check out the official repository at https://github.com/CUBRID/node-cubrid.

What's New

In this new release we have improved many aspects of the driver.

We now follow "One driver to rule them all" concept. In other words, you can use node-cubrid with any version of CUBRID Database including 8.4.1, 8.4.3, and 9.0.0 (beta).
The new driver comes with many code fixes, bugs fixes, and design changes.
Rich database support: Connect, Query, Fetch, Execute, Commit, Rollback, DB Schema etc.
Out of the box driver events model.
10.000+ LOC, including the driver test code and demos
New test cases added. Now 50+.
We have significantly refactored the code using JSHint/JSLint code analysis.
New documentation.
Created new tutorials that we will publish later.

Testing

The 1.1 release was successfully tested, using the driver test suite, on 2 OS x 3 CUBRID engine = 6 different test environments.

What's next

The upcoming new version, 2.0, will feature many new features and improvements. For example:

CUBRID 9.1 stable release compatibility.
Improved queries support. In particular, implementing queries queuing support similar to MySQL.
Extended database schema support
... and many more.

If you have questions, ask at our CUBRID Q&A site, or at #cubrid IRC. We will be glad to answer you!

Inside Vert.x. Comparison with Node.js.

2013-04-08T03:30:00-07:00

Vert.x is a server framework which is rapidly arising. Each server framework claims its strong points are high performance with a variety of protocols supported. Vert.x takes a step forward from that. Vert.x considers the environment of establishing and operating the server network environment. In other words, Vert.x includes careful consideration in producing several 'server process DAEMONs' that run on the clustering environment, as well as producing one server process DAEMON.

Therefore, it is important to review into Vert.x: which network environment it considers as well as how it delivers high performance. So, I think it will be valuable to pay sufficient time examining Vert.x structure.

Philosophy of Vert.x

Vert.x is a project affected by Node.js. Like Node.js, Vert.x is a server framework providing an event-based programming model. Therefore, Vert.x API is very similar to that of Node.js. That's because both of these models provide asynchronous API.

Node.js is created by using JavaScript, but Vert.x is created by using Java. However, it is too much to understand Vert.x as a Java version of Node.js. Vert.x has been affected by Node.js; but Vert.x has its own unique philosophy, different from Node.js.

The most typical design philosophy of Vert.x is summarized as follows:

Polyglot - supports several languages
Vert.x itself is built in Java. However, Java is not required to use Vert.x.
As well as languages based on JVM operation, such as Java or Groovy, Vert.x can be used with Ruby, Python, and even JavaScript. If you need to build a server application by using JavaScript, there is an alternative to Node.js. In addition, Scala and Closure are planned to be supported.
Super Simple Concurrency model
When building a server application by using Vert.x, users can write code as a single thread application. That means that the multi-thread programming effect can be achieved without synchronization, lock, or volatility.
In Node.js, the JavaScript execution engine does not support multi-thread. However, to utilize all CPU cores, several same JavaScript programs have to be executed.
However, Vert.x allows to create multiple threads based on the number of CPU cores whlie only one process is executed. It handle the multi-threading so users can focus on implementing business logic.
Provides Event Bus
As described in the introduction, the goal of Vert.x is not only to produce a ‘one server process DAEMON'. Vert.x aims to make a variety of Vert.x-built server programs communicate well with each other. For this, Vert.x provides Event Bus. Therefore, MQ functions such as Point to Point or Pub/Sub can be used (to provide Event Bus function, Vert.x uses Hazelcastm, an In-Memory Data Grid).
With this Event Bus, a server application built with different languages can easily communicate with each other.
Module System & Public Module Repository
Vert.x has a module system. This module system can be understood as a type of component. That means the Vert.x-built server application project itself is modularized. It aims at reusability. This module can be registered to Public Module Repository. Through the Public Module Repository, the module can be shared.

What is the relationship and defference between Netty and Vert.x?

Before discussing the Vert.x performance, we should summarize the relationship between Netty and Vert.x. Vert.x uses Netty. In other words, it processes all IOs by using Netty. Therefore, it is meaningless to verify differences of the performance between Vert.x and Netty.

Vert.x is a server framework that provides API and functions different and independent from Netty, designed with different purpose from Netty.

Netty is a framework that can process the low-level IO and Vert.x can process the higher-level IO than Netty.

Comparison of Performance with Node.js

Even if functions provided by Vert.x are different from those of Node.js, comparing the performance between them is a significant matter. Figure 1 and Figure 2 below show the performance of Vert.x (Java, Ruby, Groovy) and Node.js. (Source: http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/).

Figure 1 shows a comparison of performance when an HTTP server is built and only a 200/OK response has been returned. Figure 2 shows the comparison of performance when a 72-byte static HTML file is returned as a response.

Figure 1: Comparison of Performance When Only 200/OK Response Has Been Returned.

Figure 2: Comparison of Performance When a 72-byte Static HTML File is Returned.

This performance is proclaimed by Vert.x developers but the test has not been made under a strict environment. Just look the relative differences in performance.

Also, a notable point is that the performance of Vert.x-JavaScript is better than Node.js. However, even if the performance result is very reliable, it may be difficult to say that Vert.x is better than Node.js. That's because Node.js provides great models such as Socket.io and has lots of references.

Vert.x Terminology

Vert.x defines its unique terms and redefines general terms for Vert.x itself. Therefore, to understand Vert.x, it is necessary to understand the Vert.x-defined terms. The followings are popular terms used in Vert.x:

Verticle

For Java, it is the class with a main method. Verticle can also include other scripts referred to by the main method. It can also include the jars files or resources. An application may consist of one Verticle or several Verticles, which communicate through Event Bus. Alongside Java, it can be understood as an independently executable class or a jar file.

Vert.x Instance

A Verticle is executed within a Vert.x instance and the Vert.x instance is executed in its JVM instance. So there will be a lot of Verticles which are simultaneously executed in a single Vert.x instance. Each Verticle can have its own unique class loader. In this manner, direct interactions between Verticles, made through static members and global variables, can be prevented. A lot of Verticles can be simultaneously executed in several hosts on the network and the Vert.x instances can be clustered through Event Bus.

Concurrency

The Verticle instance guarantees it is always executed on an identical thread. As all codes can be developed as a single thread operation type, developers who use environment where Vert.x can be easily developed. In addition, race condition or deadlock can be prevented.

Event-based Programming Model

Like the Node.js framework, Vert.x provides an event-based programming model. When programming a server by using Vert.x, most codes for development are related to event handlers. For example, an event handler should be set to receive data from a TCP socket or an event handler, which will be called when data is received, should be created. In addition, event handlers should be created to receive alarms 'when Event Bus receives a message,' 'when HTTP messages are received,' 'when a connection has been disconnected,' and 'when a timer is timeout.'

Event Loops

Vert.x instance internally manages the thread pool. Vert.x matches the number of thread pools to the number of CPU cores as closely as possible.

Each thread executes Event Loop. Event Loop verifies the events as rounding the loop. For example, verifying whether there is data to read in the socket or on which timer an event has occurred. If there is an event to process on the loop, Vert.x calls the corresponding handler (of course, additional work is necessary if the handler-processing period is too long or there is a blocking I/O).

Message Passing

Verticles use Event Bus for communication. If a Verticle is assumed as an actor, Message Passing is similar to an actor model, which was famous in Erlang programming languages. The Vert.x server instance has a lot of Verticle instances and allows message passing among the instances. Therefore, the system can be extended according to the usable cores without executing the Verticle code through multi-thread.

Shared data

Message passing is very useful. However, it is not always the best approach in all types of application concurrency situations. Cache is one of the most popular examples. If only one vertical has a certain cache, it is very inefficient. If other Verticles need the cache, each Verticle should manage the same cache data.

Therefore, Vert.x provides a method for global access. It is the Shared Map. Verticles share immutable data only.

Vert.x Core

As named, this is the core function of Vert.x. Functions that the Verticle can directly call are included in the core. Therefore, the core can be accessed by each programming language API supported by Vert.x.

Vert.x Architecture

The simple architecture of Vert.x is shown in the following Figure 3.

Figure 3: Vert.x Architecture (source: http://www.javacodegeeks.com/2012/07/osgi-case-study-modular-vertx.html)

The default execution unit of Vert.x is Verticle and several Verticles can be simultaneously executed on one Vert.x instance. The Verticles are executed on the event-loop thread. Several Vert.x instances can be executed on several hosts, as well as on one host on the network. At this time, the Verticles or modules communicate by using the event bus.

To sum up, the vert.x application consists of combinations of Verticles or modules and communication among those is made by using Event Bus.

vert.x Project Structure

The following is the Vert.x project structure viewed from eclipse when clonning the source code from the Vert.x Github page.

Figure 4: Vert.x source tree.

The overall configuration is as follows:

vertx-core is the core library.
vertx-platform manages distribution and lifecycle.
vert-lang used to expose the Core Java API to another language.

Gradle is used as its project build system. It has the advantages of ant and maven.

Installing Vert.x and Executing Simple Examples

To use Vert.x, JDK7 is required because Vert.x uses the invokeDynamic in JDK7.

Vert.x can easily be installed.

Download the compressed installation file from http://vertx.io/downloads.html to a desired location.
Decompress the file and add the bin directory to the PATH environment variables.

This is all about the installation of Vert.x. In the command window, execute the vertx version. If the version information successfully prints out, the installation is completed.

Example 1

Now, let's build and execute a simple Web server with JavaScript which prints out "hello world". After writing the following codes, save it in server.js file. It is almost identical to the Node.js code.

load('vertx.js');

vertx.createHttpServer().requestHandler(function(req) {
req.response.end("Hello World!");}) .listen(8080, 'localhost');

Execute the created server.js application by using the vertx command as follows:

$ vertx run server.js

Open a browser and connect to http://localhost:8080. If you can see the 'Hello World!' message, you have succeeded.

Example 2

Let's see another example built other languages. The following code is written in Java. It shows a Web server that reads a static file and returns it as an HTTP response.

Vertx vertx = Vertx.newVertx();
vertx.createHttpServer().requestHandler(new Handler() {
    public void handle(HttpServerRequest req) {
        String file = req.path.equals("/") ? "index.html" : req.path;
        req.response.sendFile("webroot/" + file);
    }
}).listen(8080);

The following code is written in Groovy and provide the same functionality:

def vertx = Vertx.newVertx()
vertx.createHttpServer().requestHandler { req ->
def file = req.uri == "/" ? "index.html" : req.uri
req.response.sendFile "webroot/$file"
}.listen(8080)

Future of Vert.x and NHN

At NHN we have been observing Vert.x development since its preofficial release. We think highly of Vert.x. We have been communicating with the main developer, Tim Fox, since June 2012 to discuss ways to improve Vert.x. For example, Socket.io on Vert.x. Socket.io is available on Node.js only. So we have ported it to Java and sent a Pull Request https://github.com/vert-x/vert.x/pull/320 to Vert.x repository on Gitub. It's now merged to Vert.x-mod project.

Our effort, socket.io vert.x module, will be used for RTCS 2.0 version (vert.x + Socket.io) which is under-developing in NHN.

Node.js could remain very popular because of Socket.io. If Vert.x can use Socket.io, Vert.x may have many use cases. Furthermore, if this socket.io vertx module is used as an embedded library, it will be meaningful to use socket.io in Java based applications.

What is RTCS?

RTCS (Real Time Communication System) is a Real Time Web Development Platform created by NHN. It helps to transfer messages between a Browser and a Server in real time. RTCS has been deployed for NHN Web services such as Baseball 9, Me2Day Chatting, BAND Chatting and so on.

Wrap-up

The first version of Vert.x was released in May 2012. Compared to Node.js where the first version was released in 2009, the history of Vert.x is very short. Therefore, Vert.x does not have many references yet. However, Vert.x is supported by VmWare and can run on CloudFoundry. So, we expect that many references will soon be obtained.

By Seongmin Woo, Software Engineer at Web Platform Development Lab, NHN Corporation.

References

"Main Manual" http://vertx.io/manual.html
"Installation Guide" http://vertx.io/install.html
"The C10K problem" http://www.kegel.com/c10k.html
Gim Seongbak, Song Jihun "Java I/O & NIO Network Programming”, Hanbit Media 2004.

Understanding Encryption and Security through Java Cryptography Architecture

2013-04-08T18:36:20-07:00

For a long time, Java has provided security-related functions. Among the security-related functions, Java Cryptography Architecture (JCA) is the core one. JCA uses a provider structure with a variety of APIs related to security. These functions are essential for modern IT communication encryption technology, including Digital Signature, Message Digest (hashs), Certificate, Certificate Validation, creation and management of Key, and creation of Secure Random Number.

With JCA, even developers who do not have specialized knowledge of encryption can successfully implement security-related functions. You don't need to use algorithms like those you had to rack your brain for a long time to understand in computer science classes and cryptology-related classes. JCA allows you to implement the algorithms with a few lines of codes. Of course, utilizing the APIs well will be highly valuable for business. But, it does not mean that you do not need to understand how JCA runs. Understanding how JCA runs internally will be important to using the functions more efficiently.

To be a better software developer and architect, you may need to trace how the result, JCA, was created from the cryptology and security-related algorithms. This article is a summary of JCA architecture that I learned while producing the nClavis (Symmetric-key cryptography) at NHN. Of course, I do not understand all of JCA yet. However, I was so happy to understand JCA at this level that I decided to write this article to share my experience with you.

Design Principles

As I mentioned, JCA is a Java security platform, based on the provider structure, having implementation independence, implementation interoperability, and algorithm extensibility.

An application can utilize the information protection encryption technology just by requesting security services on the Java platform, without implementing security algorithms. JCA-provided security services are implemented by the provider mounted on the Java security platform. An application can introduce a variety of security functions by using several independent providers. The list of providers is described in the jre/lib/security/java.security file. The Java platform includes many providers and installs them by default when JRE is installed.

Code 1: java.security file.

#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider
security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.8=sun.security.smartcardio.SunPCSC
security.provider.9=sun.security.mscapi.SunMSCAPI

The providers mounted on the Java security platform by default are compatible with all Java applications and so widely used to regard them as trusted. Of course, JCA supports mounting of custom providers for applications which want to introduce the latest security technology that has not been implemented yet.

Architecture

Cryptographic Service Providers

All providers are an implementation of java.security.Provider. This provider implementation includes the list of security algorithm implementations. When an instance of a specific algorithm is necessary, the JCA framework searches the proper implementation class of the corresponding algorithm from the provider repository and creates a class instance. The providers defined in the java.security file are included in the repository by default. In this way, a provider can be statically included. In addition, it can be dynamically added in runtime. When several providers are defined, they may implement an identical encryption algorithm in different ways. In this case, an application can specify the provider or specify the preference in the repository.

To use JCA, an application simply requests a specific object type (such as MessageDigest) and an algorithm or service (e.g., MD5). Then, the application obtains an implementation from one of the installed providers. Of course, it can explicitly request an object of a specific provider.

Code 2: Requesting an Object of a Provider.

md = MessageDigest.getInstance("MD5");
md = MessageDigest.getInstance("MD5", "ProviderC");

Figure 1: Provider Framework of JCA (Source: http://docs.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html).

Oracle JRE (Sun JDK) has a variety of providers (Sun, SunJSSE, SunJCE, and SUnRsaSign) included by default. The criteria of classifying the providers are made based on the production process; functions or algorithms used by each provider are not so different from each other. JRE except Oracle has no need to include the providers mandatorily. Therefore, it is not recommended to implement an application in the provider-dependent way. All encryption technology implementations required by an application are provided by default and implemented with a fully reliable level. Therefore, developers do not need to pay attention to a provider itself.

Key Management

Two of the most important things in JCA are provider and key management. Java uses a kind of key database, called "keystore", to manage the key/certificate repository. KeyStore can be usefully used in an application that needs information for authentication, encryption, or signature.

An application can access the KeyStore by using the implementation of the java.security.KeyStore class. The default KeyStore implementation is provided by Sun Microsystems (the package name still starts with com.sun even though Oracle acquired it long time ago :D). The KeyStore is created as a file with the naming rule of “jks”. It can also be converted to the type of “jceks” or “pkcs12” in order to suppoprt applications which use another format of KeyStore implemetation.

"jceks" format is in PBE format which uses triple DES, which uses even stronger encryption algorithm than "jks" type, to protect KeyStore.

"pkcs12" format is a standard syntax to exchange personal information, based on RSA. For machines, applications, and browser Internet kiosks that support this standard, users can export, import, or activate the personally identifying information (certificates for identification, pkcs12 format certificates). Safari, Chrome, and IE browsers follow this standard. Therefore, when the pkcs12 certificate file is installed once, it is applied to all browsers. However, Firefox does not follow this standard.

KeyStore is to save and to manage SecretKey (SymmetricKey), Public/Private KeyPair (AsymmetricKey), self-signed certificate, and the certificates signed by trusted CA (Certificate Authority) or the private CA on a file. Here, experienced developers may recall OpenSSL which was used to create a certificate file. Both certificate files have the same purpose but different file format. However their formats are convertible.

Java even provides 'keytool' command-line utility in JDK_HOME/bin directory, which is similar as OpenSSL's utility. Therefore you can handle certificates using a keytool on Windows, unlikely OpeSSL only on Linux. Of course, this tool runs with the KeyStore implementation provided by Oracle JRE. If JDK is installed in the system, a certificate can be created by using the keytool. However, note that the keytool can provide the functions at the same level provided by OpenSSL from JAVA. A KeyStore file created by using the keytool is compatible with the lower Java versions, so you do not need to worry about it.

In-depth 1: Certificate.

Here, I need to address the correct meaning of certificate. In a narrow sense, KeyStore is a kind of certificate. A certificate is used for two purposes; a "lock" required for encrypting the information and a tool for "identification" to identify the opponent technically. The authenticated certificate used for bank transactions is a security technology which utilizes both purposes of a certificate.

The cryptographic meaning of a certificate is an electronic document that uses a digital signature to sign the public key created by the RSA algorithm (asymmetric-key cryptography) with the private key of the certificate authority (CA).

It is popular practice that a pair of keys created by using the RSA algorithm is solid. It had been proven long ago that calculating the other key by using one key within a meaningful time is impossible. So, why is electronic signature required? When "A" and "B" communicate with each other by encrypting their data, A opens its public key and then protects the private key paired with the public key. Then B encrypts the data to be sent to A by using the public key of A. In this case, there is a problem of how B will know it can trust the public key; is it really provided by A? When a malicious attacker "C" disguises its public key as A's to deceive B, the public key cannot be used. To solve this problem, a trusted certificate authority "D" is necessary. The CA has a chain structure from the top root certificate authority to the sub certificate authority. In this structure, the upper layer certifies the lower layer by using signature. As it had been rooted as a worldwide standard so long ago, the signature chain of the certificates around the world includes a few common top root CAs (e.g. VeriSign, Thawte). Countries of which IT communication environment reaches a certain standard have their national root CA (e.g., KISA in Korea). The top root CAs may have a circular structure that allows signatures with each other. In some cases, one CA can sign the public key with its private key.

So, the signature of CA should not be valid limitlessly but updated regularly or irregularly by other security events.

It is possible for an individual or a company to establish a private certificate authority if required. Certificates signed by the private certificate authority have a more complex certification process. You may have seen the following Figure 2 on your browser:

Figure 2: Browser Display when Private Certificate Authority is used.

It is displayed when the website uses the private certificate authority. On the browser (even though it is not recommended), the user can skip this situation by clicking the mouse. However, for server-to-server connection, the opponent's certificate issued by the private certificate authority should be imported to JAVA_HOME/jre/lib/security/cacerts or added to SSLContext when creating the connection of the program code.

The certificates from official certificate authorities are widely granted as trusted CAs and included to the OS or JRE by default. Therefore, the situation illustrated in Figure 2 does not occur. If a private certificate authority can obtain the pkcs12 format certificate, the private certificate authority is considered as a trusted CA. Therefore, it can be easily installed in the system.

Code 2: Certificate Verification Program installed in the system.

@Windows>certmgr.msc
@MAC OS X>Keychain Access
@Linux> keychain

In-depth 2: HTTPS

To understand encryption, you need to understand SSL/TLS, the certificate-based cryptographic protocols, as well as certificate. The purpose of a certificate can be easily misunderstood when encrypting the HTTPS protocol communication section. When a server and a client communicate via HTTPS, if only the communication section is encrypted without identifying the client (setting the clientAuth attribute of HTTPS Connector to false in the server.xml setting of Tomcat), only a simple certification verification is executed. Symmetric-key cryptography is used for encryption of data.

Here, I will describe how a certificate is used through HTTPS.

A client connects to a server via the HTTPS protocol (at this time, the SSL connection-defined server port is used).
The server sends its certificate public key to the client.(including several meta information for validation and Cipher supported by the server)
The client validates the server with the public key and meta information sent by the server.(checking whether the public key is signed with a trusted official Root CA)
1. When the certificate of the server is signed with the official CA, it is passed (the official CAs are registered to the system as a trusted authority by default).
2. When the certificate of the server is signed with a private CA, checking the trust manager of the SSL socket (SSLContext) created by the client. If the certificate is registered, it is passed.
When the server's certificate passes the validation check, the client creates a symmetric key, encrypts the symmetric key and the cipher as a public key of the server, and then sends them to the server.(the symmetric key is created by selecting one of cipher algorithms supported by the server)
The server decodes the [symmetric key and cipher: encrypted as the public key of the server] to its private key and acquires the symmetric key to be used for encrypted communication.
After that, data communication between the server and the client is made to be encrypted with the symmetric key created by the client.

JCA Structure

JCA structure can be described with Engine class and algorithm. In JCA, an Engine class provides interfaces for all encryption service types regardless of a specific encryption algorithm or provider. The engine class provides one of the following functions:

Encryption operations (encryption, digital signatures, message digests, etc.)
Creating and converting the elements (keys and algorithm parameters) required for encryption
Objects (keystores or certificates) which imply the encryption data or can be used by an object or the upper abstraction layer

Let's take a look at the Engine classes provided by JCA and talk about encryption in detail.

SecureRandom

SecureRandom class is used to create a Pseudo Random number. In Java, random refers to Pseudo Random, as a more accurate expression. If so, are random and pseudo random different? Technically, they are different. There are two random types: True Random and Pseudo Random.

True Random is a random number which cannot be forecasted. You may say that Pseudo Random cannot be forecasted. Pseudo Random is a random progression determined by seed and a mathematical algorithm. It has a sequence which is eventually repeated even if it takes a very long time or its probability is very low. In addition, if you know the seed and the random algorithm, you can forecast the sequence of the Pseudo Random. True Random creates a random number based on atomic physical phenomena, not the mathematical way used by the Pseudo Random. If there is no hardware equipment to measure the atomic physical phenomena, e.g., electromagnetic noises and radioactive element decay, it is impossible to create True Random.

JCA SecureRandom class is an engine class that provides a powerful function to create random numbers. As I described, it is not easy to implement a True Random Number Generator (TRNG). Therefore, many implementations implement Pseudo Random Number Generator (PRNG). As mentioned before, the random level of the pseudo random is incomplete. The popular random class cannot satisfy the minimum level that is required cryptographically. Therefore, the implementation of SecureRandom should be verified that it satisfies the requirements of cryptographic level (CSPRNG).

Figure 3: Classification of Encryption Type (source: http://en.wikipedia.org/wiki/Cipher).

Now, you may ask why creating the random number is considered such an very important thing for encryption.

The core of modern cryptology is the key used for encryption. Previously, cryptology had been based on a conversion table like Base64 or UTF8 encoding. For modern cryptology, that kind of traditional method is not considered as encryption any more. The key is a random sequence generated by the random sequence generator. We naturally think of an encryption algorithm as thinking of cryptology. However, the open symmetric key encryption algorithm can be simplified to XOR operation (or multiplication/division operation) for the input values and key streams. As I said, the core is the key.

If the random level of a random sequence generator is not ensured, the entire outline of the key may be revealed to an attacker when a part of created key or some sequential random sequences are leaked out to the attacker. For modern cryptology, the key used for encryption is the core. So, the random sequence generator is very important.

JCA uses SecureRandom as the random sequence generator. Implementation of an algorithm to create random numbers is provided by a provider, like other encryption algorithms.

MessageDigest

MessageDigest is used to calculate the message digest (hash) of input data.

The purpose of message digest is the integrity check to check whether the original file is reserved as it is. Message digest algorithm processes a variable-length original message into a fixed-length hash output. Message digest algorithm consists of a unidirectional hash function, so it is not possible to draw the original value from the hash value. When A and B are communicating with each other, A sends the original message, message hash value of the original message, and the message digest algorithm to B. B calculates the message hash value by using the algorithm and original message sent from A. When the message hash value calculated by B is identical to the message hash value sent from A, it means that the original message sent from A has not been changed or modified until B receives it via the network.

Figure 4: Example of Using MessageDigest at a Download Site.

You can frequently see Checksums or digital fingerprints at download sites. It is an alternative name for MessageDigest. MD5 or SHA1 is the well-known message digest algorithm.

Signature

Signature is used to sign data and to decide validity of the digital signature with a key received during initialization. Receiving a key means that key-based encryption is executed.

Figure 5: Flow of Actions of Signature Object (source: http://docs.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html).

In initialization, Signature Object receives the private key and the original data to be signed as parameters and finishes preparation for signing. The sign() method of Signature signs the original data with the private key and returns Signature Bytes. To validate the signed data, the verification signature object is initialized by using the public key paired with the private key used for signing. The object additionally receives the original data, signature output and Signature Bytes, and the verify method checks whether the two parameters are identical to determine the reliability of the original data. Signature can be made only by the person who holds the private key. However, verification is made by using the public key. So, anyone who acquired the public key can perform verification.

Digital Signature vs Cryptography vs MessageDigest

For cryptography, users can select either symmetric key method or asymmetric key method based on the user's request. Digital signature is also a kind of cryptography. However, asymmetric key encryption is a prerequisite for digital signature. In addition, digital signature is a combination of MessageDigest and asymmetric key encryption. Large-capacity data with a variable length is compressed to a fixed-length format which is easy to manage by the MessageDigest and then signed with a private key to create fixed-length signature bytes. When creating a signature instance, you can see the principle of digital signature from the signature algorithm names, such as SHA1withRSA, MD5withRSA, SHA1withDSA; the signature algorithms are sent as a signature.getInstance() parameter and their names are made by combining RSA (asymmetric key encryption algorithm), MessageDigest algorithm, SHA1, and MD5.

Signature dsa = Signature.getInstance("SHA1withDSA");

Signed Certificate

Keystore Type: jks
Keystore Provider: SUN

Keystore includes the following two items:

Alias: rootcaalias
Written on: 2012. 9. 26
Input Type: trustedCertEntry

Holder: CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC
Issuer: CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC
Serial Number: 
Opened on: Fri Apr 06 10:17:08 KST 2012 Expired on: Sun Mar 13 10:17:08 KST 2112
Certificate Fingerprint:
MD5: 0C:FC:12:C5:68:E5:95:0B:95:7D:B0:2F:FA:4F:DB:B4
SHA1: 90:37:1C:E6:F4:64:AD:E6:27:AA:4F:58:88:16:11:24:6D:A5:EB:2B


*******************************************
*******************************************

Alias: nplatform
Written on: 2012. 9. 26
Input Type: keyEntry
Length of Certificate Chain: 2
Certificate[1]:
Holder: O=NHN INC, OU=NHN NBP, CN= NPLAFORM, UID=1
Issuer: CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC
Serial Number: 
Opened on: Fri Sep 21 17:26:22 KST 2012 Expired on: Sun Aug 28 17:26:22 KST 2112
Certificate Fingerprint:
MD5: 48:8C:46:A3:E7:54:58:97:60:0D:5C:56:08:B0:D1:E7
SHA1: 12:64:3C:DA:C1:2C:94:1A:2B:EB:E9:98:2B:DA:8F:06:78:6E:26:1E

Certificate[2]:
Holder: CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC
Issuer: CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC
Serial Number: 
Opened on: Fri Apr 06 10:17:08 KST 2012 Expired on: Sun Mar 13 10:17:08 KST 2112
Certificate Fingerprint:
MD5: 0C:FC:12:C5:68:E5:95:0B:95:7D:B0:2F:FA:4F:DB:B4
SHA1: 90:37:1C:E6:F4:64:AD:E6:27:AA:4F:58:88:16:11:24:6D:A5:EB:2B

Let's review certificate and signature, which were described in the previous in depth section, with JCA Signature object mechanism.

The above text box is the KeyStore certificate file created by using the keytool of Java. From Holder and Issuer of Certificate[1], you can see that “CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC” has signed the certificate of Holder “O=NHN INC, OU=NHN NBP, CN= NPLAFORM, UID=1” by using its private key. The result of the signature is the Certificate Fingerprint. The length of the Certificate Fingerprint is decided by the MessageDigest algorithm (MD5 or SHA1). As following the Certificate Chain, you can see that the Holder and Issuer of Certificate[2] are identical. It means that “CN=NSYMKEY Root CA, OU=NHN NBP, O=NHN INC” is self-signed by using its private key. As the Certificate[2] has self-signed, the Certificate Chain is ended here.

Cipher Class

Figure 6: Flow of Actions of Cipher Object (source: http://docs.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html).

Cipher class provides encryption/decryption functions. The encryption/decryption algorithms are variously classified as follows: Symmetric bulk encryption (AES, DES, DESede, Blowfish, IDEA), Stream encryption (RC4), Asymmetric encryption (RSA), and Password-based encryption (PBE).

I will not describe classification of encryption to Symmetric and Asymmetric because the classification is so well-known.

Stream vs. Block Cipher

Symmetric bulk encryption can be classified into Stream and Block Cipher. Block Cipher encodes the data in the fixed-length block unit. Data whose length does not fit the fixed length is padded with dummy values. Bytes padded are removed while decrypting the data. This padding is executed by the padding type (e.g., PKCS5PADDING) which is sent as a parameter while initializing Cipher. On the contrary, Stream Cipher processes input data in the unit of byte or bit. Therefore, it can process variable-length data without padding.

Modes Of Operation

The important concept of Block Cipher you should know is Feedback Modes. Assume a very simple block cipher. If the input data is identical, the encrypted result is identical. From this characteristic, attackers obtain a hint to decrypt the encrypted data with a repeated same pattern.

To avoid security vulnerabilities and make Cipher more complex, Feedback Mode was introduced. Feedback Mode is an operation which combines (XOR operation) the Nth input data block (or the Nth encrypted result data block) and the N-1st input data block (or the N-1st encrypted result data block) at the Nth encryption process. Therefore, when the input data blocks are identical, the result values are different corresponding to the variables used in the previous encryption process. Note one more thing: if N = 1, any variable cannot be acquired from the N-1st encryption process. In this case, Initial Value (IV) takes the role instead of the variable in the previous process. To use Feedback Mode, the IV value should be randomly created and prepared for encryption. The IV value used for encryption should be stored because it is necessary for decryption as well.

The feedback modes provided by JCA are CBC, CFB, and OFB. The mode that no feedback mode is used is called ECB for distinction. More detailed description of each mode will not be provided here.

Figure 7 shows the importance of feedback modes. If the original image data is encrypted without using the feedback mode (ECB MODE), identical input data is used and an identical encryption result is acquired. Therefore, the entire outline is drawn up.

Figure 7: Image Encryption (source: http://en.wikipedia.org/wiki/Modes_of_operation).

Creating Cipher Object

The essential thing for creating a Cipher instance is to specify transformation. Transformation consists of encryption algorithms (/feedback modes/paddings) described before. Only the encryption algorithm values can be specified. But, the default feedback mode/padding (ECB/PKCS5Padding) is internally specified.

Cipher c1 = Cipher.getInstance("DES/ECB/PKCS5Padding");

or

Cipher c1 = Cipher.getInstance("DES");

The Cipher class instance can be initialized by selecting one from four modes (opmode: Encryption, Decryption, Wrap, Unwrap) for initialization.

WRAP_MODE: Wraps Java.security.Key to convert it to the byte unit for secured key transmission
UNWRAP_MODE: Unwraps the wrapped key to the Java.security.Key object

When initializing the cipher class instance, the init() method is called as its parameter. It requests opmode, key(certificate), params, and random as its parameters. Here, note the AlgorithmParameters-type params parameter. This instance is used to store the IV value of feedback mode and the salt value and the iteration count value of the PBS algorithm. These values are not required when initializing cipher of ENCRYPTION_MODE. These can be randomly created by ScureRandom and used for the encryption process. The values created are stored in the AlgorithmParameters field of the encryption cipher object. On the other hand, the params value is required for initializing DECRYPTION_MODE Cipher. In the decryption process, the params value identical to the value used for the encryption process is required. When the init() method is called, all existing values are deleted from the cipher class. Therefore, before initializing the cipher instance again, the getParameters() method should be called to store the AlgorithmParameter object used for the encryption process.

To make the jobs simpler, SealedObject can be used in the encryption result. SealedObject class receives a target statement to encrypt and Cipher object as arguments(sealing process in SealedObject class). The SealedObject itself is an encrypted data and it manages algorithm arguments used for encryption. If a key used in encryption process is passed, you can obtain decrypted data(unsealing process in SealedObject class).

Code 4: Encryption using SealedObject.

//Create Cipher object
Cipher c = Cipher.getInstance("DES");
c.init(Cipher.ENCRYPT_MODE, sKey);
// Create SealedObject: it is an encryped data
SealedObject so = new SealedObject("This is a secret", c);

Code 5 Decryption using SealedObject

//Note: sKey is as same as encryption key
//Note: so is SealedObject which was previously created. 
//Decryption using SealedObject #1
//Decrypt using Cipher object
c.init(Cipher.DECRYPT_MODE, sKey);
try {
	String s = (String)so.getObject(c);
} catch (Exception e) {
	//do something
};
//Decryption using SealedObject #2
//Decrypt using encryption key
try {
	String s = (String)so.getObject(sKey);
} catch (Exception e) {
	//do something
};

Message Authentication Codes(MAC)

MAC is similar to MessageDigest because it creates the hash value; however, it is different from MessageDigest in that it requires SecretKey (symmetric key) for initialization. MessageDigest allows any receiving party to execute integrity check for the received message. However, MAC allows only the party which has the identical SecretKey to execute integrity check for the received message. MAC is used among those who share the SecretKey.

Figure 8. Flow of Actions of MAC (source: http://docs.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html).

HMAC is a MAC based on the encryption hash function (MessageDigest algorithm: MD5 or SHA1). HMAC is a combination of MessageDigest algorithm and shared SecretKey.

Signature is different from HMAC because Signature uses the asymmetric key. HMAC allows identifying the opponent faster than the signature that uses the RSA algorithm. So, some services strategically use HMAC.

Conclusion

So far, I have described half of JCA functions. However, the rest of the functions are also important even if they are not described here. This article does not include the other core of JCA, such as Key, KeyPair, KeyFactory, KeyGenerator, KeyStore, CertificateFactory, and CertStore. I think that the functions should be deeply and fully described. For lack of space, I won't deal with the functions here. They will be described in the next article if possible.

It was very difficult to study JCA and prepare this article. I felt that there were more things to study and research as I prepared the article and was left with even more questions while writing. I hope my article will help you to understand the "vague" concept of encryption more clearly.

By Jaehee Ahn, Software Egnineer at Web Platform Development Lab, NHN Corporation.

References

Database Sharding Platform at NHN

2012-04-09T22:40:32-07:00

"Ins and outs of NHN" is a series of articles that compares platforms and services from third-party vendors with NHN's own solutions. The topic of this first article in the series is Database Sharding Platform. I will introduce about the efforts being made from inside and outside of NHN to implement Database Sharding. I will first explain the concept of Sharding data vs. Partitioning data. Then review the common methodology to implement sharding. And finally I will compare all sharding platforms. This article is recommented for developers interested in big data management.

Database Expansion

To store and search a volume of data which is so big that it cannot be handled by one database, you must find a way to use multiple databases. Although there are some databases made for distributed environments, such as Cassandra or Dynamo, these have many functional constraints, such as weakness in terms of search range or inability to use the JOIN operations. In order to expand data while using a relatively feature-rich functionality, it is recommended to use RDBMS by sharding the databases.

In the past, the Sharding logic was implemented directly on the application layer, but now there is an increasing number of examples which introduce Sharding platforms which allow to move the Sharding logic from the application layer to database or middleware layers. At a core level, Sharding platforms must respond effectively to ever-increasing data without failure, and handle different data characteristics and models depending on services.

In this article I will compare Spock Proxy, the Sharding platform based on MySQL Proxy, Gizzard, created by Twitter, and CUBRID SHARD, native database sharding feature in CUBRID which is set to launch in the first half of 2012. A table of other solutions was previous posted at Database Sharding with CUBRID.

Horizontal Partitioning and Sharding

Horizontal partitioning is a design that divides and stores data, such as a schema, into two or more tables within one database. For example, to handle large data of user messages, such schema may be created so that messages by users from city A are stored in one table, while messages by users from city B are stored in another table. This allows to reduce the size of the index and increase the concurrency of operations. The key point in this approach is that the data is partitioned between tables within a single database.

Sharding, on the other hand, is the distributed approach where data is horizontally partitioned between tables created in physically different databases. Thus, Sharding is a method to store the messages by users from city A in database A and messages by users from city B in database B. Here each database is called a Shard.

As you have to work on multiple databases, there could be functional limitations based on circumstances and also drawbacks in terms of consistency and replication, including JOIN operations. In many cases, Sharding used to be implemented at the application server level. There have been many attempts to provide this at the platform level. These can be classified into a pattern that operates in the application server, such as Hibernate Shards, a middle tier pattern such as CUBRID SHARD, Spock Proxy, or Gizzard, and a pattern that provides the Sharding functionality from a database itself, e.g. nStore or MongoDB.

Middle Tier Sharding Platform

By default, a Sharding platform should consider the following items:

Database location abstraction
Scalability
Monitoring/Ease of operations

Database location abstraction and scalability are different from each other but connected. Database location abstraction ensures that on the application layer you do not need to know which data (which Row) is located in which database. The application is connected only with the Sharding platform. Connecting to a database is what the Sharding platform should do. In addition, Sharding platform should carry out the task to add a replicated storage to a specific Shard (one of the partitioned databases) in order to migrate database for replacement without restarting or changing the application code. When the Sharding platform comes to a stop, it is obvious that an application server will not be able to access the database. For this reason, the Sharding platform should provide redundancy.

For monitoring, the Sharding platform should be able to provide a number of requests and error information according to Shard keys (a standard to determine which data is stored in which database).

Comparison between Spock Proxy and CUBRID SHARD

The key function of the Sharding platform is the CRUD operation (CREATE, READ, UPDATE, DELETE) which chooses one database among many according to 'a standard set by a developer', in other words the Sharding strategy. We will compare Spock Proxy, a typical Sharding platform of MySQL, with the CUBRID SHARD platform developed by NHN.

Spock Proxy

Spock Proxy is a Sharding platform designed based on MySQL Proxy. In MySQL Proxy, you can execute the Lau scrip code, which is written by a developer before and after performing SQL. The primary purpose of using MySQL Proxy is to analyze and modify SQL. To use Spock Proxy, you need to create a MySQL database to manage the information about shards and how the data should be distributed. The row is a rule for Sharding.

Figure 1: Specifying Sharding rules in Spock Proxy.

CUBRID SHARD

CUBRID SHARD is the Sharding platform for CUBRID. The uniqueness of CUBRID SHARD is that it can also be used with MySQL as well as Oracle. It is set to be launched in the first half of 2012, and is planned to be used for processing the meta information database system of NDrive service, the cloud storage system developed by NHN.

The following table shows a summary of a comparison between Spock Proxy and CUBRID SHARD.

Table 1 Comparison between Spock Proxy and CUBRID SHARD

	Spock Proxy	CUBRID SHARD
Sharding rule storage	DBMS Table	Configuration file
How to create Shard keys	Modulo	Modulo Developer's own sharding strategy provided in a library
How to find a Shard key	SQL parsing	Using HINT
Strength	No need to change SQL	Supports CURBID, MySQL, and Oracle Higher performance
Weakness	Lower performance due to extra SQL parsing Supports MySQL only	Requires the change to SQL queries to insert sharding HINT

Spock Proxy stores Sharding rules in the table in MySQL universal_db database. The SQL received from an application server is parsed and checked whether the query has shard keys. If shard keys are provided, the MySQL instance will be identified according to the standard recorded in universal_db, then SQL will be relayed to that MySQL instance.

When using this method, you do not need to describe information related to shard keys in SQL, unlike in CUBRID SHARD. Therefore, if you did not use Sharding before for your coding, but recently had to use due to the data increase, you may use Spock Proxy which will work without requiring you to change the SQL in your application. Note that this is limited to cases where there is no need to change the schema for Sharding, or when Sharding can be applied without changing the SQL that you use. However, the method used to find the Shard after parsing SQL, as used by Spock Proxy, has weaknesses in terms of performance. It could lead to unnecessary work, as SQL should be parsed twice: once by Spock Proxy to determine the MySQL instance, then by MySQL itself.

In CUBRID SHARD, SQL HINT is used which allows to avoid parsing SQL twice. Suppose there is a table as follows:

student
student_no	name	age
...	...	...

To use the student_no column data as a shard key, an application server sends the following prepared SQL to CUBRID SHARD.

SELECT student_no, name, age FROM student WHERE student_no = /*+ SHARD_KEY */ ?

CUBRID SHARD checks if a HINT displays /*+ SHARD_KEY */ in SQL, in which case the column data with the corresponding HINT will be used as a Shard key. It then reads the student_no value, which follows the hint, and identifies RDBMS based on the configuration file and transmits the corresponding query. As such, the benefit of using HINT is that you can improve processing efficiency by avoiding parsing SQL twice, and react to various RDBMS without violating the database location abstract.

CUBRID SHARDING provides various HINTs, in addition to /*+ SHARD_KEY */.
Typically, there is /*+ SHARD_ID(__id__) */, a HINT that allows you to find a special shard.
The /*+ SHARD_VAL(__shard_key_val__) */ HINT can also play a role in finding a special shard for tables which have no shard keys. While this HINT is the same in that it searches for tables that have no specific shard keys, it configures a value for shard key column directly without choosing a shard, and selects the shard according to the internal rules of the middleware.

Unlike Spock Proxy, where Sharding rules are inserted into a database table, in CUBRID SHARD Sharding rules are specified in the configuration file. If there are three access addresses for actual RDBMS storage, you should specify DB addresses to CUBRID SHARD as follows.

0     shardDB     shardNODE1:3306
1     shardDB     shardNODE2:3306
2     shardDB     shardNODE3:3306
3     shardDB     shardNODE4:3306

If you want to use the Modulo method, write the corresponding value in the configuration file as below.

SHARD_KEY_MODULAR = 256

In addition, specify [MIN..MAX] according to the value generated from SHARD_KEY_MODULAR, and describe which shard to send the corresponding query to.

#min max shard_id
0         63       0
64       127     1
128     191     2
192     255     3

If you need a more subtle method than Modulo, you can create program code of your own that calculates the Shard ID for Shard keys.

SHARD_KEY_LIBRARY_NAME = libshardkeyid.so
SHARD_FUNCTION_NAME = user_get_shard_key

As shown in the above example, you can register your own library which will calculate the Shard ID logic.

The common weakness of both Spock Proxy and CUBRID SHARD is that they both require additional network IO time for each additional hop because they are implemented as a middle tier.

The following figure displays a general process of internal execution in CUBRID SHARD, which is performed when a developer executes a query. When a developer executes a query, the query is analyzed by the DB shard middleware, which determines to which shard it will be sent. Then the query is transmitted to the selected shard, and finally the middleware delivers the response to the client.

Figure 2: CUBRID SHARD process.

Gizzard

Gizzard is a Sharding platform developed by Twitter. It is also a middle tier just like Spock Proxy and CUBRID SHARD. However, its usage and architecture are quite different from Spock Proxy or CUBRID SHARD.

The following figure shows how the data can be shared between several databases deploying two units of Gizzard.

Figure 3: Gizzard deployment diagram.

In this case, the interface being used by an application server is Thrift, an RPC protocol, not JDBC. Therefore, it is similar to DBGW (CUBRID SHARD) platform developed by NHN. When the schema of a storage changes, Gizzard may also need modifications. While it can be a constraint, it could also open up access not only to RDBMS, but also to various other databases (e.g. Lucene).

Gizzard is written based on Scala, and you can expand its functions by adding Scala codes as needed. Instead of viewing it as a complete product, we recommend that you download the source and modify it to suit your needs.

The biggest advantage of Gizzard is that you can perform hotspot response and database migration in the middle tier platform level. For example, Gizzard provides the direct replication feature, as shown in the following figure.

Figure 4: The replication feature of Gizzard.

Spock Proxy or CUBRID SHARD do not offer this feature, as they do not need it. CUBRID, MySQL, and Oracle have their own replication feature, thus they do not need to offer the feature from the middle tier.

The replication feature provided by Gizzard is very simple - send each request to several replicas - but cannot handle variety of requests that may occur while running the replication feature. However, it can be useful when using a database without the replication feature.

Figure 5: The migration feature of Gizzard.

As shown in the figure above, it offers the replication feature which does not require a service halt. When migrating a certain database, the feature replicates the details of a database storage before replication configuration, after allowing the new data to be written in two locations by configuring the replication. This approach, of course, cannot be used on RDBMS due to consistency issue.

Limitations of Sharding

Sharding, on the other hand, also has its own limitations. Typical constraints of Sharding are as follows:

It cannot perform JOIN operations for two or more Shardings.
auto_increment (serial) values can vary depending on Sharding.
last_insert_id() value is no more valid.
shard key column value must not be updated; requires delete, then insert.
does not allow to access two or more Shards from one transaction.

When using Sharding, therefore, it is important that you perform right data modeling and schema design to prevent the above constraint issues.

To learn more about Sharding and other features of CUBRID I suggest you to see the Slideshare presentation at:

By Jeon Won Hee, Senior Software Engineer at CUBRID DBMS Development Lab, NHN Corporation.

Basic Operations of CUBRID Processes

2012-08-20T19:20:30-07:00

This article is a part of "CUBRID Internals" series. In the previous article I have explained about CUBRID Processes and where to configure them. In this article I would like to dive into how those processes run. If you have not read the previous article, I recommend you to do so before reading further.

Figure 1 below illustrates the basic operation flow in CUBRID.

First, when a user enters the cubrid service start command, CUBRID gets started.
Then, the cub_master and cub_broker processes are started. At this time, a number of cub_cas processes will be started which corresponds to the value of MIN_NUM_APPL_SERVER parameter in the cubrid_broker.conf file.
Then, a user enters the cubrid server start demodb command which creates the cub_server process that mounts the demodb database volume. As described in CUBRID Processes, the cub_master process connects the cub_server process with the cub_cas process, which sends the requests, or the csql program.

Figure 1: CUBRID Operation Procedure.

As shown in Figure 1, if a JDBC application is connected to the server process which mounts the database volume file, the SQL statement can be executed in CUBRID. Two most popular methods to use CUBRID is to execute SQL manually using the command line CSQL Interpreter program or to write a program which uses various APIs like JDBC, PHP, Node.js, etc.

Note:
As we have discussed, the JDBC program connects to the database server through a broker while CSQL Interpreter directly connects to the database server bypassing the broker. This is an important difference between APIs and CSQL.

The CUBRID Manager database administration tool is developed on top of the JDBC driver. The SQL statement executed in CUBRID Manager are passed to the server through the JDBC API. Other database management functions that are available in CUBRID Manager (database creating/deleting, etc.) are executed through a management utility called CUBRID Manager Server which runs as a separate manager server daemon outside of the database.

Now, let's take a look at CUBRID Manager as an example to see how an SQL statement is executed in CUBRID. Refer to Figure 2 for this example.

The SQL statement entered in the query editor of the CUBRID Manager (for example, SELECT * FROM olympic WHERE host_year > 1988 LIMIT 4;) is sent to CUBRID JDBC driver. This assumes that a database has already been selected in CUBRID Manager before executing this SQL, and the JDBC connection has been established.
In the JDBC connection process, the connection is made to the port of the host specified in the JDBC connection information (JDBC Connection URL).
The cub_broker process receives the connection and allocates a cub_cas process for the session to the connection.
Then, it sends the socket connection to the cub_cas process so the connection between the JDBC driver and the cub_cas process is established.
Back to the SQL statement execution, the JDBC driver included in the CUBRID Manager sends the SQL statement of the query editor to the connected broker, the cub_cas process.

The broker sequentially calls db_open_buffer(), db_compile_statement() and db_execute_statement() among C APIs provided by the client library, to execute the received SQL statement. The db_open_buffer() function parses the SQL statement, the db_compile_statement() function compiles the execution plan, and the db_execute_statement() function executes SQL by sending XASL (eXtended Access Spec Language) to the server.

Figure 2: SQL Statement Execution Procedure in CUBRID.

As shown in Figure 2, the qmgr_execute_query() function is executed in the cub_cas, and the xqmgr_execute_query() function is executed in the cub_server process. The qmgr_execute_query() function in the client is the xqmgr_execute_query() function in the server. As shown above, CUBRID implements a communication interface of Remote Procedure Call (RPC) between a client and a server. In the server, the qexec_execute_query() and qexec_execute_mainblock() functions of the query processing module are used to execute XASL.

If you are interesting how queries are processed in CUBRID, I recommend you to read CUBRID Query Processing which provides the detailed step by step explanation using real examples.

By Park Kieun, Senior Software Engineer and Architect at Service Platform Development Center & IT Service Center, NHN Corporation.

Announcing CUBRID 9.1 stable release with big improvements

2013-03-15T03:51:03-07:00

We released CUBRID 9.0 beta version in October last year. Since then we have been working hard on stabilizing the beta features, fixing bugs, and improving the overall engine performance. Today I am excited to announce the immediate availability of the CUBRID 9.1 stable release. You can download CUBRID Database Server from http://www.cubrid.org/?mid=downloads&item=cubrid&os=detect&cubrid=9.1.0.

I would also like to announce that we will give a talk about CUBRID Database Sharding at Percona MySQL Conference on April 24, 2013, in Santa Clara, CA. Join us there to meet CUBRID Engineers and get the first-hand insight into the new CUBRID 9.1.

Below I will provide an overview of the latest changes and improvements in CUBRID 9.1.

Overview

CUBRID 9.1 is an upgraded and stabilized version of CUBRID 9.0 Beta. To learn more about the biggest features introduced in 9.x family, refer to 9.0 official announcement. Issues found in the 9.0 Beta version have been fixed and stabilized in this new 9.1 stable release. With a variety of query-related functionalities, CUBRID 9.1 offers improved query processing performance as well as improved query optimization. In addition, its multi-language related functionalities have been further improved. This new 9.1 release is accompanied by new CUBRID Tools and Drivers releases.

Backward Compatibility

Database compatibility

As a database volume of CUBRID 9.1 is not compatible with the database of CUBRID 9.0 Beta, users of CUBRID 9.0 Beta or previous versions should migrate their database. We have created a migration instructions which you can find in Upgrade section of the Release Notes.

Driver compatibility

The JDBC and CCI driver of CUBRID 9.1 are compatible with CUBRID 9.0 Beta and CUBRID 2008 R4.x version. Some features that are fixed and improved for 9.1 are not supported when 9.1 drivers connect to the previous versions.

Major enhancements

New SQL functions and index hints

New SQL analytics functions like NTILE, LEAD and LAG have been introduced in CUBRID 9.1.
WIDTH_BUCKET new SQL numeric function is also introduced.
TRUNC and ROUND functions now also accept the date types.
New SQL Hints:
- Support a new index hint clause.
- SQL hints for Multi UPDATE and DELETE statement.
- SQL hints for MERGE statement.

Performance improvements and optimizations

The performance of data replication in HA environment has been significantly improved in CUBRID 9.1.
Improved multi-key range optimization.
Enhanced optimization of ORDER BY and GROUP BY clause.
Improved analytic function performance.
Improved performance of INSERT ON DUPLICATE KEY UPDATE and REPLACE statement.
Improved search and delete performance for non-unique indexes with many duplicate keys.
Improved delete performance when insert and delete operations are repeated.
The overall performance of SELECT operations has been improved by nearly 20%.
Based on the results obtained from the basic performance test, we have found that the performance of INSERT, DELETE, and UPDATE operations are almost same as that of 9.0 Beta.

Multi-language support

In CUBRID 9.1 we now support collation for tables.
SHOW COLLATION statement and new CHARSET, COLLATION, and COERCIBILITY functions are now supported.
Support collation with expansion which sorts French with backward accent order.
Improved and fixed restrictions and issues of 9.0 Beta version.

CUBRID SHARD

We have added cubrid shard getid command to verify the shard ID of the shard key.
CUBRID SHARD is now available on Windows as well.

Administration utility

cubrid applyinfo utility now also shows information about the replication delay.
cubrid killtran utility now has the ability to show the query execution information of each transaction as well as the ability to remove transactions which executes a designated SQL.
When a query timeout occurs, added a functionality to log the query execution information to the server error log and the CAS log files.

Behavioral Changes

CUBRID_LANG environment variable is no longer used.
CUBRID_CHARSET environment variable which sets the database charset instead of CUBRID_LANG and the CUBRID_MSG_LANG environment variable which sets the charset for utility and error messages.
Change array execution functions such as cci_execute_array, cci_execute_batch function and Statement.executeBatch and PreparedStatement.executeBatch method of JDBC to commit whenever it executes an individual query under auto commit mode, while the previous versions commit once for entire execution.
Change the behavior of cci_execute_array, cci_execute_batch and cci_execute_result function when an error occurs while they are executing multiple statements. These functions now continue to execute the entire given queries while the previous versions stop execution and return an error. Users can access the results and identify the errors with CCI_QUERY_RESULT_* macros.
OFF is no longer supported for KEEP_CONNECTION broker parameter.
SELECT_AUTO_COMMIT broker parameter is no longer supported.
Change the allowed value range of a broker parameter APPL_SERVER_MAX_SIZE_HARD_LIMIT to 1 - 2,097,151.
Change the default value of a broker parameter SQL_LOG_MAX_SIZE from 100 MB to 10 MB.
Change the behavior of the call_stack_dump_activation_list parameter.

Numerous Improvements and Bug Fixes

Fix many critical issues of the previous versions.
Improve of fix many issues of stability, SQL, partitioning, HA, Sharding, utilities, and drivers.

For more details on changes, see the Release Notes in English or Korean.

So far CUBRID 9.1 is our biggest release which we would like you to try. In fact we have released new improved drivers for Node.js, PHP, PDO, Python, Perl, JDBC, ODBC, OLEDB, ADO.NET, and C. So you should definitely try the new, more performant and stable CUBRID 9.1 Database.

If you have any questions, feel free to leave your comment below.

CUBRID SHARD talk at 2013 Percona MySQL Conference. Don't miss!

2013-03-15T03:42:45-07:00

I would like to announce that on April 24, 2013, six weeks from now, we will talk at Percona MySQL Conference & Expo in Santa Clara, CA. The topic of the presentation is Easy MySQL Database Sharding with CUBRID SHARD. The presentation will be @ 3:30 PM in Ballroom A. Come and join us!

Abstract

If you ask companies who operate mission-critical services, they will tell:

that a relational database system is still the best choice for mission-critical data;
that service availability is more important than performance;
that high performance is good, but predictable performance is the king.

This is a fact, and we know it. At NHN we have over 30,000 Web servers that operate over 150 large scale Web and mobile services. At such scale we must know what scales, how to provide high-availability and operate at predictable speed.

At Percona Live MySQL Conference 2013 I will talk about CUBRID SHARD, a universal database sharding solution for CUBRID, MySQL, and Oracle. CUBRID SHARD can be used with a heterogeneous database backend, i.e. some shards can be stored in CUBRID, some in MySQL or even Oracle. At NHN we deploy various combinations: MySQL only, MySQL + Oracle, MySQL + CUBRID, CUBRID only, and Oracle only. I will explain how DBAs can easily configure it, and how we have implemented this feature.

CUBRID SHARD allows to store unlimited number of database shards and distribute data based on modulo, DATETIME, or hash/range calculations. The developers can even feed in their own library to calculate the SHARD_ID using some complicated custom algorithm. At the session I will show how easy it is to setup all this. No need for a third-party management tool. With CUBRID SHARD application developers do not need to modify the application logic to provide data sharding. This is DBAs job as all this is handled by the database system automatically.

CUBRID SHARD is designed to be very efficient. It provides built-in (*) distributed load balancing and (*) connection and statement pooling. At the conference I will present several cases where CUBRID SHARD is deployed as a shard manager and a connection manager, or where it's used as a way for seamless data migration between different systems.

Who should come to the session?

If you run a service which spends money on a database solution, on tools you need to shard databases or manage connections, you should come and learn how CUBRID SHARD can provide your applications native scale-out through single database view.

If you would like to learn more about CUBRID Database Sharding, see our Database Sharding the Right Way: Easy, Reliable, and Open source I have presented at 2012 HighLoad++ Conference. More about CUBRID you can find at Important Facts to Know about CUBRID.

If you have questions, feel free to leave your comment below.

A free CUBRID Cloud Database Service for our dear users

2013-03-11T00:10:26-07:00

Hello reader,

What do you know about CUBRID Database? Let this be your first introduction to CUBRID. Today I would like to tell a story about what we do at CUBRID in order to improve the experience our users have when they get started with CUBRID Database.

So far we have published numerous installation instructions and short HOWTO tutorials which help our users to quickly get started with installing and configuring CUBRID Database. These include instructions for apt-get and yum package managers. Once their server is up and running, users can continue their learning experience with more tutorials.

To improve user experience, couple months ago we have written multiple Vagrant and Chef Cookbook tutorials which provide easy step by step instructions on how to create a clean virtual machine image for VirtualBox and install CUBRID and any other necessary software on a new Linux operating system in a matter of minutes.

Vagrant in combination with Chef cookbooks is a great tool and time saver for developers, especially for testers and those users who would like to just try a new software in a VM without polutting their host machine. You can fire up a single command like vagrant up, and Vagrant will built up a new VM machine with all the software you need preinstalled and configured for you. How cool is that! In fact, I use Vagrant and Chef on every day basis to reproduce issues users have reported on CUBRID forums, or to quickly start hacking some new features for another project.

While Vagrant is a great tool for local development, Knife Solo is the guy you need for remote server provisioning. Just like Vagrant, Knife Solo prepares and cooks the Chef cookbooks on a remote server, be it a VM on your local machine or a remote Amazon EC2 server. And we have written a Knife Solo tutorial which introduces this tool and shows how to install CUBRID Database, its tools and drivers on a remote machine.

For those users who wish to directly download a virtual machine with a preconfigured CUBRID Database, we have built and uploaded CentOS and Ubuntu VirtualBox images with different versions of CUBRID Server.

To further improve the user experience, today I am immensely happy to announce the CUBRID Cloud Database Service at http://cloud.cubrid.org. We have come from installing CUBRID manually on a user machine to installing automatically on a VM or a remote server, to not having to install CUBRID at all. Now you can request a connection information to a remote CUBRID cloud database for free as soon as you need. All you need is a valid email address.

Figure 1: CUBRID Cloud Database Service front page.

Once requested, you will receive a confirmation email address. You confirm your email, and we will start cooking your very own CUBRID cloud database. In a minute or so you will receive a second email with the database credentials. This will include the remote database host IP address, the port number, your database name, a username, and a password. We will also include a short getting started tutorial for you to take off quickly.

We have built this CUBRID Cloud Service for educational purpose. We want our users be able to get their hands on a CUBRID database as soon as they need. You can use this cloud database for testing, for learning CUBRID, or for building non-critical demo applications.

To discuss CUBRID Cloud Service, we have created a dedicated forum thread. You are welcome to join us! If you want to chat with our engineers, head to #cubrid freenode chat room. We will be glad to see you there!

So, go ahead and creat your first CUBRID cloud database!

Overview of Bootstrap from Twitter

2012-07-30T23:10:56-07:00

Update: Bootstrap 2.1.0 has been released yesterday on August 20, 2012. Check out their official blog for release notes.

Bootstrap is a front-end toolkit open sourced by Twitter which provides flexible HTML, CSS, and JavaScript-based design templates, a variety of UI components and interactions for building a beautiful website. In this article I will provide an overview of Bootstrap, describe its characteristics, and will show how to start building websites with it.

Bootstrap from Twitter

Developing a Web UI looks easy; however, so many developers encounter a number of difficulties when trying to develop it. Different from general UI development which requires a single client environment, Web UI development requires Cross Browsing that satisfies compatibility among a variety of browser environments.

There are five browsers that are mainly used in PCs - Internet Explorer, Firefox, Chrome, Safari and Opera - and additional work is necessary depending on the version and the operating system. In addition, we need to consider the current mobile environment that has a variety of screen sizes, such as smartphone and tablet PC, and the browsers specialized for each device type.

These various and complex browser environments make it difficult to process Cross Browsing for the small- and mid-size Web sites. Bootstrap is the front-end toolkit from Twitter which tries to clear this difficulty and provide a start point that allows developers to quickly and easily implement a flexible web site.

Bootstrap is an open source front-end toolkit developed by Mark Otto, a Twitter UI designer, and Jacob Thornton, a Twitter developer. As its meaning "to start up a system by turning on the power of a computer or pressing a reset key," Bootstrap is a starting point for building websites which provides flexible HTML, CSS and JavaScript-based design templates, variety of frequently-used UI components and interactions.

Grid System

Grid is a basic element for layout design. Generally, the screen is split by using the <div> tag and CSS to place elements on the webpage layout. Popular CSS frameworks that standardize screen split and provide a grid layout system are Blueprint, 960 grid and YUI CSS grids.

Bootstrap also provides its own grid layout system. The grid system of Bootstrap consists of 12-column grid and there are two versions: default and fluid. In addition, it has four design patterns for a variety of screen sizes. The patterns are linked to the responsive design to be discussed later.

Default Grid System

The default grid system utilizes a 12-column grid that renders out at the width of 940 pixels.

A row is expressed by using row and a column is expressed by using span*. The span* class can be span1-12.

The space between columns is 20 pixels and the width of the span1 class is 60 pixels. As the number of the span* class increases by one, the width increases by 80 pixels, so the width of the span12 class is 940 pixels.

Figure 1. 12-column Grid System.

The markup for the third row in Figure 1 is as shown below, expressed as 300 pixels + 20 pixels + 620 pixels (total 940 pixels) on the screen.

<div class="row">
    <div class="span4">…</div>
    <div class="span8">…</div>
</div>

Fluid Grid System

The fluid grid system uses percentage for column widths instead of fixed pixels. To use the fluid grid system, make any row fluid simply by changing to row-fluid.

<div class="row-fluid">
    <div class="span4">…</div>
    <div class="span8">…</div>
</div>

Offsetting Columns

Add the offset* class to a column to offset the column to the right. However, offsetting columns is available for the default grid system only.

Figure 2. Offsetting Columns

In Figure 2, the markup of the first row can be written as follows. The second row <div> will be offset to right as much as four-grid width (offset4).

<div class="row">
    <div class="span4">…</div>
    <div class="span4 offset4">…</div>
</div>

Nesting Columns

To nest your content as shown in Figure 3, just add a new <div> tag containing row and set of span* columns within an existing .span* column.

Figure 3. Nesting Columns

For the default grid, the sum of nested rows should be the number of columns of its parents.

<div class="row">
    <div class="span6">
    Level 1 of column
        <div class="row">
            <div class="span3">Level 2</div>
            <div class="span3">Level 2</div>
        </div>
    </div>
</div>

For nesting with a fluid grid, the sum of nested rows should be 12, not the parent column.

<div class="row-fluid">
    <div class="span6">
    Level 1 of column
        <div class="row-fluid">
            <div class="span6">Level 2</div>
            <div class="span6">Level 2</div>
        </div>
    </div>
</div>

Responsive Design

Responsive Design requires to design the layout of a Web page using the CSS3 media queries in a way so that the layout automatically adapts to the screen size and resolution of various devices such as smartphones, PC, and tablet PC.

Bootstrap supports a flexible screen layout with its grid system and Responsive Design. The size of the grid is changed according to the screen size as shown in the following table.

Table 1. Changed Grid Size by Device.

Label	Layout width	Column width	Gutter width
Smartphones	480 pixels and lower	Fluid columns, no fixed widths
Portrait tablet PCs	767 pixels and lower	Fluid columns, no fixed widths
Landscape tablet PCs	768 pixels and higher	42 pixels	20 pixels
Default	980 pixels and higher	60 pixels	20 pixels
Wide-screen monitor	1200 pixels and higher	70 pixels	30 pixels

The following figure shows the changed layouts with responsive design according to the screen size.

Figure 4. Responsive Design by Screen Size.

The CSS that makes media queries responsive to the device size is as follows.

// Landscape phones and down
@media (max-width: 480px) { ... }

// Landscape phone to portrait tablet
@media (max-width: 767px) { ... }

// Portrait tablet to landscape and desktop
@media (min-width: 768px) and (max-width: 979px) { ... }

// Large desktop
@media (min-width: 1200px) { ... }

In addition, this CSS includes a selector as a utility that displays the optimized contents on the device.

Table 2. Responsive CSS Classes by Device.

Class	Phones 480 pixels and lower	Tablet PCs 767 pixels and lower	Desktops 768 pixels and higher
.visible-phone	Visible	Hidden	Hidden
.visible-tablet	Hidden	Visible	Hidden
.visible-desktop	Hidden	Hidden	Visible
.hidden-phone	Hidden	Visible	Visible
.hidden-tablet	Visible	Hidden	Visible
.hidden-desktop	Visible	Visible	Hidden

CSS with a Clean and Organized Look

Bootstrap provides buttons and a variety of styles offering a clean and organized look with the default CSS.

The following table shows the default button designs of Bootstrap.

Table 3. CSS Class by Button Type.

Button	class=""	Description
	btn	Gray default button with gradation
	btn btn-primary	Button indicating something that needs to be visually highlighted or specialized
	btn btn-info	Button that can replace the default button
	btn btn-success	Button indicating a successful or positive behavior
	btn btn-warning	Warning button
	btn btn-danger	Button indicating danger or a negative effect
	btn btn-inverse	Dark gray button that can be used generally without any special meaning

The following table shows the default form designs of Bootstrap.

Figure 5. CSS Class for Form.

Initially, Bootstrap did not support Internet Explorer. However, it has been changed to be compatible with Internet Explorer 7 and above. Therefore, form design for Internet Explorer is not identical according to the version. A minor difference may be seen according to CSS support, such as gradation and rounded border. However, the overall layout and colors remain the same.

Figure 6. Comparison of Examples of Expressing Form Design in Internet Explorer.

CSS Reset uses Nicolas Gallagher's normalize.css and UI icon image uses a set of icons provided by Glyphicons. The icon set is a single image in which the CSS Sprite technique is used.

Figure 7. Glyphicons Icon Set.

The following is an example of a UI component where the icon set is applied.

Figure 8. UI Component with Icon Set.

Bootstrap CSS is written in LESS so it is easy to modify and reproduce a theme. This will be discussed later.

UI Component and jQuery Plug-in

There are UI patterns that are frequently used for producing webpages. UI components refers to a bundle of markup, CSS and JavaScript for easy pattern reuse.

Bootstrap has a default CSS that provides a variety of easy-to-use UI components. Some components that require dynamic interaction can be controlled by jQuery plug-in type JavaScript.

A component that uses the default CSS has the button group, dropdown list, navigation, tab, Move Page button, thumbnail layout, warning display and close icon. For a more detailed component list and examples, see http://twitter.github.com/bootstrap/components.html.

Components that control the interaction by using the jQuery plug-in are modal layer, dropdown, toggle tap and tooltip. For a more detailed component list and examples, see http://twitter.github.com/bootstrap/javascript.html.

Note:

NHN has open sourced UIO Factory, which provides developers a static UI pattern set, as well as Jindo framework-based Jindo Component which provides components for dynamic UI.

LESS

Different from general programming language, CSS has no inheritance or variable. Therefore, developers should modify the code when they need to modify just the screen height. It is quite cumbersome to write and manage CSS. To solve this problem, the CSS preprocessors have been developed.

CSS preprocessors have self-defined syntax by expanding the existing CSS syntax. The expanded syntax allows for describing variables, mixin, inheritance, operation, and function, and creating the final CSS that is compatible with the browser by compiling the code written with the expanded CSS. Some people have been inconvenienced by not having dynamic and object-oriented concept from CSS3.

The most popular CSS preprocessors are LESS, Sass and Stylus, and the syntax and functions are similar. Bootstrap uses LESS as its CSS preprocessor.

Even though it has significant benefits, most domestic sites have not applied the CSS preprocessors so far. As well as the inconvenience of compiling, in most cases, homepage design has been outsourced just for one-time so the existing CSS has not been reused and there are not many professionals who can create well-structured markup and CSS. In foreign countries, after releasing Bootstrap on Github, an active movement for applying CSS preprocessors is made and many developers started to create projects for which CSS preprocessors are changed to Sass or Stylus instead of LESS.

Let's discuss LESS in detail.

LESS Syntax

LESS syntax uses a syntax that expands the functions only on the standard CSS syntax.

Variable Declaration

To declare a variable, use the @ symbol.

@mainColor: #0982c1;
@siteWidth: 1024px;
@borderStyle: dotted;

body {
  color: @mainColor;
  border: 1px @borderStyle @mainColor;
  max-width: @siteWidth;
}

Mixin Function

You can create a mixin function similar to the class selector as shown below.

/* LESS mixin error with (optional) argument @borderWidth which defaults to 2px if not specified */
.error(@borderWidth: 2px) {
border: @borderWidth solid #F00;
color: #F00;
}

.generic-error {
padding: 20px;
margin: 4px;
.error(); /* Applies styles from mixin error */
}
.login-error {
left: 12px;
position: absolute;
top: 20px;
.error(5px); /* Applies styles from mixin error with argument @borderWidth equal to 5px */
}

Inheritance

LESS defines the mixin function as a class selector. Therefore, the general class selector can be used as a mixin function so that inheritance can be expressed as shown below.

.block {
margin: 10px 5px;
padding: 2px;
}

p {
.block; /* Inherit styles from '.block' */
border: 1px solid #EEE;
}
ul, ol {
.block; /* Inherit styles from '.block' */
color: #333;
text-transform: uppercase;
}

Operator

A simple operator can be used for the operable attributes as shown below.

body {
margin: (14px/2);
top: 50px + 100px;
right: 100px - 50px;
left: 10 * 10;
}

Nested Selector

Nested selector is expressed as shown below.

section {
margin: 10px;

nav {
height: 25px;

a {
color: #0982C1;

&:hover {
text-decoration: underline;
}
}
}
}

When compiling a nested selector, the following CSS is made.

section {
margin: 10px;
}
section nav {
height: 25px;
}
section nav a {
color: #0982C1;
}
section nav a:hover {
text-decoration: underline;
}

Bootstrap LESS

There are a lot of pre-defined LESS variables and mixins related to the layouts and components. By modifying the predefined variables and mixins, it is simple to change and create a variety of themes.

The following is an example of LESS variables and values defined by Bootstrap by default. For more details, see http://twitter.github.com/bootstrap/less.html#variables.

Figure 9. A Variety of LESS Variables of Bootstrap.

The following is an example defined in Bootstrap by default. For more details, see http://twitter.github.com/bootstrap/less.html#mixins.

Figure 10. A Variety of LESS Mixins of Bootstrap.

LESS Compiling

LESS compiling is classified into client side and server side.

For client side compiling, the less.js file is saved in the Web server and linked as shown below.

<link rel="stylesheet/less" type="texr/css" href="bootstrap.less">
<script type="text/javascript" src="less.js"></script>

For server side compiling, download the compiler from the node.js package manager and then install and use it.

$ npm install –g less
$ lessc bootstrap.less > bootstrap.css

Customizing

The Bootstrap homepage allows developers to customize a set of CSS and jQuery plug-in by changing the variables applied to LESS and selecting the desired CSS and jQuery plug-in. You can select the components and jQuery plug-ins and download the desired CSS and jQuery JavaScript.

Figure 11. Bootstrap Customizing Page.

Even though there is no LESS compiler, you can change the values applied to the LESS variables as desired on the Customizing Page.

Figure 12. LESS Variable Value Customizing.

Select the components and jQuery plug-ins, modify the LESS variable values, and then click the Download button to download the .zip file compressing the selected CSS and JavaScript.

Figure 13. Customizing Result Download.

As well as the customizing page, you can use a variety of themes from http://bootswatch.com/, http://wrapbootstrap.com/, and http://stylebootstrap.info/.

Conclusion

If we take a deeper look into Bootstrap from Twitter, we can see very little new technology. There seems to be nothing special. Nevertheless, Bootstrap is very popular which is proved by the number of watches and forks of Bootstrap project on Github. Also there are a variety of Bootstrap-related sites. There are a large number of well designed components accompanied by a set of jQuery plugins. It is a great framework if you want to build beautiful Web sites quickly.

By Kyungil Park, Senior Software Engineer, NHN Platform Ajax Team, NHN Corporation.

About the author:

I develop the JavaScript CI server and a variety of tools in NHN Platform Ajax Team. I like everything related to development. Dreaming of being a gray-haired developer...

References

Bootstrap, from Twitter: http://twitter.github.com/bootstrap/
Stepping Out Width Bootstrap from Twitter: http://webdesign.tutsplus.com/tutorials/htmlcss-tutorials/stepping-out-with-bootstrap-from-twitter/
CSS3 Media Queries: http://www.w3.org/TR/css3-mediaqueries/
Sass vs. LESS vs. Stylus: http://net.tutsplus.com/tutorials/html-css-techniques/sass-vs-less-vs-stylus-a-preprocessor-shootout/
20 Resources for Bootstrap Lovers: http://designshack.net/articles/css/20-awesome-resources-for-twitter-bootstrap-lovers/

What is SPDY? Deployment Recommendations

2012-11-25T16:22:07-08:00

In the middle of this year a news report announced that Facebook is planning to support Google's SPDY protocol in large scale, and that they are already implementing SPDY/v2. Here is the official response from Facebook that I have found on this topic. Among various efforts devised and suggested by Google to make the Web faster, I think SPDY will be the one to become a new industry standard, and it will be included in HTTP/2.0.

As an acronym of SPeeDy, SPDY is a new protocol Google has suggested as a part of its efforts to "make the Web faster." It was suggested as a protocol to use the present and future Internet environment more efficiently by addressing the disadvantages of HTTP devised in the early Internet environment.

In this article, I will provide a brief introduction to the features and merits of SPDY. I will explain about the state of SPDY support, and what to do and what to consider when it is introduced.

When was the latest version of HTTP released?

HTTP version 0.9 was first announced in 1991, and HTTP 1.0 and 1.1 were released in 1996 and 1999, respectively, and since then nothing has been changed in HTTP for the last 10 years. These days, however, a webpage has a size 20 times bigger with 20 times more HTTP requests than a webpage in the 1990s. The Table 1 below shows the data quoted from Google I/O 2012.

**Table 1: Comparison of Mean Webpage Size in 2010 and 2012.**
	Mean page size	Mean # of requests per page	Mean No. of domains
2010 Nov. 15	702 KB	74	10
2012 May 5	1059 KB	84	12

The capacity of the Yahoo! main page in 1996 was 34 KB. That is only 1/30 of the mean webpage capacity in 2012. There is a significant gap even between 2010 and 2012, let alone between the 1990s and 2012. This is because the mean page size and number of requests are ever increasing as User UX is becoming more and more sophisticated along with the dissemination of high-speed Internet.

The characteristics of today's webpage have changed from that in the past as follows:

Consists of much more resources.
Uses multiple domains.
Operates more dynamically.
Emphasizes security more.

Considering how today's web environment is different from the past, Google has announced the SPDY protocol, which complemented the disadvantages of HTTP. SPDY focuses especially on resolving the problem of load latency.

Features of SPDY

The Figure 1 below shows the layers of SPDY compared to the traditional TCP/IP layer model.

Figure 1: HTTP vs. SPDY.

The features of SPDY can be summarized as follows:

Always operates on Transport Layer Security (TLS).
- Transport Layer Security (TLS) is the next version of Secure Sockets Layer (SSL). TLS and SSL are sometimes used to refer to the same protocol because they are the name for two different versions of the same protocol, and this also applies to this article.
Therefore, SPDY applies only to websites written with HTTPS.

HTTP Header compression

As HTTP headers have many redundant contents whenever a request is made, you can improve the performance significantly just by compressing headers. According to a report by Google, it is possible to reduce the size by 10-35% by compressing HTTP headers even in the initial request, and reduce the size of headers by 80-97% when requests are made several times (long-lived connection). Also, when the upload bandwidth is relatively small, as in mobile devices, this HTTP header compression is more useful. These days, as the HTTP header is 2 KB on average and is growing bigger, the merit of compressing HTTP headers is expected to grow in the future.

Binary protocol

As it uses binary framing rather than text-based framing, it provides faster parsing and is less sensitive to errors.

Multiplexing

SPDY handles multiple independent streams in a single connection concurrently. For this reason, unlike HTTP which handles one request at a time in a single connection with response to requests made consecutively, SPDY handles multiple requests and responses concurrently with a small number of connections. In addition, unlike HTTP pipelining in which if one response is delayed, the others are all delayed, SPDY handles each request and response independently as it uses a FIFO queue.

Full-duplex interleaving and stream prioritization

As SPDY allows interleaving in which one stream in a process can be interleaved with another and stream prioritization, data of higher priority can jump into the process of transportation of data of lower priority and can be transported earlier.

Server push

Servers can push content without client requests unlike Comet and Long-polling. Unlike methods such as inlining, SPDY supports resource caching and uses the same bandwidth as that of inlining or a smaller bandwidth. When implementing server push, however, you need to implement additional Web server application logic.

No need to re-write a website

Except for some features that require additional implementation, such as server push, you don't need to change a website itself to apply SPDY. However, the browser and the server should support SPDY. SPDY can be applied completely and transparently to browser users. In other words, there is no protocol scheme like "spdy://." The browser also displays nothing with regard to the use of the SPDY protocol.

Based on such characteristics of SPDY, the following table shows the difference between HTTP and SPDY.

**Table 2: HTTP/1.1 vs. SPDY.**
	HTTP	SPDY
Secure	Not Default	Default
Header Compression	No	Yes
Multiplexing	No	Yes
Full-Duplex	No	Yes
Prioritization	No, (instead, a browser employs heuristics.)	Yes
Server push	No	Yes
DNS Lookup	More	Less
Connections	More	Less

Therefore, it can be said that SPDY is a protocol designed to use TCP connection more efficiently by improving the data transfer format and connection management of HTTP. As a result of such efforts, in a test with the top 25 websites, SPDY worked 39-55% faster compared to HTTP + SSL.

Why does SPDY need TLS?

Why does SPDY use TLS even though using TLS causes latency due to encryption/decryption? Google's SPDY Whitepaper states the following answer for this question:

"In the long term, the importance of web security will be emphasized more and more, and thus we want to get a better security in the future by specifying TLS as a sub protocol of SPDY. We need TLS for compatibility with the current network infrastructure. In other words, we need it to prevent any compatibility issue with the communication going through the existing proxy.”

Despite this reason, when looking into the implementation of the actual SPDY, you can see that SPDY depends much on TLS's Next Protocol Negotiation (NPN) extension mechanism. This TLS NPN extension mechanism determines whether a request coming from Port 443 is SPDY or not and identifies the version of SPDY used by the request, to decide whether to use SPDY to handle the next communication. Without TLS NPN, you will need to get additional RTT to use SPDY.

Efforts for Standardization

SPDY is being developed as an open networking protocol and has been suggested to IETF as a method of HTTP/2.0. SPDY is a sub-project of the Google Chromium project, and thus Chromium client implementations and server tools are all being developed with open sources.

Future of SPDY

Most recently SPDY Draft 3 was released, and SPDY Draft 4 is under development. The features likely to be added in Draft 4 are as follows:

Name resolution push
Certificate data push
Explicit proxy support

The final goal of SPDY is to provide a page within "a single connection setup time + bytes/bandwidth time".

Browsers, Servers, Libraries and Web Services Supporting SPDY

Currently a variety of browsers and servers support SPDY, and Google, who originally suggested SPDY, already provides almost all of its services with SPDY. The browsers, servers, libraries and services supporting SPDY are as follows.

Browsers Supporting SPDY

As of July 2012, the following is a list of browsers which support the SPDY protocol.

Google Chrome/Chromium

Chrome and Chromium have supported SPDY from their initial version. If you enter the following URI, you can inspect SPDY sessions in Chrome/Chromium: chrome://net-internals/#events&q=type:SPDY_SESSION%20is:active. For example, if you visit www.gmail.com, in the chrome:// tab you can see multiple SPDY sessions being created. Android mobile Chrome also supports SPDY.

Firefox 11 and later versions

While added in version 11, SPDY was not enabled by default until Firefox version 13. If you enter about:config in FF, which is the URI for Firefox settings, and see network.http.spdy.enabled, you can check whether support for SPDY has been enabled or not. Android mobile Firefox 14 also supports SPDY.

Amazon Silk

The Silk browser equipped in Kindle Fire, an Android-based e-book reader from Amazon, also supports SPDY. It communicates with Amazon EC2 service by using SPDY.

Default browser of Android 3.0 and higher

The default browser of Android 3.0 (Honeycomb) and 4.0 (Ice Cream Sandwich) supports SPDY.

For more information about which browsers support SPDY, see http://caniuse.com/spdy.

Servers and Libraries Supporting SPDY

Support for SPDY is being vitalized mainly by major web servers and application servers, and a variety of libraries that implement SPDY are also being developed.

Nginx

Nginx released a beta version of SPDY module on June 15, 2012, and has continuously provided patches. See SPDY: 146% faster Slideshare presentation by Nginx team to learn more about it.

Jetty

Jetty also provides the SPDY module.

Apache

The SPDY module for Apache 2.2 is also being developed.

Libraries

In addition, SPDY implementation structures for Python, Ruby and node.js servers have already been developed or are currently being developed. There are a variety of versions for SPDY C library, including libspdy, spindly and spdylay. A library to use SPDY on iOS is also being developed.

Netty

Netty began to provide SPDY package from its 3.3.1 version released in 2012.

Tomcat

SPDY support in Tomcat is currently under development and it should come in Tomcat version 8.

Services Using SPDY

As mentioned earlier, Google has already converted almost all of its services, including search, Gmail and Google plus, into HTTPS, and provides them through SPDY. In addition, when Google App Engine uses HTTPS, it also supports SPDY. Twitter also uses SPDY when providing service via HTTPS.

However, among numerous Web sites on the Internet, only a few websites use SPDY. According to a survey by Netcraft conducted in May 2012, of a total of 660 million websites only 339 websites are currently using SPDY. In other words, except for Google and Twitter, there is hardly any major website using SPDY.

When SPDY is Not Very Efficient

SPDY is not always fast. Sometimes you may not get any performance improvement with SPDY. Such situations are as follows.

When using only HTTP

As SPDY always requires SSL, you need additional SSL handshake time. Therefore, when you convert an HTTP site into HTTPS to support SPDY, you may not obtain distinct performance improvement due to SSL handshake.

When there are too many domains

SPDY operates by domain. This means that it requires as many connections as the number of domains, and that request multiplexing is available only in a single domain. Moreover, as it is difficult to make all domains support SPDY, you may not get the merits of SPDY when there are too many domains. Especially when a CDN does not support SPDY, you may not expect the performance improvement with SPDY.

When HTTP is not the bottleneck

For most pages, HTTP is not the bottleneck. For example, in cases when a resource can be downloaded only after another resource is downloaded, SPDY will not be that effective.

When Round-Trip-Time (RTT) is low

SPDY is more efficient when RTT is high. When RTT is very low, for example, in communications between servers within IDC, SPDY has few merits.

When resource in a page is very small

For pages with six or fewer resources, SPDY has few merits because the value of reusing connection is not significant.

Things to Do to Introduce SPDY

When you introduce SPDY, you need to carry out the following tasks to apply it most efficiently.

Application Level

Use only one connection

For better SPDY performance and more efficient use of Internet resources, you need to use as few connections as possible. If you use a small number of connections when using SPDY, you can see the benefits such as putting data into packets in a better way, getting better header compression efficiency, reducing the frequency of checking the status of connection and reducing the frequency of handshake. Also, in terms of Internet resources, with a small number of connections, you can also have a more efficient TCP and reduce Bufferbloat.

Bufferbloat is a phenomenon in which excess buffering of packets at a router or a switch causes high latency and reduced throughput. With the idea of avoiding packet discards as much as possible and low memory prices, the buffer size of a router or a switch is continuously increasing. The packets that should have been discarded earlier could survive longer and as a result these packets hinder TCP congestion avoidance algorithm, deteriorating the overall network performance.

Avoid domain sharding

Domain sharding is a kind of expedient used to avoid the restriction on the number of concurrent downloads (in general, six downloads per hostname in modern browsers) in web applications. If you use SPDY and comply with "use a single connection" recommendation, you don't need to use domain sharding. To make it worse, domain sharding causes additional DNS queries and makes applications more complex.

Use server push instead of inlining

Inlining stylesheets or scripts is often used to reduce the number of HTTP requests, thus RTT, in web applications. However, inlining makes Web pages less cacheable and increases the size of webpages due to base64 encoding. If you use the SPDY server push feature used to push content, you can avoid these problems.

Use request prioritization

You can enable the client to inform the server of the relative priority of resources by using the request prioritization feature of SPDY. The simple common heuristic prioritization could be html > js, css > *.

Choose the proper size of a SPDY frame

Although SPDY spec allows large frames, sometimes a small frame is more desirable. This is because a small frame allows interleaving to work better.

SSL Level

Use a smaller, full certificate chain

The size of a certificate chain makes a huge influence on the performance of the initialization of a connection. If there are more certificates in a certificate chain, it will take more time to verify the validity of certificates, and more space will be occupied in initcwnd.

initcwnd: the initial value used in TCP initial congestion window and TCP congestion control algorithm. The congestion window is a sender-side window to control the size according to TCP congestion control algorithm, which is different from the TCP window, which is a receiver-side limit on the size.

In addition, if a server does not provide a full certificate chain, the client will use additional RTT to get certificates in the middle. Therefore, if a large, incomplete certificate chain is used, it will take longer time for an application to use the connection.

Use a wildcard certificate (e.g., *.naver.com) if possible

If you use wildcard certificates, you can reduce the number of connections and use the connection sharing of SPDY. As a wildcard certificate is provided by a certification institute, however, you may need to discuss with the institute and pay extra costs.

Do not set the size of SSL write buffer too large

If SSL write buffer is too large, TLS application record will be put on multiple packets. As an application can process only after the entire TLS application record is completed, the record put on multiple packets will cause additional latency. Google servers use 2 KB buffers.

TCP Level

Set the initcwnd of the server to at least 10

Initcwnd is the main bottleneck that affects the initial loading time of a page. If you use only HTTP, you can avoid this problem by attaining the initial congestion window size of n × initcwnd by opening multiple connections concurrently. As a single connection is advantageous in SPDY, however, it is better to set initcwnd to a large value from the beginning. This value in old Linux kernels is fixed to 2-3, and the method to adjust this value is not provided. As this value was determined according to the reliability and bandwidth of TCP network when it was first considered, it is not suitable to today's TCP network with higher stability and bandwidth. The method to adjust this value was added in Linux Kernel 3.0, and the latest Linux kernels already use the default value of 10 or higher.

Disable tcp_slow_start_after_idle

The tcp_slow_start_after_idle on Linux is set to 1. This causes a congestion window to return to the size of initcwnd when the connection goes idle, and makes TCP Slow Start restart.

TCP Slow Start is an algorithm that works by sending packets to the congestion window of the initcwnd size and increasing the TCP congestion window up to the maximum value allowed by the network or up to the TCP window of the receiver side. If the initcwnd value is small, it takes more round-trips until the window reaches the maximum size allowed by the network, and as a result, the initial page loading time will increase.

As this will eliminate the advantage of a single connection of SPDY, this setting should be disabled. You can change the setting by using the sysctl command. It is also advantageous to disable this setting when using HTTP keepalive.

If SPDY is Really Introduced …

In conclusion, to actually introduce SPDY, you need to consider a variety of matters and modify applications and servers. You cannot ignore the costs of introducing the protocol, either. For a Web application written for Tomcat, which does not yet support SPDY, you should consider the cost required to change the Web application server, as well as the cost required to implement the server push functionality.

Except for the costs required to introduce SPDY, what should we take into account first when we introduce SPDY in a real service? I chose three matters to consider.

The service should be the one that already uses HTTPS

You cannot get many advantages when introducing SPDY for services using only HTTP. You should pay the costs for introducing SSL as well.

You should be able to change the Linux kernel

Even CentOS 6.3, the latest version released on July 9, 2012, still uses the kernel 2.6.32. Adjustment of initcwnd is supported only from the kernel 3.0, and you need to change the kernel if possible because the performance improvement you can get by adjusting initcwnd is very significant.

Consider the ratio of users of SPDY supported browsers

In Korea, many users still use IE which does not support SPDY. On mobile devices, iOS does not yet support SPDY, while Android 3.0 or higher supports SPDY. Therefore, until there are sufficient users of SPDY-supported browsers, you should carefully compare the advantages you can get from the performance improvement derived from SPDY with the costs required to introduce the protocol.

By Sehoon Park, Senior Engineer at Web Platform Development Lab, NHN Corporation.

Understanding TCP/IP Network Stack & Writing Network Apps

2013-02-25T16:40:05-08:00

We cannot imagine Internet service without TCP/IP. All Internet services we have developed and used at NHN are based on a solid basis, TCP/IP. Understanding how data is transferred via the network will help you to improve performance through tuning, troubleshooting, or introduction to a new technology.

This article will describe the overall operation scheme of the network stack based on data flow and control flow in Linux OS and the hardware layer.

Key Characteristics of TCP/IP

How should I design a network protocol to transmit data quickly while keeping the data order without any data loss? TCP/IP has been designed with this consideration. The following are the key characteristics of TCP/IP required to understand the concept of the stack.

TCP and IP

Technically, since TCP and IP have different layer structures, it would be correct to describe them separately. However, here we will describe them as one.

1. Connection-oriented

First, a connection is made between two endpoints (local and remote) and then data is transferred. Here, the "TCP connection identifier" is a combination of addresses of the two endpoints, having <local IP address, local port number, remote IP address, remote port number> type.

2. Bidirectional Byte Stream

Bidirectional data communication is made by using byte stream.

3. In-order Delivery

A receiver receives data in the order of sending data from a sender. For that, the order of data is required. To mark the order, 32-bit integer data type is used.

4. Reliability through ACK

When a sender did not receive ACK (acknowledgement) from a receiver after sending data to the receiver, the sender TCP re-sends the data to the receiver. Therefore, the sender TCP buffers unacknowledged data from the receiver.

5. Flow Control

A sender sends as much data as a receiver can afford. A receiver sends the maximum number of bytes that it can receive (unused buffer size, receive window) to the sender. The sender sends as much data as the size of bytes that the receiver's receive window allows.

6. Congestion Control

The congestion window is used separately from the receive window to prevent network congestion by limiting the volume of data flowing in the network. Like the receive window, the sender sends as much data as the size of bytes that the receiver's congestion window allows by using a variety of algorithms such as TCP Vegas, Westwood, BIC, and CUBIC. Different from flow control, congestion control is implemented by the sender only.

Data Transmission

As indicated by its name, a network stack has many layers. The following Figure 1 shows the layer types.

Figure 1: Operation Process by Each Layer of TCP/IP Network Stack for Data Transmission.

There are several layers and the layers are briefly classified into three areas:

User area
Kernel area
Device area

Tasks at the user area and the kernel area are performed by the CPU. The user area and the kernel area are called "host" to distinguish them from the device area. Here, the device is the Network Interface Card (NIC) that sends and receives packets. It is a more accurate term than the commonly used "LAN card".

Let's take a look at the user area. First, the application creates data to send (the "User data" box in Figure 1) and then calls the write() system call to send the data. Assume that the socket (fd in Figure 1) has been already created. When the system call is called, the area is switched to the kernel area.

POSIX-series operating systems including Linux and Unix expose the socket to the application by using a file descriptor. In the POSIX-series operating system, the socket is a kind of a file. The file layer executes a simple examination and calls the socket function by using the socket structure connected to the file structure.

The kernel socket has two buffers.

One is the send socket buffer for sending;
And the other is the receive socket buffer for receiving.

When the write system call is called, the data in the user area is copied to the kernel memory and then added to the end of the send socket buffer. This is to send data in order. In the Figure 1, the light-gray box refers to the data in the socket buffer. Then, TCP is called.

There is the TCP Control Block (TCB) structure connected to the socket. The TCB includes data required for processing the TCP connection. Data in the TCB are connection state (LISTEN, ESTABLISHED, TIME_WAIT), receive window, congestion window, sequence number, resending timer, etc.

If the current TCP state allows for data transmission, a new TCP segment (in other words, a packet) is created. If data transmission is impossible due to flow control or such a reason, the system call is ended here and then the mode is returned to the user mode (in other words, the control is passed to the application).

There are two TCP segments as shown in Figure 2:

TCP header;
And payload.

Figure 2: TCP Frame Structure (source).

The payload includes the data saved in the unacknowledged send socket buffer. The maximum length of the payload is the maximum value among the receive window, congestion window, and maximum segment size (MSS).

Then, TCP checksum is computed. In this checksum computation, pseudo header information (IP addresses, segment length, and protocol number) is included. One or more packets can be transmitted according to the TCP state.

In fact, since the current network stack uses the checksum offload, the TCP checksum is computed by NIC, not by the kernel. However, we assume that the kernel computes the TCP checksum for convenience.

The created TCP segment goes down to the IP layer. The IP layer adds an IP header to the TCP segment and performs IP routing. IP routing is a procedure of searching the next hop IP in order to go to the destination IP.

After the IP layer has computed and added the IP header checksum, it sends the data to the Ethernet layer. The Ethernet layer searches for the MAC address of the next hop IP by using the Address Resolution Protocol (ARP). It then adds the Ethernet header to the packet. The host packet is completed by adding the Ethernet header.

After IP routing is performed, the transmit interface (NIC) is known as the result of IP routing. The interface is used for transmitting a packet to the next hop IP and the IP. Therefore, the transmit NIC driver is called.

At this time, if a packet capture program such as tcpdump or Wireshark is running, the kernel copies the packet data onto the memory buffer that the program uses. In that way, the receiving packet is directly captured on the driver. Generally, the traffic shaper function is implemented to run on this layer.

The driver requests packet transmission according to the driver-NIC communication protocol defined by the NIC manufacturer.

After receiving the packet transmission request, the NIC copies the packets from the main memory to its memory and then sends it to the network line. At this time, by complying with the Ethernet standard, it adds the IFG (Inter-Frame Gap), preamble, and CRC to the packet. The IFG and preamble are used to distinguish the start of the packet (as a networking term, framing), and the CRC is used to protect the data (the same purpose as TCP and IP checksum). Packet transmission is started based on the physical speed of the Ethernet and the condition of Ethernet flow control. It is like getting the floor and speaking in a conference room.

When an NIC sends a packet, the NIC generates interrupts on the host CPU. Every interrupt has its own interrupt number and the OS searches an adequate driver to handle the interrupt by using the number. The driver registers a function to handle the interrupt (an interrupt handler) when the driver is started. The OS calls the interrupt handler and then the interrupt handler returns the transmitted packet to the OS.

So far we have discussed the procedure of data transmission through the kernel and the device when an application performs write. However, without a direct write request from the application, the kernel can transmit a packet by directly calling TCP. For example, when an ACK is received and the receive window is expanded, the kernel creates a TCP segment including the data left in the socket buffer and sends the TCP segment to the receiver.

Data Receiving

Now, let's take a look at how data is received. Data receiving is a procedure for how the network stack handles a packet coming in. Figure 3 shows how the network stack handles a packet received.

Figure 3: Operation Process by Each Layer of TCP/IP Network Stack for Handling Data Received.

First, the NIC writes the packet onto its memory. It checks whether the packet is valid by performing the CRC check and then sends the packet to the memory buffer of the host. This buffer is a memory that has already been requested by the driver to the kernel and allocated for receiving packets. After the buffer has been allocated, the driver tells the memory address and size to the NIC. When there is no host memory buffer allocated by the driver even though the NIC receives a packet, the NIC may drop the packet.

After sending the packet to the host memory buffer, the NIC sends an interrupt to the host OS.

Then, the driver checks whether it can handle the new packet or not. So far, the driver-NIC communication protocol defined by the manufacturer is used.

When the driver should send a packet to the upper layer, the packet must be wrapped in a packet structure that the OS uses for the OS to understand the packet. For example, sk_buff of Linux, mbuf of BSD-series kernel, and NET_BUFFER_LIST of Microsoft Windows are the packet structures of the corresponding OS. The driver sends the wrapped packets to the upper layer.

The Ethernet layer checks whether the packet is valid and then de-multiplexes the upper protocol (network protocol). At this time, it uses the ethertype value of the Ethernet header. The IPv4 ethertype value is 0x0800. It removes the Ethernet header and then sends the packet to the IP layer.

The IP layer also checks whether the packet is valid. In other words, it checks the IP header checksum. It logically determines whether it should perform IP routing and make the local system handle the packet, or send the packet to the other system. If the packet must be handled by the local system, the IP layer de-multiplexes the upper protocol (transport protocol) by referring to the proto value of the IP header. The TCP proto value is 6. It removes the IP header and then sends the packet to the TCP layer.

Like the lower layer, the TCP layer checks whether the packet is valid. It also checks the TCP checksum. As mentioned before, since the current network stack uses the checksum offload, the TCP checksum is computed by NIC, not by the kernel.

Then it searches the TCP control block where the packet is connected. At this time, <source IP, source port, target IP, target port> of the packet is used as an identifier. After searching the connection, it performs the protocol to handle the packet. If it has received new data, it adds the data to the receive socket buffer. According to the TCP state, it can send a new TCP packet (for example, an ACK packet). Now TCP/IP receiving packet handling has completed.

The size of the receive socket buffer is the TCP receive window. To a certain point, the TCP throughput increases when the receive window is large. In the past, the socket buffer size had been adjusted on the application or the OS configuration. The latest network stack has a function to adjust the receive socket buffer size, i.e., the receive window, automatically.

When the application calls the read system call, the area is changed to the kernel area and the data in the socket buffer is copied to the memory in the user area. The copied data is removed from the socket buffer. And then the TCP is called. The TCP increases the receive window because there is new space in the socket buffer. And it sends a packet according to the protocol status. If no packet is transferred, the system call is terminated.

Network Stack Development Direction

The functions of network stack layers described so far are the most basic functions. The network stack in the early 1990s had few more functions than the functions described above. However, the latest network stack has many more functions and complexity as the network stack implementation structure gets higher.

The latest network stack is classified by purpose as follows.

Packet Processing Procedure Manipulation

It is a function like Netfilter (firewall, NAT) and traffic control. By inserting the user-controllable code to the basic processing flow, the function can work differently according to the user configuration.

Protocol Performance

It aims to improve the throughput, latency, and stability that the TCP protocol can achieve within the given network environment. Various congestion control algorithms and additional TCP functions such as SACK are the typical examples. The protocol improvement will not be discussed here since it is out of the scope.

Packet Processing Efficiency

The packet processing efficiency aims to improve the maximum number of packets that can be processed per second by reducing the CPU cycle, memory usage, and memory accesses that one system consumes to process packets. There have been several attempts to reduce the latency in the system. The attempts include stack parallel processing, header prediction, zero-copy, single-copy, checksum offload, TSO, LRO, RSS, etc.

Control Flow in the Stack

Now, we will take a more detailed look at the internal flow of the Linux network stack. Like a subsystem which is not a network stack, a network stack basically runs as the event-driven way that reacts when the event occurs. Therefore, there is no separated thread to execute the stack. Figure 1 and Figure 3 showed the simplified diagrams of control flow. Figure 4 below illustrates more exact control flow.

Figure 4: Control Flow in the Stack.

At Flow (1) in Figure 4, an application calls a system call to execute (use) the TCP. For example, calls the read system call and the write system call and then executes TCP. However, there is no packet transmission.

Flow (2) is same as Flow (1) if it requires packet transmission after executing TCP. It creates a packet and sends down the packet to the driver. A queue is in front of the driver. The packet comes into the queue first, and then the queue implementation structure decides the time to send the packet to the driver. This is queue discipline (qdisc) of Linux. The function of Linux traffic control is to manipulate the qdisc. The default qdisc is a simple First-In-First-Out (FIFO) queue. By using another qdisc, operators can achieve various effects such as artificial packet loss, packet delay, transmission rate limit, etc. At Flow (1) and Flow (2), the process thread of the application also executes the driver.

Flow (3) shows the case in which the timer used by the TCP has expired. For example, when the TIME_WAIT timer has expired, the TCP is called to delete the connection.

Like Flow (3), Flow (4) is the case in which the timer used by the TCP has expired and the TCP execution result packet should be transmitted. For example, when the retransmit timer has expired, the packet of which ACK has not been received is transmitted.

Flow (3) and Flow (4) show the procedure of executing the timer softirq that has processed the timer interrupt.

When the NIC driver receives an interrupt, it frees the transmitted packet. In most cases, execution of the driver is terminated here. Flow (5) is the case of packet accumulation in the transmit queue. The driver requests softirq and the softirq handler executes the transmit queue to send the accumulated packet to the driver.

When the NIC driver receives an interrupt and finds a newly received packet, it requests softirq. The softirq that processes the received packet calls the driver and transmits the received packet to the upper layer. In Linux, processing the received packet as shown above is called New API (NAPI). It is similar to polling because the driver does not directly transmit the packet to the upper layer, but the upper layer directly gets the packet. The actual code is called NAPI poll or poll.

Flow (6) shows the case that completes execution of TCP, and Flow (7) shows the case that requires additional packet transmission. All of Flow (5), (6), and (7) are executed by the softirq which has processed the NIC interrupt.

How to Process Interrupt and Received Packet

Interrupt processing is complex; however, you need to understand the performance issue related to processing of packets received. Figure 5 shows the procedure of processing an interrupt.

Figure 5: Processing Interrupt, softirq, and Received Packet.

Assume that the CPU 0 is executing an application program (user program). At this time, the NIC receives a packet and generates an interrupt for the CPU 0. Then the CPU executes the kernel interrupt (called irq) handler. This handler refers to the interrupt number and then calls the driver interrupt handler. The driver frees the packet transmitted and then calls the napi_schedule() function to process the received packet. This function requests the softirq (software interrupt).

After execution of the driver interrupt handler has been terminated, the control is passed to the kernel handler. The kernel handler executes the interrupt handler for the softirq.

After the interrupt context has been executed, the softirq context will be executed. The interrupt context and the softirq context are executed by an identical thread. However, they use different stacks. And, the interrupt context blocks hardware interrupts; however, the softirq context allows for hardware interrupts.

The softirq handler that processes the received packet is the net_rx_action() function. This function calls the poll() function of the driver. The poll() function calls the netif_receive_skb() function and then sends the received packets one by one to the upper layer. After processing the softirq, the application restarts execution from the stopped point in order to request a system call.

Therefore, the CPU that has received the interrupt processes the received packets from the first to the last. In Linux, BSD, and Microsoft Windows, the processing procedure is basically the same on this wise.

When you check the server CPU utilization, sometimes you can check that only one CPU executes the softirq hard among the server CPUs. The phenomenon occurs due to the way of processing received packets explained so far. To solve the problem, multi-queue NIC, RSS, and RPS have been developed.

Data Structure

The followings are some key data structures. Take a look at them and review the code.

sk_buff structure

First, there is the sk_buff structure or skb structure that means a packet. Figure 6 shows some of the sk_buff structure. As the functions have been advanced, they get more complicated. However, the basic functions are very common that anyone can think.

Figure 6: Packet Structure sk_buff.

Including Packet Data and meta data

The structure directly includes the packet data or refers to it by using a pointer. In Figure 6, some of the packets (from Ethernet to buffer) refer to using the data pointer and the additional data (frags) refer to the actual page.

The necessary information such as header and payload length is saved in the meta data area. For example, in Figure 6, the mac_header, the network_header, and the transport_header have the corresponding pointer data that points the starting position of the Ethernet header, IP header and TCP header, respectively. This way makes TCP protocol processing easy.

How to Add or Delete a Header

The header is added or deleted as up and down each layer of the network stack. Pointers are used for more efficient processing. For example, to remove the Ethernet header, just increase the head pointer.

How to Combine and Divide Packet

The linked list is used for efficient execution of tasks such as adding or deleting packet payload data to the socket buffer, or packet chain. The next pointer and the prev pointer are used for this purpose.

Quick Allocation and Free

As a structure is allocated whenever creating a packet, the quick allocator is used. For example, if data is transmitted at the speed of 10-Gigabit Ethernet, more than one million packets per second must be created and deleted.

TCP Control Block

Second, there is a structure that represents the TCP connection. Previously, it was abstractly called a TCP control block. Linux uses tcp_sock for the structure. In Figure 7, you can see the relationship among the file, the socket, and the tcp_sock.

Figure 7: TCP Connection Structure.

When a system call has occurred, it searches the file in the file descriptor used by the application that has called the system call. For the Unix-series OS, the socket, the file and the device for general file system for storage are abstracted to a file. Therefore, the file structure includes the least information. For a socket, a separate socket structure saves the socket-related information and the file refers to the socket as a pointer. The socket refers to the tcp_sock again. The tcp_sock is classified into sock, inet_sock, etc to support various protocols except TCP. It may be considered as a kind of polymorphism.

All status information used by the TCP protocol is saved in the tcp_sock. For example, the sequence number, receive window, congestion control, and retransmit timer are saved in the tcp_sock.

The send socket buffer and the receive socket buffer are the sk_buff lists and they include the tcp_sock. The dst_entry, the IP routing result, is referred to in order to avoid too frequent routing. The dst_entry allows for easy search of the ARP result, i.e., the destination MAC address. The dst_entry is part of the routing table. The structure of the routing table is very complex that it will not be discussed in this document. The NIC to be used for packet transmission is searched by using the dst_entry. The NIC is expressed as the net_device structure.

Therefore, by searching just the file, it is very easy to find all structures (from the file to the driver) required to process the TCP connection with the pointer. The size of the structures is the memory size used by one TCP connection. The memory size is a few KBs (excluding the packet data). As more functions have been added, the memory usage has been gradually increased.

Finally, let's see the TCP connection lookup table. It is a hash table used to search the TCP connection where the received packet belongs. The hash value is calculated by using the input data of <source IP, target IP, source port, target port> of the packet and the Jenkins hash algorithm. It is told that the hash function has been selected by considering defense against attacks to the hash table.

Following Code: How to Transmit Data

We will check the key tasks performed by the stack by following the actual Linux kernel source code. Here, we will observe two paths which are frequently used.

First, this is a path used to transmit data when an application calls the write system call.

SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf, ...)

{

struct file *file;

[...]

file = fget_light(fd, &fput_needed);

[...] ===>

ret = filp->f_op->aio_write(&kiocb, &iov, 1, kiocb.ki_pos);



struct file_operations {

[...]

ssize_t (*aio_read) (struct kiocb *, const struct iovec *, ...)

ssize_t (*aio_write) (struct kiocb *, const struct iovec *, ...)

[...]

};



static const struct file_operations socket_file_ops = {

[...]

.aio_read = sock_aio_read,

.aio_write = sock_aio_write,

[...]

};

When the application calls the write system call, the kernel performs the write() function of the file layer. First, the actual file structure of the file descriptor fd is fetched. And then the aio_write is called. This is the function pointer. In the file structure, you will see the file_operations structure pointer. The structure is generally called function table and includes the function pointers such as aio_read and aio_write. The actual table for the socket is socket_file_ops. The aio_write function used by the socket is sock_aio_write. The function table is used for the purpose that is similar to the Java interface. It is generally used for the kernel to perform code abstraction or refactoring.

static ssize_t sock_aio_write(struct kiocb *iocb, const struct iovec *iov, ..)

{

[...]

struct socket *sock = file->private_data;

[...] ===>

return sock->ops->sendmsg(iocb, sock, msg, size);



struct socket {

[...]

struct file *file;

struct sock *sk;

const struct proto_ops *ops;

};



const struct proto_ops inet_stream_ops = {

.family = PF_INET,

[...]

.connect = inet_stream_connect,

.accept = inet_accept,

.listen = inet_listen, .sendmsg = tcp_sendmsg,

.recvmsg = inet_recvmsg,

[...]

};



struct proto_ops {

[...]

int (*connect) (struct socket *sock, ...)

int (*accept) (struct socket *sock, ...)

int (*listen) (struct socket *sock, int len);

int (*sendmsg) (struct kiocb *iocb, struct socket *sock, ...)

int (*recvmsg) (struct kiocb *iocb, struct socket *sock, ...)

[...]

};

The sock_aio_write() function gets the socket structure from the file and then calls sendmsg. It is also the function pointer. The socket structure includes the proto_ops function table. The proto_ops implemented by the IPv4 TCP is inet_stream_ops and the sendmsg is implemented by tcp_sendmsg.

int tcp_sendmsg(struct kiocb *iocb, struct socket *sock,

struct msghdr *msg, size_t size)

{

struct sock *sk = sock->sk;

struct iovec *iov;

struct tcp_sock *tp = tcp_sk(sk);

struct sk_buff *skb;

[...]

mss_now = tcp_send_mss(sk, &size_goal, flags);



/* Ok commence sending. */

iovlen = msg->msg_iovlen;

iov = msg->msg_iov;

copied = 0;

[...]

while (--iovlen >= 0) {

int seglen = iov->iov_len;

unsigned char __user *from = iov->iov_base;



iov++;

while (seglen > 0) {

int copy = 0;

int max = size_goal;

[...]

skb = sk_stream_alloc_skb(sk,

select_size(sk, sg),

sk->sk_allocation);

if (!skb)

goto wait_for_memory;

/*

* Check whether we can use HW checksum.

*/

if (sk->sk_route_caps & NETIF_F_ALL_CSUM)

skb->ip_summed = CHECKSUM_PARTIAL;

[...]

skb_entail(sk, skb);

[...]

/* Where to copy to? */

if (skb_tailroom(skb) > 0) {

/* We have some space in skb head. Superb! */

if (copy > skb_tailroom(skb))

copy = skb_tailroom(skb);

if ((err = skb_add_data(skb, from, copy)) != 0)

goto do_fault;

[...]

if (copied)

tcp_push(sk, flags, mss_now, tp->nonagle);

[...]

}

tcp_sengmsg gets tcp_sock (i.e.,TCP control block) from the socket and copies the data that the application has requested to transmit to the send socket buffer. When copying data to sk_buff, how many bytes will one sk_buff include? One sk_buff copies and includes MSS (tcp_send_mss) bytes to help the code that actually creates packets. Maximum Segment Size (MSS) stands for the maximum payload size that one TCP packet includes. By using TSO and GSO, one sk_buff can save more data than MSS. This will be discussed later, not in this document.

The sk_stream_alloc_skb function creates a new sk_buff, and skb_entail adds the new sk_buff to the tail of the send_socket_buffer. The skb_add_data function copies the actual application data to the data buffer of the sk_buff. All the data is copied by repeating the procedure (creating an sk_buff and adding it to the send socket buffer) several times. Therefore, sk_buffs at the size of the MSS are in the send socket buffer as a list. Finally, the tcp_push is called to make the data which can be transmitted now as a packet, and the packet is sent.

static inline void tcp_push(struct sock *sk, int flags, int mss_now, ...)

[...] ===>

static int tcp_write_xmit(struct sock *sk, unsigned int mss_now, ...)

int nonagle,

{

struct tcp_sock *tp = tcp_sk(sk);

struct sk_buff *skb;

[...]

while ((skb = tcp_send_head(sk))) {

[...]

cwnd_quota = tcp_cwnd_test(tp, skb);

if (!cwnd_quota)

break;



if (unlikely(!tcp_snd_wnd_test(tp, skb, mss_now)))

break;

[...]

if (unlikely(tcp_transmit_skb(sk, skb, 1, gfp)))

break;



/* Advance the send_head. This one is sent out.

* This call will increment packets_out.

*/

tcp_event_new_data_sent(sk, skb);

[...]

The tcp_push function transmits as many of the sk_buffs in the send socket buffer as the TCP allows in sequence. First, the tcp_send_head is called to get the first sk_buff in the socket buffer and the tcp_cwnd_test and the tcp_snd_wnd_test are performed to check whether the congestion window and the receive window of the receiving TCP allow new packets to be transmitted. Then, the tcp_transmit_skb function is called to create a packet.

static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,

int clone_it, gfp_t gfp_mask)

{

const struct inet_connection_sock *icsk = inet_csk(sk);

struct inet_sock *inet;

struct tcp_sock *tp;

[...]





if (likely(clone_it)) {

if (unlikely(skb_cloned(skb)))

skb = pskb_copy(skb, gfp_mask);

else

skb = skb_clone(skb, gfp_mask);

if (unlikely(!skb))

return -ENOBUFS;

}



[...]

skb_push(skb, tcp_header_size);

skb_reset_transport_header(skb);

skb_set_owner_w(skb, sk);



/* Build TCP header and checksum it. */

th = tcp_hdr(skb);

th->source = inet->inet_sport;

th->dest = inet->inet_dport;

th->seq = htonl(tcb->seq);

th->ack_seq = htonl(tp->rcv_nxt);

[...]

icsk->icsk_af_ops->send_check(sk, skb);

[...]

err = icsk->icsk_af_ops->queue_xmit(skb);

if (likely(err <= 0))

return err;



tcp_enter_cwr(sk, 1);



return net_xmit_eval(err);

}

tcp_transmit_skb creates the copy of the given sk_buff (pskb_copy). At this time, it does not copy the entire data of the application but the metadata. And then it calls skb_push to secure the header area and records the header field value. Send_check computes the TCP checksum. With the checksum offload, the payload data is not computed. Finally, queue_xmit is called to send the packet to the IP layer. Queue_xmit for IPv4 is implemented by the ip_queue_xmit function.

int ip_queue_xmit(struct sk_buff *skb)

[...]

rt = (struct rtable *)__sk_dst_check(sk, 0);

[...]

/* OK, we know where to send it, allocate and build IP header. */

skb_push(skb, sizeof(struct iphdr) + (opt ? opt->optlen : 0));

skb_reset_network_header(skb);

iph = ip_hdr(skb);

*((__be16 *)iph) = htons((4 << 12) | (5 << 8) | (inet->tos & 0xff));

if (ip_dont_fragment(sk, &rt->dst) && !skb->local_df)

iph->frag_off = htons(IP_DF);

else

iph->frag_off = 0;

iph->ttl = ip_select_ttl(inet, &rt->dst);

iph->protocol = sk->sk_protocol;

iph->saddr = rt->rt_src;

iph->daddr = rt->rt_dst;

[...]

res = ip_local_out(skb);

[...] ===>

int __ip_local_out(struct sk_buff *skb)

[...]

ip_send_check(iph);

return nf_hook(NFPROTO_IPV4, NF_INET_LOCAL_OUT, skb, NULL,

skb_dst(skb)->dev, dst_output);

[...] ===>

int ip_output(struct sk_buff *skb)

{

struct net_device *dev = skb_dst(skb)->dev;

[...]

skb->dev = dev;

skb->protocol = htons(ETH_P_IP);



return NF_HOOK_COND(NFPROTO_IPV4, NF_INET_POST_ROUTING, skb, NULL, dev,

ip_finish_output,

[...] ===>

static int ip_finish_output(struct sk_buff *skb)

[...]

if (skb->len > ip_skb_dst_mtu(skb) && !skb_is_gso(skb))

return ip_fragment(skb, ip_finish_output2);

else

return ip_finish_output2(skb);

The ip_queue_xmit function executes tasks required by the IP layers. __sk_dst_check checks whether the cached route is valid. If there is no cached route or the cached route is invalid, it performs IP routing. And then it calls skb_push to secure the IP header area and records the IP header field value. After that, as following the function call, ip_send_check computes the IP header checksum and calls the netfilter function. IP fragment is created when ip_finish_output function needs IP fragmentation. No fragmentation is generated when TCP is used. Therefore, ip_finish_output2 is called and it adds the Ethernet header. Finally, a packet is completed.

int dev_queue_xmit(struct sk_buff *skb)

[...] ===>

static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q, ...)

[...]

if (...) {

....

} else

if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q) &&



qdisc_run_begin(q)) {

[...]

if (sch_direct_xmit(skb, q, dev, txq, root_lock)) {

[...] ===>

int sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q, ...)

[...]

HARD_TX_LOCK(dev, txq, smp_processor_id());

if (!netif_tx_queue_frozen_or_stopped(txq))

ret = dev_hard_start_xmit(skb, dev, txq);



HARD_TX_UNLOCK(dev, txq);

[...]

}



int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev, ...)

[...]

if (!list_empty(&ptype_all))

dev_queue_xmit_nit(skb, dev);

[...]

rc = ops->ndo_start_xmit(skb, dev);

[...]

}

The completed packet is transmitted through the dev_queue_xmit function. First, the packet passes via the qdisc. If the default qdisc is used and the queue is empty, the sch_direct_xmit function is called to directly send down the packet to the driver, skipping the queue. Dev_hard_start_xmit function calls the actual driver. Before calling the driver, the device TX is locked first. This is to prevent several threads from accessing the device simultaneously. As the kernel locks the device TX, the driver transmission code does not need an additional lock. It is closely related to the parallel processing that will be discussed next time.

Ndo_start_xmit function calls the driver code. Just before, you will see ptype_all and dev_queue_xmit_nit. The ptype_all is a list that includes the modules such as packet capture. If a capture program is running, the packet is copied by ptype_all to the separate program. Therefore, the packet that tcpdump shows is the packet transmitted to the driver. When checksum offload or TSO is used, the NIC manipulates the packet. So the tcpdump packet is different from the packet transmitted to the network line. After completing packet transmission, the driver interrupt handler returns the sk_buff.

Following Code: How to Receive Data

The general executed path is to receive a packet and then to add the data to the receive socket buffer. After executing the driver interrupt handler, follow the napi poll handle first.

static void net_rx_action(struct softirq_action *h)

{

struct softnet_data *sd = &__get_cpu_var(softnet_data);

unsigned long time_limit = jiffies + 2;

int budget = netdev_budget;

void *have;



local_irq_disable();



while (!list_empty(&sd->poll_list)) {

struct napi_struct *n;

[...]

n = list_first_entry(&sd->poll_list, struct napi_struct,

poll_list);

if (test_bit(NAPI_STATE_SCHED, &n->state)) {

work = n->poll(n, weight);

trace_napi_poll(n);

}

[...]

}



int netif_receive_skb(struct sk_buff *skb)

[...] ===>

static int __netif_receive_skb(struct sk_buff *skb)

{

struct packet_type *ptype, *pt_prev;

[...]

__be16 type;

[...]

list_for_each_entry_rcu(ptype, &ptype_all, list) {

if (!ptype->dev || ptype->dev == skb->dev) {

if (pt_prev)

ret = deliver_skb(skb, pt_prev, orig_dev);

pt_prev = ptype;

}

}

[...]

type = skb->protocol;

list_for_each_entry_rcu(ptype,

&ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {

if (ptype->type == type &&



(ptype->dev == null_or_dev || ptype->dev == skb->dev ||

ptype->dev == orig_dev)) {

if (pt_prev)

ret = deliver_skb(skb, pt_prev, orig_dev);

pt_prev = ptype;

}

}



if (pt_prev) {

ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);



static struct packet_type ip_packet_type __read_mostly = {

.type = cpu_to_be16(ETH_P_IP),

.func = ip_rcv,

[...]

};

As mentioned before, the net_rx_action function is the softirq handler that receives a packet. First, the driver that has requested the napi poll is retrieved from the poll_list and the poll handler of the driver is called. The driver wraps the received packet with sk_buff and then calls netif_receive_skb.

When there is a module that requests all packets, the netif_receive_skb sends packets to the module. Like packet transmission, the packets are transmitted to the module registered to the ptype_all list. The packets are captured here.

Then, the packets are transmitted to the upper layer based on the packet type. The Ethernet packet includes 2-byte ethertype field in the header. The value indicates the packet type. The driver records the value in sk_buff (skb->protocol). Each protocol has its own packet_type structure and registers the pointer of the structure to the ptype_base hash table. IPv4 uses ip_packet_type. The Type field value is the IPv4 ethertype (ETH_P_IP) value. Therefore, the IPv4 packet calls the ip_rcv function.

int ip_rcv(struct sk_buff *skb, struct net_device *dev, ...)

{

struct iphdr *iph;

u32 len;

[...]

iph = ip_hdr(skb);

[...]

if (iph->ihl < 5 || iph->version != 4)

goto inhdr_error;



if (!pskb_may_pull(skb, iph->ihl*4))

goto inhdr_error;



iph = ip_hdr(skb);



if (unlikely(ip_fast_csum((u8 *)iph, iph->ihl)))

goto inhdr_error;



len = ntohs(iph->tot_len);

if (skb->len < len) {

IP_INC_STATS_BH(dev_net(dev), IPSTATS_MIB_INTRUNCATEDPKTS);

goto drop;

} else if (len < (iph->ihl*4))

goto inhdr_error;

[...]

return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,

ip_rcv_finish);

[...] ===>

int ip_local_deliver(struct sk_buff *skb)

[...]

if (ip_hdr(skb)->frag_off & htons(IP_MF | IP_OFFSET)) {

if (ip_defrag(skb, IP_DEFRAG_LOCAL_DELIVER))

return 0;

}



return NF_HOOK(NFPROTO_IPV4, NF_INET_LOCAL_IN, skb, skb->dev, NULL,

ip_local_deliver_finish);

[...] ===>





static int ip_local_deliver_finish(struct sk_buff *skb)

[...]

__skb_pull(skb, ip_hdrlen(skb));

[...]

int protocol = ip_hdr(skb)->protocol;

int hash, raw;

const struct net_protocol *ipprot;

[...]

hash = protocol & (MAX_INET_PROTOS - 1);

ipprot = rcu_dereference(inet_protos[hash]);

if (ipprot != NULL) {

[...]

ret = ipprot->handler(skb);

[...] ===>



static const struct net_protocol tcp_protocol = {

.handler = tcp_v4_rcv,

[...]

};

The ip_rcv function executes tasks required by the IP layers. It examines packets such as the length and header checksum. After passing through the netfilter code, it performs the ip_local_deliver function. If required, it assembles IP fragments. Then, it calls ip_local_deliver_finish through the netfilter code. The ip_local_deliver_finish function removes the IP header by using the __skb_pull and then searches the upper protocol whose value is identical to the IP header protocol value. Similar to the Ptype_base, each transport protocol registers its own net_protocol structure in inet_protos. IPv4 TCP uses tcp_protocol and calls tcp_v4_rcv that has been registered as a handler.

When packets come into the TCP layer, the packet processing flow varies depending on the TCP status and the packet type. Here, we will see the packet processing procedure when the expected next data packet has been received in the ESTABLISHED status of the TCP connection. This path is frequently executed by the server receiving data when there is no packet loss or out-of-order delivery.

int tcp_v4_rcv(struct sk_buff *skb)

{

const struct iphdr *iph;

struct tcphdr *th;

struct sock *sk;

[...]

th = tcp_hdr(skb);



if (th->doff < sizeof(struct tcphdr) / 4)

goto bad_packet;

if (!pskb_may_pull(skb, th->doff * 4))

goto discard_it;

[...]

th = tcp_hdr(skb);

iph = ip_hdr(skb);

TCP_SKB_CB(skb)->seq = ntohl(th->seq);

TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin +

skb->len - th->doff * 4);

TCP_SKB_CB(skb)->ack_seq = ntohl(th->ack_seq);

TCP_SKB_CB(skb)->when = 0;

TCP_SKB_CB(skb)->flags = iph->tos;

TCP_SKB_CB(skb)->sacked = 0;



sk = __inet_lookup_skb(&tcp_hashinfo, skb, th->source, th->dest);

[...]

ret = tcp_v4_do_rcv(sk, skb);

First, the tcp_v4_rcv function validates the received packets. When the header size is larger than the data offset (th->doff < sizeof(struct tcphdr) / 4), it is the header error. And then __inet_lookup_skb is called to look for the connection where the packet belongs from the TCP connection hash table. From the sock structure found, all required structures such as tcp_sock and socket can be got.

int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)

[...]

if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */

sock_rps_save_rxhash(sk, skb->rxhash);

if (tcp_rcv_established(sk, skb, tcp_hdr(skb), skb->len)) {

[...] ===>

int tcp_rcv_established(struct sock *sk, struct sk_buff *skb,

[...]

/*

* Header prediction.

*/

if ((tcp_flag_word(th) & TCP_HP_BITS) == tp->pred_flags &&



TCP_SKB_CB(skb)->seq == tp->rcv_nxt &&



!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt))) {

[...]

if ((int)skb->truesize > sk->sk_forward_alloc)

goto step5;



NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPHPHITS);



/* Bulk data transfer: receiver */

__skb_pull(skb, tcp_header_len);

__skb_queue_tail(&sk->sk_receive_queue, skb);

skb_set_owner_r(skb, sk);

tp->rcv_nxt = TCP_SKB_CB(skb)->end_seq;

[...]

if (!copied_early || tp->rcv_nxt != tp->rcv_wup)

__tcp_ack_snd_check(sk, 0);

[...]

step5:

if (th->ack && tcp_ack(sk, skb, FLAG_SLOWPATH) < 0)

goto discard;



tcp_rcv_rtt_measure_ts(sk, skb);



/* Process urgent data. */

tcp_urg(sk, skb, th);



/* step 7: process the segment text */

tcp_data_queue(sk, skb);



tcp_data_snd_check(sk);

tcp_ack_snd_check(sk);

return 0;

[...]

}

The actual protocol is executed from the tcp_v4_do_rcv function. If the TCP is in the ESTABLISHED status, tcp_rcv_esablished is called. Processing of the ESTABLISHED status is separately handled and optimized since it is the most common status. The tcp_rcv_established first executes the header prediction code. The header prediction is also quickly processed to detect in the common state. The common case here is that there is no data to transmit and the received data packet is the packet that must be received next time, i.e., the sequence number is the sequence number that the receiving TCP expects. In this case, the procedure is completed by adding the data to the socket buffer and then transmitting ACK.

Go forward and you will see the sentence comparing truesize with sk_forward_alloc. It is to check whether there is any free space in the receive socket buffer to add new packet data. If there is, header prediction is "hit" (prediction succeeded). Then __skb_pull is called to remove the TCP header. After that, __skb_queue_tail is called to add the packet to the receive socket buffer. Finally, __tcp_ack_snd_check is called for transmitting ACK if necessary. In this way, packet processing is completed.

If there is not enough free space, a slow path is executed. The tcp_data_queue function newly allocates the buffer space and adds the data packet to the socket buffer. At this time, the receive socket buffer size is automatically increased if possible. Different from the quick path, tcp_data_snd_check is called to transmit a new data packet if possible. Finally, tcp_ack_snd_check is called to create and transmit the ACK packet if necessary.

The amount of code executed by the two paths is not much. This is accomplished by optimizing the common case. In other words, it means that the uncommon case will be processed significantly more slowly. The out-of-order delivery is one of the uncommon cases.

How to Communicate between Driver and NIC

Communication between a driver and the NIC is the bottom of the stack and most people do not care about it. However, the NIC is executing more and more tasks to solve the performance issue. Understanding the basic operation scheme will help you understand the additional technology.

A driver and the NIC asynchronously communicate. First, a driver requests packet transmission (call) and the CPU performs another task without waiting for the response. And then the NIC sends packets and notifies the CPU of that, the driver returns the received packets (returns the result). Like packet transmission, packet receiving is asynchronous. First, a driver requests packet receiving and the CPU performs another task (call). Then, the NIC receives packets and notifies the CPU of that, and the driver processes the received packets received (returns the result).

Therefore, a space to save the request and the response is necessary. In most cases, the NIC uses the ring structure. The ring is similar to the common queue structure. With the fixed number of entries, one entry saves one request data or one response data. The entries are sequentially used in turn. The name "ring" is generally used since the fixed entries are reused in turn.

As following the packet transmission procedure shown in the following Figure 8, you will see how the ring is used.

Figure 8: Driver-NIC Communication: How to Transmit Packet.

The driver receives packets from the upper layer and creates the send descriptor that the NIC can understand. The send descriptor includes the packet size and the memory address by default. As the NIC needs the physical address to access the memory, the driver should change the virtual address of the packets to the physical address. Then, it adds the send descriptor to the TX ring (1). The TX ring is the send descriptor ring.

Next, it notifies the NIC of the new request (2). The driver directly writes the data to a specific NIC memory address. In this way, Programmed I/O (PIO) is the data transmission method in which the CPU directly sends data to the device.

The notified NIC gets the send descriptor of the TX ring from the host memory (3). Since the device directly accesses the memory without intervention of the CPU, the access is called Direct Memory Access (DMA).

After getting the send descriptor, the NIC determines the packet address and the size and then gets the actual packets from the host memory (4). With the checksum offload, the NIC computes the checksum when the NIC gets the packet data from the memory. Therefore, overhead rarely occurs.

The NIC sends packets (5) and then writes the number of packets that are sent to the host memory (6). Then, it sends an interrupt (7). The driver reads the number of packets that are sent and then returns the packets that have been sent so far.

In the following Figure 9, you will see the procedure of receiving packets.

Figure 9: Driver-NIC Communication: How to Receive Packets.

First, the driver allocates the host memory buffer for receiving packets and then creates the receive descriptor. The receive descriptor includes the buffer size and the memory address by default. Like the send descriptor, it saves the physical address that the DMA uses in the receive descriptor. Then, it adds the receive descriptor to the RX ring (1). It is the receive request and the RX ring is the receive request ring.

Through the PIO, the driver notifies that there is a new descriptor in the NIC (2). The NIC gets the new descriptor of the RX ring. And then it saves the size and location of the buffer included in the descriptor to the NIC memory (3).

After the packets have been received (4), the NIC sends the packets to the host memory buffer (5). If the checksum offload function is existing, the NIC computes the checksum at this time. The actual size of received packets, the checksum result, and any other information are saved in the separate ring (the receive return ring) (6). The receive return ring saves the result of processing the receive request, i.e., the response. And then the NIC sends an interrupt (7). The driver gets packet information from the receive return ring and processes the received packets. If necessary, it allocates new memory buffer and repeats Step (1) and Step (2).

To tune the stack, most people say that the ring and interrupt setting should be adjusted. When the TX ring is large, a lot of send requests can be made at once. When the RX ring is large, a lot of packet receives can be done at once. A large ring is useful for the workload that has a huge burst of packet transmission/receiving. In most cases, the NIC uses a timer to reduce the number of interrupts since the CPU may suffer from large overhead to process interrupts. To avoid flooding the host system with too many interrupts, interrupts are collected and sent regularly(interrupt coalescing) while sending and receiving the packets.

Stack Buffer and Flow Control

Flow control is executed in several stages in the stack. Figure 10 shows buffers used to transmit data. First, an application creates data and adds it to the send socket buffer. If there is no free space in the buffer, the system call is failed or the blocking occurs in the application thread. Therefore, the application data rate flowing into the kernel must be controlled by using the socket buffer size limit.

Figure 10: Buffers Related to Packet Transmission.

The TCP creates and sends packets to the driver through the transmit queue (qdisc). It is a typical FIFO queue type and the maximum length of the queue is the value of txqueuelen which can be checked by executing the ifconfig command. Generally, it is thousands of packets.

The TX ring is between the driver and the NIC. As mentioned before, it is considered as a transmission request queue. If there is no free space in the queue, no transmission request is made and the packets are accumulated in the transmit queue. If too many packets are accumulated, packets are dropped.

The NIC saves the packets to transmit in the internal buffer. The packet rate from this buffer is affected by the physical rate (ex: 1 Gb/s NIC cannot offer performance of 10 Gb/s). And with the Ethernet flow control, packet transmission is stopped if there is no free space in the receive NIC buffer.

When the packet rate from the kernel is faster than the packet rate from the NIC, packets are accumulated in the buffer of the NIC. If there is no free space in the buffer, processing of transmission request from the TX ring is stopped. More and more requests are accumulated in the TX ring and finally there is no free space in the queue. The driver cannot make any transmission request and the packets are accumulated in the transmit queue. Like this, backpressure is sent from the bottom to the top through many buffers.

Figure 11 shows the buffers that the receive packets are passing. The packets are saved in the receive buffer of the NIC. From the view of flow control, the RX ring between the driver and the NIC is considered as a packet buffer. The driver gets packets coming into the RX ring and then sends them to the upper layer. There is no buffer between the driver and the upper layer since the NIC driver that is used by the server system uses NAPI by default. Therefore, it can be considered as the upper layer directly gets packets from the RX ring. The payload data of packets is saved in the receive socket buffer. The application gets the data from the socket buffer later.

Figure 11: Buffers Related to Packet Receiving.

The driver that does not support NAPI saves packets in the backlog queue. Later, the NAPI handler gets packets. Therefore, the backlog queue can be considered as a buffer between the upper layer and the driver.

If the packet processing rate of the kernel is slower than the packet flow rate into the NIC, the RX ring space is full. And the space of the buffer in the NIC is full, too. When the Ethernet flow control is used, the NIC sends a request to stop transmission to the transmission NIC or makes the packet drop.

There is no packet drop due to lack of space in the receive socket buffer because the TCP supports end-to-end flow control. However, packet drop occurs due to lack of space in the socket buffer when the application rate is slow because the UDP does not support flow control.

The sizes of the TX ring and the RX ring used by the driver in Figure 10 and Figure 11 are the sizes of the rings shown by the ethtool. For most workloads which regard throughput as important, it will be helpful to increase the ring size and the socket buffer size. Increasing the sizes reduces the possibility of failures caused by lack of space in the buffer while receiving and transmitting a lot of packets at a fast rate.

Conclusion

Initially, I planned to explain only the things that would be helpful for you to develop network programs, execute performance tests, and perform troubleshooting. In spite of my initial plan, the amount of description included in this document is not small. I hope this document will help you to develop network applications and monitor their performance. The TCP/IP protocol itself is very complicated and has many exceptions. However, you don't need to understand every line of TCP/IP-related code of the OS to understand performance and analyze the phenomena. Just understanding its context will be very helpful for you.

With continuous advancement of system performance and implementation of the OS network stack, the latest server can offer 10-20 Gb/s TCP throughput without any problem. These days, there are too many technology types related to performance, such as TSO, LRO, RSS, GSO, GRO, UFO, XPS, IOAT, DDIO, and TOE, just like alphabet soup, to make us confused.

In the next article, I will explain about the network stack from the performance perspective and discuss the problems and effects of this technology.

By Hyeongyeop Kim, Senior Engineer at Performance Engineering Lab, NHN Corporation.

How is SSD Changing Software Architecture?

2013-02-24T20:38:58-08:00

This is the second article on SSD and the performance implications of switching to SSD. In the first article, Will the use of SSD increase the speed of DBMS?, we have reviewed how much performance gain we would see if we switched from HDD to SSD for a DBMS like CUBRID, and what factors would affect the performance.

In this article, I will compare the SSD structure to that of HDD and discuss how software architecture is changing.

Overview

Last year in July Amazon released a new cloud product - High I/O Quadruple Extra Large. This product uses 2 TB Solid-State Drive (SSD). So now even cloud products have started using SSD. That means the SSD price is becoming low enough to be adopted by cloud products. The operational know-how for cloud services is now established. Then, can we assume that this is the era for using SSDs?

A Solid-State Drive uses NAND flash memory and its operation is different from the operation of a Hard Disk Drive. To develop a storage system, including a database, we needed to fully understand the HDD structure for higher performance. Do we also need to fully understand the SSD structure for greater performance?

The answer is Yes. As SSDs become popular, software architecture is changing. The architecture designed according to the HDD characteristics is being redesigned considering the SSD characteristics.

Introduction to the Experiences of EC2 Users

Below I will use some of the reviews posted on Amazon High IO Instance related High Scalability blog.

Case #1: Conversocial

Conversocial, the social CRM firm, was using MongoDB when the High IO product was released, requiring MongoDB to run on High IO. Compared to the previous HDD, the average response time was reduced by 43% (from 392 ms to 274 ms), the large data for random IO was reduced by 74% (from 877 ms to 504 ms), and the iowait was reduced from 90% to 3%.

Case #2: Netflix

Netflix, the DVD rental site, has posted a long article related to SSD as well. In the previous configuration, the performance was 100K IOPS or 1 GB/sec. As hi1.4xlarge was used, the response to the average read request was reduced from 10 ms to 2.2 ms and the 99% request response time was reduced from 65 ms to 10 ms.

Case #3: MySQL on EC2

Though SSD adoption does not guarantee enhanced performance all the time, on the High Scalability page we can also see the following statement:

According to the benchmark, now we can choose EC2 to run MySQL on.

Implications

If you use HDDs, you should prepare the memory based on the working set. If the size exceeds, increase the number of equipments for horizontal partitioning.

On the other hand, SSDs can help to reduce the required memory and can delay introduction of horizontal partitioning. For example, Netflix has removed the memcached layer for 48 IO instances. Then it replaced the layer with 15 instances where no cache was involved. Finally, the DBMS did not need to use additional cache to reduce IO time.

Combined with the flexibility of EC2, SSDs can cut costs. For example, for a large capacity index of the genome mapping, the index operation can be performed intensively for a few hours per week. For the rest of the time, there is no need to perform the computation. Without EC2, you must purchase an expensive equipment, use it periodically, and leave it unused for most of the time. With EC2 you can save on the equipment, and pay for only what you use.

While the popularization of SSD cuts costs as well as enhances the performance, the cost-cutting effect can be further maximized when SSDs are combined with cloud products.

Characteristic Comparisons between SSD and HDD IO

An SSD is a non-volatile memory, running in an electronic manner. Compared to HDDs, SSDs provide lower seek times, fewer mechanical delays, and lower fault rates. Therefore, they provide a very fast random read. However, an overhead occurs in random write. The reason is that SSDs cannot overwrite data while writing data. An SSD is divided into many layers in the order of block, page, row, and cell. Data can be written by changing some cells to 0 while all cells in the block have been initialized to 1. To raise 0 to 1, high voltage is required. However, high voltage affects the neighboring cells (write amplification). Therefore, writing data requires two stages of erase and program.

Figure 1: SSD Cell Structure (source).

An SSD writes by page and the number of writes on the page is limited (write endurance). If the endurance is exceeded, the block fails. To solve this problem, SSD manufacturers use the following methods on their firmware:

Wear Leveling

Uses blocks equally for lifetime extension. Counts the number of writes per block, and selects and uses the blocks with the fewest counts.

Write Gathering

If there is an erased empty block, gathers the blocks in use, in order to make the blocks empty. In addition, metadata is updated whenever data is written. As write positions are scattered, metadata update times become longer. Sometimes, the time is almost the same as HDD seek time. Therefore, to reduce write overhead, several write requests are gathered and executed at once.

Garbage Collection

The write unit is page and the erase unit is block. Among blocks, blocks that have the largest number of unused pages are initialized (erased). It is a type of defragmentation method; first, back up valid pages in a block and then initialize the block to an empty block, then push the valid pages backed up on another block into the empty block.

Firmware runs differently by manufacturer. This is the competitive element of an SSD. Manufacturers keep their secrets and provide products by level. There are two levels, the low-end Multi Level Cell (MLC) and the high-end Single Level Cell (SLC). The two show significant differences in reliability and speed.

When comparing the performance of an SSD to that of HDD, an SSD is generally 120,000 random reads and 10,000~85,000 random writes, based on IOPS. However, an HDD with 15,000 RPM generally shows performance of 175~210 IOPS. As a result, SSD shows 50~600 times of high performance, compared to an HDD.

An SSD has no seek time, so it provides faster IO than an HDD. However, there is overhead in random write, as we have discussed earlier. The write optimization method of each manufacturer cannot be known. So, the exact IO pattern can be determined by testing each product. However, forecasting the write optimization method seems possible to a certain level. The next chapter will show how to optimize IO under these SSD constraints.

Trials to Optimize IO on SSD

An HDD is significantly slower than the memory. The performance of 10,000 RPM HDD is different from the performance of DDR3-2500 by approximately 800 times. Since HDDs have been used, many methods have been created to improve storage system performance. For example, when the OS and the device driver receive IO requests, they schedule the requests to minimize seek time. When an SSD is used, they do not schedule requests because SSD firmware performs that scheduling.

Systems like a DBMS effectively implement the buffer (cache) to minimize IO on HDDs. As a result, when the hit rate is high, replacing an HDD to an SSD may improve the performance but slightly. However, with SSD, there is no need to keep the buffer size large, you may not need to retain a large memory capacity.

In addition, a variety of methods are used to write data compactly, minimizing movement of the HDD head. However, these methods are not necessary since the random read of SSD is fast.

The operation methods of SSD firmware are not open. So, it is difficult to model the IO pattern at the OS or device driver.

However, at the application level, the architecture is being changed considering that the performance of random read and sequential write of SSD is great but random write performance is poor. Let's see how the architecture is changed.

Log Structured File System

As the memory capacity of a server increases, random read occurs less often than before due to memory cache. Therefore, random write becomes an important performance issue. An HDD performs an in-place write. If the random IO becomes larger, the head must move, causing an increase of seek time and degradation of performance. To compensate for that, there is a way to log the write history only and perform data write occasionally. This method is called journal file system or logging file system. Furthermore, you can avoid the in-place write by writing the data, instead of the history, whenever logging is performed. This is called log-structured file system (LFS).

When writing data, the LFS reads the previous version to create the latest version and appends the latest version to the end of a file.

Figure 2: Architecture of Log Structured File System (source).

At this time, the previous version is marked as an empty space. The version is always appended at the end of the file whenever it is required, only the sequential write is needed, so you can obtain the good performance at the early stage. However, as empty space increases, data is fragmented and an overhead occurs to gather the fragmented data. LFS writes a file with the serial chunk unit, called segment. In order to reduce the overhead of metadata updates, which occurs per write, several writes are collected and written at once. Since most of segments are partially written, a task (GC) to create an empty segment is performed. It is similar to the efficient write of an SSD where data is gathered and written on sequential spaces with defragmentation for creating an empty space.

If an SSD does not perform wear leveling and write gathering, the file system will be more effective. The Journaling Flash File System (JFFS2), which is frequently used for embedded devices, is an example of the file system.

On the performance side, in some cases, LFS can reduce disk usage by 40% compared to general file systems. However, LFS uses most of its time gathering segments. So, performance is lowered because it must process the segments at a certain period.

B-Tree Performance and Copy on-write B-Tree

B-Tree is a data architecture frequently used by file systems and databases (eg. CUBRID). Will the B-Tree work well on an SSD? The following graph shows the performance result calculated by measuring the B-Tree insert on the X25M SSD with three workloads.

Figure 3: Performance of B-Tree Insert on SSD (source).

The red color is the random write at 4 K size, the green color is the sequential write at 4 K size, and the blue color is the random write at 512 K size. At 4 K size, it seems natural that the sequential write shows the highest performance. An interesting thing is that the write performance is significantly lowered at 512 K size when the device capacity (160 G) is exceeded. First, performance is degraded when device capacity is full with the write volume. Why? The reason is that most SSDs have a log structure due to wear leveling and error correction. Originally, a block is 512 K; however, the latest MLC devices allow erasing data with the larger size. Finally, write can erase hundreds of MB with device firmware execution.

Anyway, an SSD shows better performance of the sequential write than the random write. Is there a B-Tree that has a method allowing sequential write? Copy on-write (CoW) copies the path of B-Tree. It copies the traversed nodes whenever the nodes of the tree are changed, and then changes the nodes to create a new tree. See the following two sides:

Traverses the tree to find a key
Rewrites the path

In the case of (1) for the CoW, the random IO (random write) occurs (of course, IO may not occur when cache is used); however, case (2) is completely the sequential write. Because the existing nodes are immutable, only the sequential write occurs (append only). This method has already been used in file systems such as ZFS, WAFL, and Btrfs.

Figure 4: CoW B-Tree Structure (source).

Then, does the CoW B-Tree work well with the log-structured storage structure of an SSD? Basically, it works well as it is 'append only'. However, there are two issues: space blowup and garbage collection.

CoW B-tree potentially creates a large space blowup. If a 16-byte key/value should be saved in an index with a depth of 3 and the block size is 256 K, data write should perform as much as 256*3 K (768 K). Compared to the small change, larger data should be written. In addition, as the size of space blowup becomes larger, GC should perform more tasks. Since GC runs when there are few IO, performance may be degraded.

Fractal Tree

At B-Tree, sequential insert is performed quickly because it has optimum locality by changing only specific pages. However, random insert causes high entropy. At the early stage, most pages are loaded on the memory. As time goes by, the possibility that some pages to be accessed in seeking are on the disk (aging) becomes higher. Especially, when terminal pages are scattered and saved on the disk, performance worsens.

In the DAM model, IO is performed in the unit of block size between memory and storage. Generally, IO is performed by seeking the optimum block size for tuning. However, B-Tree generally uses one key but loads all blocks on the disk, consuming a lot of IO bandwidth. To optimize IO, the Cache Oblivious model is sometimes used. For this model, the IO size should be decided based on the algorithm applied because it is impossible to estimate optimum block size. In addition, as with LFS, B-Tree requires append to reduce random IO.

Fractal Tree is based on them. First, data is inserted as several arrays which are increased exponentially. When insert is performed, the entry is saved in the smallest array. Arrays where data is inserted are merged into a larger array; at this time, the arrays are sorted in the order of keys. This is called Doubling Array.

Figure 5: Data Insert Process in Fractal Tree (source).

This array performs sequential write whenever the array grows. In this structure, insert is performed quickly. However, if the key should be navigated, binary navigation is performed so it performs more slowly than B-Tree. In order to improve this, the forward pointer is made and the tree is configured with levels. This is called Fractal Cascading.

Figure 6: Fractal Cascading.

This method is used by TokuDB, one of the MySQL storage engines. TokuDB is storage for write-intensive workloads.

Stratified B-Tree

Weak points of the CoW B-Tree are space blowup and performance degradation caused by GC. The Stratified B-Tree makes up for the weak points. It is one of the versioned dictionaries. It follows the Cache Oblivious model similar to Fractal Tree. It is a hierarchical tree, which runs by using Doubling Array and has the forward pointer. The difference is that it has the version by key.

First, when a key is changed, the existing key version is maintained with the entry {key, version, value_or_pointer}. When Insert comes in, the key value is saved in the memory buffer, arranged, and written in the array with the lowest level when flushed. After that, as the arrays grow, the key versions of arrays with the same level may be duplicated. In this case, the arrays are promoted and then demoted by disjointing them to avoid version duplication. The reason for demoting arrays is a concept, called density. Density is the density degree of a live key in an array based on version v. When the density is low, unnecessary keys should be searched when range query is performed based on version v. On the contrary, when density is high, most keys are under the corresponding condition. However, a great deal of space blowup is made because data should be duplicated to increase the density. Finally, Stratified B-Tree promotes arrays to prevent each array version from being different, and then demotes the arrays with high density in order to improve the range query performance.

This tree provides slow point query performance, but is good for analysis query and range query for big data. In addition, Acunu is a storage platform implemented with this method. It is used for NoSQL including Cassandra.

Changes in Software Architecture Caused by SSD

So far, I have introduced how the software architecture of storage systems is being changed in the era of SSDs. When SSDs are generalized, products leading the marketplace will be software with architecture suitable for SSDs. No one knows whether HDD era winners will hold dominant positions in the SSD era or if a new hero will appear. Successful methods for existing HDDs are not suitable for SSDs. Therefore, it is very interesting to imagine the changes we will meet in this ongoing evolution.

By Hyejeong Lee, Senior Software Engineer at Storage System Development Team, NHN Corporation.

My other posts:

The Principles of Java Application Performance Tuning

2013-02-07T01:26:09-08:00

This is the fifth article in the series of "Become a Java GC Expert". In the first issue Understanding Java Garbage Collection we have learned about the processes for different GC algorithms, about how GC works, what Young and Old Generation is, what you should know about the 5 types of GC in the new JDK 7, and what the performance implications are for each of these GC types.

In the second article How to Monitor Java Garbage Collection we have explained how JVM actually runs the Garbage Collection in the real time, how we can monitor GC, and which tools we can use to make this process faster and more effective.

In the third article How to Tune Java Garbage Collection we have shown some of the best options based on real cases as our examples that you can use for GC tuning. Also we have explained how to minimize the number of objects passed to Old Area, decreasing Full GC time, as well as how to set GC type and the memory size.

In the fourth article MaxClients in Apache and its effect on Tomcat during Full GC we have explained the importance of MaxClients parameter in Apache that significantly affects the overall system performance when GC occurs.

In this fifth article I will explain about the principles of Java application performance tuning. Specifically, I will explain what is required in order to tune the performance of Java application, the steps you need to perform to identify whether your application needs tuning. I will also explain the problems you may encounter during performance tuning. The article will be finalized with the recommendations you need to follow to make better decisions when tuning Java applications.

Overview

Not every application requires tuning. If an application performs as well as expected, you don't need to exert additional efforts to enhance its performance. However, it would be difficult to expect an application would reach its target performance as soon as it finishes debugging. This is when tuning is required. Regardless of the implementation language, tuning an application requires high expertise and concentration. Also, you may not use the same method for tuning a certain application to tune another application. This is because each application has its unique action and a different type of resource usage. For this reason, tuning an application requires more basic knowledge compared to the knowledge required to write an application. For example, you need knowledge on virtual machines, operating systems and computer architectures. When you focus on an application domain based on such knowledge, you can successfully tune an application.

Sometimes Java application tuning requires only changing JVM options, such as Garbage Collector, but sometimes it requires changing the application source code. Whichever method you choose, you need to monitor the process of executing the Java application first. For this reason, the issues this article will deal with are as follows:

How can I monitor a Java application?
What JVM options should I give?
How can I know if modifying source codes is required or not?

Knowledge Required to Tune the Performance of Java Applications

Java applications operate inside Java Virtual Machine (JVM). Therefore, to tune a Java application, you need to understand the JVM operation process. I have previously blogged about Understanding JVM Internals where you can find great insights about JVM.

The knowledge regarding the process of the operation of JVM in this article mainly refers to the knowledge of Garbage Collection (GC) and Hotspot. Although you may not be able to tune the performance of all kinds of Java applications only with the knowledge on GC or Hotspot, these two factors influence the performance of Java applications in most cases.

It is noted that from the perspective of an operating system JVM is also an application process. To make an environment in which a JVM can operate well, you should understand how an OS allocates resources to processes. This means, to tune the performance of Java applications, you should have an understanding of OS or hardware as well as JVM itself.

Another aspect is that knowledge of Java language domain is also important. It is also important to understand lock or concurrency and to be familiar with class loading or object creation.

When you carry out Java application performance tuning, you should approach it by integrating all this knowledge.

The Process of Java Application Performance Tuning

Figure 1 shows a flow chart from the book <Java Performance> co-authored by Charlie Hunt and Binu John. This chart shows the process of Java application performance tuning.

Figure 1: The Process of Tuning the Performance of Java Applications.

The above process is not a one-time process. You may need to repeat it until the tuning is completed. This also applies to determining an expected performance value. In the process of tuning, sometimes you should lower the expected performance value, and sometimes raise it.

JVM distribution model

A JVM distribution model is related with making a decision on whether to operate Java applications on a single JVM or to operate them on multiple JVMs. You can decide it according to its availability, responsiveness and maintainability. When operating JVM on multiple servers, you can also decide whether to run multiple JVMs on a single server or to run a single JVM per server. For example, for each server, you can decide whether to run a single JVM using a heap of 8 GB, or to use four JVMs each using a heap of 2 GB. Of course, you can decide the number of JVMs running on a single server depending on the number of cores and the characteristics of the application. When comparing the two settings in terms of responsiveness, it might be more advantageous to use a heap of 2 GB rather than 8 GB for the same application, for it takes shorter to perform a full garbage collection when using a heap of 2 GB. If you use a heap of 8 GB, however, you can reduce the frequency of full GCs. You can also improve responsiveness by increasing the hit rate if the application uses internal cache. Therefore, you can choose a suitable distribution model by taking into account the characteristics of the application and the method to overcome the disadvantage of the model you chose for some advantages.

JVM architecture

Selecting a JVM means whether to use a 32-bit JVM or a 64-bit JVM. Under the same conditions, you had better choose a 32-bit JVM. This is because a 32-bit JVM performs better than a 64-bit JVM. However, the maximum logical heap size of a 32-bit JVM is 4 GB. (However, actual allocatable size for both 32-bit OS and 64-bit OS is 2-3 GB.) It is appropriate to use a 64-bit JVM when a heap size larger than this is required.

**Table 1: Performance Comparison (**source).
Benchmark	Time (sec)	Factor
C++ Opt	23	1.0x
C++ Dbg	197	8.6x
Java 64-bit	134	5.8x
Java 32-bit	290	12.6x
Java 32-bit GC*	106	4.6x
Java 32-bit SPEC GC*	89	3.7x
Scala	82	3.6x
Scala low-level*	67	2.9x
Scala low-level GC*	58	2.5x
Go 6g	161	7.0x
Go Pro*	126	5.5x

The next step is to run the application and to measure its performance. This process includes tuning GC, changing OS settings and modifying codes. For these tasks, you can use a system monitoring tool or a profiling tool.

It should be noted that tuning for responsiveness and tuning for throughput could be different approaches. Responsiveness will be reduced if stop-the-world occurs from time to time, for example, for a full garbage collection despite a large amount of throughput per unit time. You also need to consider that a trade-off could occur. Such trade-off could occur not only between responsiveness and throughput. You may need to use more CPU resources to reduce memory usage or put up with reduction in responsiveness or throughput. As opposite cases could likewise occur, you need to approach it according to the priority.

The flow chart of Figure 1 above shows the performance tuning approach for almost all kinds of Java applications, including Swing applications. However, this chart is somewhat unsuitable for writing a server application for Internet service as our company NHN does. The flow chart in Figure 2 below is a simpler procedure designed based on Figure 1 to be more suitable for NHN.

Figure 2: A Recommended Procedure for Tuning NHN's Java Applications.

Select JVM in the above flow chart means using a 32-bit JVM as much as possible except when you need to use a 64-bit JVM to maintain cache of several GB.

Now, based on the flow chart in Figure 2, you will learn about things to do to execute each of the steps.q

JVM Options

I will explain how to specify suitable JVM options mainly for a web application server. Despite not being applied to every case, the best GC algorithm, especially for web server applications, is the Concurrent Mark Sweep GC. This is because what matters is low latency. Of course, when using the Concurrent Mark Sweep, sometimes a very long stop-the-world phenomenon could take place due to fractions. Nevertheless, this problem is likely to be resolved by adjusting the new area size or the fraction ratio.

Specifying the new area size is as important as specifying the entire heap size. You had better specify the ratio of the new area size to the entire heap size by using –XX:NewRatio or specify the desired new area size by using the –XX:NewSize option. Specifying a new area size is important because most objects cannot survive long. In web applications, most objects, except cache data, are generated when HttpResponse to HttpRequest is created. This time hardly exceeds a second. This means the life of objects does not exceed a second, either. If the new area size is not large, it should be moved to the old area to make space for newly created objects. The cost for GC for the old area is much bigger than that for the new area; therefore, it is good to set the size of the new area sufficiently.

If the new area size exceeds a certain level, however, responsiveness will be reduced. This is because the garbage collection for the new area is basically to copy data from one survivor area to another survivor area. Also, the stop-the-world phenomenon will occur even when performing GC for the new area as well as the old area. If the new area becomes bigger, the survivor area size will increase, and thus the size of the data to copy will increase as well. Given such characteristics, it is good to set a suitable new area size by referring to the NewRatio of HotSpot JVM by OS.

**Table 2: NewRatio by OS and option.**
OS and option	Default -XX:NewRatio
Sparc -server	2
Sparc -client	8
x86 -server	8
x86 -client	12

If the NewRatio is specified, 1/(NewRatio +1) of the entire heap size becomes the new area size. You will find the NewRatio of Sparc -server is very small. This is because the Sparc system was used for more high-end use than x86 when default values were specified. Now it is common to use the x86 server and its performance has also been improved. Thus it is better to specify 2 or 3, which is the value similar to that of the Sparc -server.

You can also specify NewSize and MaxNewSize instead of NewRatio. The new area is created as much as the value specified for NewSize and the size increments as much as the value specified for MaxNewSize. The Eden or Survivor area also increases according to the (specified or default) ratio. As you specify the same size for -Xs and -Xmx, it is a very good choice to specify the same size for MaxSize and MaxNewSize.

If you have specified both NewRatio and NewSize, you should use the bigger one. Therefore, when a heap has been created, you can express the initial New area size as follows:

min(MaxNewSize, max(NewSize, heap/(NewRatio+1)))

However, it is impossible to determine the appropriate entire heap size and New area size in a single attempt. Based on my experience running Web server applications at NHN, I recommend to run Java applications with the following JVM options. After monitoring the performance of the application with these options, you can use a more suitable GC algorithm or options.

**Table 3: Recommended JVM options.**
Type	Option
Operation mode	`-sever`
Entire heap size	Specify the same value for `-Xms` and `-Xmx`.
New area size	`-XX:NewRatio`: value of 2 to 4
New area size	`-XX:NewSize=?` `–XX:MaxNewSize=?`. Also good to specify `NewSize` instead of `NewRatio`.
Perm size	`-XX:PermSize=256 m` `-XX:MaxPermSize=256 m`. Specify the value to an extent not to cause any trouble in the operation because it does not affect the performance.
GC log	`-Xloggc:$CATALINA_BASE/logs/gc.log` `-XX:+PrintGCDetails` `-XX:+PrintGCDateStamps`. Leaving a GC log does not particularly affect the performance of Java applications. You are recommended to leave a GC log as much as possible.
GC algorithm	`-XX:+UseParNewGC` `-XX:+CMSParallelRemarkEnabled` `-XX:+UseConcMarkSweepGC` `-XX:CMSInitiatingOccupancyFraction=75`.This is only a generally recommendable configuration. Other choices could be better depending on the characteristics of the application.
Creating a heap dump when an OOM error occurs	`-XX:+HeapDumpOnOutOfMemoryError` `-XX:HeapDumpPath=$CATALINA_BASE/logs`
Actions after an OOM occurs	`-XX:OnOutOfMemoryError=$CATALINA_HOME/bin/stop.sh` or `-XX:OnOutOfMemoryError=$CATALINA_HOME/bin/restart.sh`. After leaving a heap dump, take a proper operation according to a management policy.

Measuring the Performance of Applications

The information to acquire to grasp the performance of an application is as follows:

TPS (OPS): The information required to understand the performance of an application conceptually.
Request Per Second (RPS): Strictly speaking, RPS is different from responsiveness, but you can understand it as responsiveness. Through RPS, you can check the time it takes for the user to see the result.
RPS Standard Deviation: It is necessary to induce even RPS if possible. If a deviation occurs, you need to check GC tuning or interworking systems.

To obtain a more accurate performance result, you should measure it after warming up the application sufficiently. This is because byte code is expected to be compiled by HotSpot JIT. In general, you can measure actual performance values after applying load to a certain feature for at least 10 minutes by using nGrinder load testing tool.

Tuning in Earnest

You don't need to tune the performance of an application if the result of the execution of nGrinder meets the expectation. If the performance does not meet the expectation, you need to carry out tuning to resolve problems. Now you will see the approach by case.

In the event the Stop-the-World takes long

Long stop-the-world time could result from inappropriate GC options or incorrect implementation. You can decide the cause according to the result of a profiler or a heap dump. This means you can judge the cause after checking the type and number of objects of a heap. If you find many unnecessary objects, you had better modify source codes. If you find no particular problem in the process of creating objects, you had better simply change GC options.

To adjust GC options appropriately, you need to have GC log secured for a sufficient period of time. You need to understand in which situation the stop-the-world takes a long time. For more information on the selection of appropriate GC options, read my colleague's blog about How to Monitor Java Garbage Collection.

In the event CPU usage rate is low

When blocking time occurs, both TPS and CPU usage rate will decrease. This might result from the problem of interworking systems or concurrency. To analyze this, you can use an analysis on the result of thread dump or a profiler. For more information on thread dump analysis, read How to Analyze Java Thread Dumps.

You can conduct a very accurate lock analysis by using a commercial profiler. In most cases, however, you can obtain a satisfactory result with only the CPU analyzer in jvisualvm.

In the event CPU usage rate is high

If TPS is low but CPU usage rate is high, this is likely to result from inefficient implementation. In this case, you should find out the location of bottlenecks by using a profiler. You can analyze this by using jvisuavm, TPTP of Eclipse or JProbe.

Approach for Tuning

You are advised to use the following approach to tune applications.

First, you should check whether performance tuning is necessary. The process of performance measuring is not easy work. You are also not guaranteed to obtain a satisfactory result all the time. Therefore, if the application already meets its target performance, you don't need to invest additionally in performance.

The problem lies in only a single place. All you have to do is to fix it. The Pareto principle applies to performance tuning as well. This does not mean to emphasize that the low performance of a certain feature results necessarily from a single problem. Rather, this emphasizes that we should focus on one factor that has the biggest influence on the performance when approaching performance tuning. Thus, you could handle another problem after fixing the most important one. You are advised to try to fix just one problem at a time.

You should consider the balloon effect. You should decide what to give up to get something. You can improve responsiveness by applying cache but if the cache size increases, the time it takes to carry out a full GC will increase as well. In general, if you want a small amount of memory usage, throughput or responsiveness could be deteriorated. Thus, you need to consider what is most important and what is less important.

So far, you have read the method for Java application performance tuning. To introduce a concrete procedure for performance measurement, I had to omit some details. Nevertheless, I think this could satisfy most of the cases for tuning Java web server applications.

Good luck with performance tuning!

By Se Hoon Park, Senior Software Engineer at Web Platform Development Lab, NHN Corporation.

All about Two-Phase Locking and a little bit MVCC

2012-12-10T23:18:29-08:00

In this blog I will describe the concurrency control methods implemented in database management systems, and the differences between them. I will also explain about what locking technique is used in CUBRID RDBMS, about locking modes and their compatibility, and finally, the deadlocks and the solution for them.

Overview

When multiple transactions, which change the data, are executed simultaneously, it is required to control the order of processing these transactions to satisfy the ACID (Atomicity, Consistency, Integrity, Durability) property of the database. Executing multiple transactions simultaneously should lead to the same result as executing each transaction independently, in other words, one transaction should not be affected by another transaction.

If different data is changed for each transaction, no interference between transactions is made, so there is no issue. However, if the same data is simultaneously changed by multiple transactions, the order of processing each transaction should be controlled.

Types of Concurrency Control

For example, the T1 transaction changes the A record from 1 to 2 and then changes the B record, the T2 transaction can simultaneously change the A record, too. Let's assume that the T2 transaction changes the A record from 2 to 4 by adding +2. If two transactions are successfully terminated, there is no issue. But it is important that all transactions can be rolled back. If the T1 transaction is rolled back, the value of the A record should be returned to 1, i.e. the value before the T1 transaction was executed. This is to satisfy the ACID property of the database. However, the T2 transaction has already changed the A record value to 3. So, it is impossible to return the A record to 1 regardless of the situation. In this case, there can be two options.

Two-phase locking (2PL)

The first one is when the T2 transaction tries to change the A record, it knows that the T1 transaction has already changed the A record and waits until the T1 transaction is completed because the T2 transaction cannot know whether the T1 transaction will be committed or rolled back. This method is called Two-phase locking (2PL).
Multi-version concurrency control (MVCC)

The other one is to allow each of them, T1 and T2 transactions, to have their own changed versions. Even when the T1 transaction has changed the A record from 1 to 2, the T1 transaction leaves the original value 1 as it is and writes that the T1 transaction version of the A record is 2. Then, the following T2 transaction changes the A record from 1 to 3, not from 2 to 4, and writes that the T2 transaction version of the A record is 3.

When the T1 transaction is rolled back, it does not matter if the 2, the T1 transaction version, is not applied to the A record. After that, if the T2 transaction is committed, the 3, the T2 transaction version, will be applied to the A record. If the T1 transaction is committed prior to the T2 transaction, the A record is changed to 2, and then to 3 at the time of committing the T2 transaction. The final database status is identical to the status of executing each transaction independently, without any impact on other transactions. Therefore, it satisfies the ACID property. This method is called Multi-version concurrency control (MVCC).

CUBRID has implemented 2PL method as well as DB2 and SQL Server, while Oracle, InnoDB and PostgreSQL have implemented MVCC.

Two-phase locking in CUBRID

The 2PL adopted by CUBRID uses locks to ensure the consistency between transactions that change the identical data. As the "lock" literally means, the locking is executed through two phases:

expanding phase (acquiring)
shrinking phase (releasing)

More accurately, all transactions should acquire lock for the data to be accessed and the acquired locks are released only when the transaction is terminated. After a transaction has acquired the lock for a certain data (regardless of the lock type, S_LOCK for read, stands for Shared Lock, or X_LOCK for write, stands for Exclusive Lock), when another transaction tries to acquire a new lock for the data, the new lock is allowed or pended depending on the lock compatibility rule. Therefore, success or failure of the prior transaction does not have impact on the following transactions, so the data consistency is maintained.

Lock Manager in CUBRID

Thus, the key point of 2PL, adopted by CUBRID, is that the lock must be processed through two phases: expanding phase and shrinking phase. Then, [Figure 1] release all locks, acquired while executing a transaction, only after the transaction ends (commit or rollback).

Figure 1: Two-Phase Locking.

2PL concurrency control method naturally controls access to the identical data from transactions by making all transactions observe the 2PL protocol. The following Figure 2 below shows an example of three transactions using 2PL: Transaction 1 executes B=B+A operation, Transaction 2 executes C=A+B operation, and Transaction 3 executes Print C operation. Since all three transactions are accessing the data A, B and C, the concurrency control is required. In this case, each transaction is executed according to the 2PL protocol so that there is no data conflict.

Figure 2: Concurrency Control by using 2PL.

Lock modes

To understand the concurrency control of multiple transactions more deeply, let's discuss about lock modes, lock conversion and transaction isolation level. In the above figure, you can see that S-lock, Shared Lock, for A was first acquired by Transaction 1, but it is also acquired by Transaction 2, too. On the contrary, the transaction which requested X-lock is blocked until S-lock is released. In this matter, a variety of lock modes are used to minimize conflicts by lockers. Major types of locks utilized in DBMSs are.

Shared (S) Lock: Used for read operation. It is generally set on the target record when SELECT statement is executed. It blocks a transaction from changing data which was already read by other transactions.
Exclusive (X) Lock: Used for write-operations such as INSERT, UPDATE, DELETE. It blocks one data from being changed by multiple transactions.
Update (U) Lock: Used to define that the target resource will be changed. It is used to minimize deadlock which may occur when multiple transactions are executing both read and write.
Intent Shared (IS) Lock: Set on the upper resource (e.g. tables) to set the S-lock on some lower resources (e.g. records or pages). It is to prevent other transactions from setting X-lock on the upper resource. Intent lock will soon be described.
Intent Exclusive (IX) Lock: Set on the upper resource to set X-lock on some lower resources.
Shared with Intent Exclusive (SIX) Lock: Set on the upper resource to set S-lock and X-lock on some lower resources.

Lock mode compatibility

Among the lock modes above, intent locks are used to improve the transaction concurrency and to prevent deadlock between the upper resources and the lower resources. For example, when Transaction A tries to read Record R on Table T, it sets IS_LOCK on Table T before setting S_LOCK on Record R. Then, Transaction B is prevented from setting X_LOCK on Table T to change the structure of Table T. If Transaction A has not set IS_LOCK on Table T, Transaction B would change the structure of Table T. Then, Transaction A would perform a wrong read operation. This way Transaction B has no need to check all records in Table T to check whether there is any lock set by other transactions for setting X_LOCK on Table T. The following lock mode compatibility table will clearly show the effect of intent locks:

**Table 1: The lock mode compatibility table of CUBRID.**
		Current Lock Mode
		NULL	IS	NS	S	IX	SIX	U	NX	X
Newly-requested Lock Mode	NULL	True	True	True	True	True	True	True	True	True
	IS	True	True	N/A	True	True	True	N/A	N/A	False
	NS	True	N/A	True	True	N/A	N/A	False	True	False
	S	True	True	True	True	False	False	False	False	False
	IX	True	True	N/A	False	True	False	N/A	N/A	False
	SIX	True	True	N/A	False	False	False	N/A	N/A	False
	U	True	N/A	True	True	N/A	N/A	False	False	False
	NX	True	N/A	True	False	N/A	N/A	False	False	False
	X	True	False	False	False	False	False	False	False	False

From the lock mode compatibility table, you can see that X_LOCK cannot be set on a table if IS_LOCK is set on the table. And only IS_LOCK can be compatible with SIX_LOCK. This means that SIX_LOCK intends to set S_LOCK and X_LOCK on the record and it will not allow any lock but IS_LOCK for S_LOCK on other non-conflicting records. From the table, you can see that IX_LOCK and IX_LOCK can be compatible with each other. IX_LOCK intends to set X_LOCK for some records. So, the compatibility is available. If there are two transactions that try to change an identical record, IX_LOCK for the table is allowed. However, there is no problem in concurrency control since only the transaction that has acquired X_LOCK for the record first can change the record (X_LOCK and X_LOCK are not compatible).

The lock mode compatibility table is expressed as a global variable lock_Comp[][] in the lock_table.c file in CUBRID source code. Among CUBRID sources, most codes related to lock modes are implemented in lock_manager.c file. To set lock on a data object, the lock_object() function is used which receives three parameters: the OID of an object where the lock mode will be set, the OID of the class where the object belongs, and the desired lock mode. In the source code of the function, you can see that the function is executed in several ways based on the target of the lock mode, the lock mode for an instance object or for a class object.

Take note of this: in CUBRID, a class object is also an object. Keep it in mind that a class object has an OID and all class objects are the instances of a root class, so it uses ROOTOID, the OID of the root object, as its class OID.

From the code, you can see that the required intent lock is set on a class object when a lock mode is required for an instance object. And there is a concept of lock waiting time in the lock mode request. To retrieve the lock timeout value set on the current transaction, the logtb_find_wait_secs() function is called. CUBRID supports the SET TRANSACTION LOCK TIMEOUT SQL command and the setLockTimeout() method in JDBC. The command is to specify the lock timeout of the current transaction. Lock waiting time means the time for a transaction, which has made a request for lock mode, to wait when a lock mode is set on an object by a transaction and the requested lock is not compatible with the already-set lock mode. As you have seen before, the 2PL concurrency control method does not allow lock from other transactions until the existing lock is released. For the following two reasons, lock timeout should be set by a transaction:

When a user does not want to wait too long because of the lock mode.
To lower the frequency of deadlock.

Deadlocks

A deadlock occurs when two or more transactions request resources locked by each of them, so all transactions cannot be progressed. Figure 8 below shows an example of a deadlock.

Figure 2: Transaction Deadlock.

First, Transaction 1 executes UPDATE participant SET gold=10 WHERE host_year=2004 AND nation_code=’KOR’ statement and sets X_LOCK on the ‘KOR’ record. Transaction 2 sets X_LOCK on the ‘JPN’ record. Transaction 3 sets X_LOCK on the ‘CHN’ record.

After that, Transaction 1 requests X_LOCK on the ‘JPN’ record for executing UPDATE for that record. However, the ‘JPN’ record is already locked with X_LOCK by Transaction 2. So, Transaction 1 should wait until Transaction 2 ends. Based on the 2PL protocol, the X_LOCK is released when the transaction ends. Transaction 2 requests X_LOCK on the ‘CHN’record and waits for Transaction 3. Finally, Transaction 3 waits for Transaction 1 to acquire the 'KOR' record of Transaction 1 as it has X_LOCK on the ‘CHN’ record. As a result,Transaction 1 waits for Transaction 2 to end, Transaction 2 waits for Transaction 3 to end, and Transaction 3 waits for Transaction 1 to end. So, no transaction can be progressed. This is called a deadlock.

Most DBMSs which use the 2PL method, including CUBRID, use the deadlock detection method to solve the deadlock problem. It periodically checks whether the cycle illustrated in the above figure occurs by drawing a Lock Wait Graph for the transactions being executed.

In CUBRID, the thread for detecting deadlock checks the Lock Wait Graph every second. When a deadlock is detected, one transaction among the transactions is randomly selected and aborted by force. This is called unilateral abort. When a transaction is selected as a victim to be sacrificed to solve the deadlock and unilaterally aborted, the corresponding SQL statement returns an error code. The error message is "The transaction has timed out due to deadlock while waiting for X_LOCK for an object. It waited until User 2 ended.” When an error is returned and the application aborts the transaction, the locks of the transaction are released and other transactions can be continuously processed.

To see how the deadlock is detected, see the lock_detect_local_deadlock() function in the source code. This function is called with the intervals (in seconds) specified by the PRM_LK_RUN_DEADLOCK_INTERVAL variable (the deadlock_detection_interval_in_secs parameter in cubrid.conf file) on the background thread which executes thread_deadlock_detect_thread().

Even if a deadlock does not occur, when the execution time of a transaction is too long, other transactions should wait for too long as well. For a certain application, it is wiser to give up rather than wait. In particular, when a web server has called DB tasks and the wait time is too long, all threads of the web server are used to process the DB, so they cannot be used to process external HTTP requests any more, causing service failures. Therefore, for a web application, the threads should be returned without waiting an unlimited amount of time for DB processing even if an error occurs. Two methods are used for that:

One is lock timeout supported by CUBRID.
The other is query cancel. JDBC is defined with an API which can cancel the SQL statement being executed.

The key data structure of the lock manager is defined in the lock_manager.c file.

typedef struct lk_entry LK_ENTRY;
struct lk_entry
{
#if defined(SERVER_MODE)
  struct lk_res *res_head;      /* back to resource entry           */
  THREAD_ENTRY *thrd_entry;     /* thread entry pointer             */
  int tran_index;               /* transaction table index          */
  LOCK granted_mode;            /* granted lock mode                */
  LOCK blocked_mode;            /* blocked lock mode                */
  int count;                    /* number of lock requests          */
  struct lk_entry *next;        /* next entry                       */
  struct lk_entry *tran_next;   /* list of locks that trans. holds  */
  struct lk_entry *class_entry; /* ptr. to class lk_entry           */
  LK_ACQUISITION_HISTORY *history;      /* lock acquisition history         */
  LK_ACQUISITION_HISTORY *recent;       /* last node of history list        */
  int ngranules;                /* number of finer granules         */
  int mlk_count;                /* number of instant lock requests  */
  unsigned char scanid_bitset[1];       /* PRM_LK_MAX_SCANID_BIT/8];       */
#else                           /* not SERVER_MODE */
  int dummy;
#endif                          /* not SERVER_MODE */
};

typedef struct lk_res LK_RES;
struct lk_res
{
  MUTEX_T res_mutex;            /* resource mutex */
  LOCK_RESOURCE_TYPE type;      /* type of resource: class,instance */
  OID oid;
  OID class_oid;
  LOCK total_holders_mode;      /* total mode of the holders */
  LOCK total_waiters_mode;      /* total mode of the waiters */
  LK_ENTRY *holder;             /* lock holder list */
  LK_ENTRY *waiter;             /* lock waiter list */
  LK_ENTRY *non2pl;             /* non2pl list */
  LK_RES *hash_next;            /* for hash chain */
};

From the file, the lk_Gl global variable of LK_GLOBAL_DATA type is the core. The LK_ENTRY structure stands for the lock itself. For example, when the Transaction T1 has requested a lock, one LK_ENTRY is created. LK_RES is a structure that shows to which resource the lock belongs. In CUBRID, all resources are objects (instance objects and class objects), so they are shaped as OIDs. In the LK_RES structure, you can see the list of holders with LK_ENTRY type and the list of waiters. The list of holders is a list of transactions that hold the lock for the resource now. For example, when Transaction T1 and Transaction T2 have acquired S_LOCK for the data record with OID1, LK_ENTRY that corresponds to the S_LOCK of T1 and T2 will be registered in the list of holders. When Transaction T3 requests X_LOCK on the OID1 record, T3 should wait because of the existing S_LOCK. So, the LK_ENTRY corresponding to X_LOCK of T3 will be registered to the list of waiters. Which lock is held by which transaction is maintained in the tran_lock_table variable which has the LK_TRAN_LOCK structure as a table.

The Wait For Graph for detecting a deadlock is expressed as TWFG_node and TWFG_edge of the LK_WFG_NODE structure and the LK_WFG_EDGE structure. The lock_detect_local_deadlock() function creates a Wait For Graph and detects whether there is a cycle on the graph. When a cycle is detected, the lock_select_deadlock_victim() function selects a victim transaction to be sacrificed for solving the deadlock. For reference, transactions are continuously executed while a Wait For Graph is drawn up and checked, the information of the ended transaction is removed from the graph. The victim transaction is selected based on the following criteria:

If a transaction is not a holder, it cannot be a victim.
When a transaction is in the commit phase or the rollback phase, it cannot be selected as a victim.
Select a transaction of which lock timeout is not set to -1 (unlimited waiting) first.
Select the latest transaction rather than the older one. (The transaction ID is an incremental number. A transaction with smaller transaction number is the older one.)

Conclusion

This concludes the talk about Two-Phase Locking in CUBRID. I briefly covered the types of concurrency control, the difference between 2PL and MVCC, about what locking technique is used in CUBRID RDBMS, about locking modes and their compatibility, and finally, the deadlocks and the solution for them.

In this article I have mentioned about OID (Object Identifiers) which are used to identify instance objects as well as class objects. In the next article I will continue this talk and explain what Object, Class, and OID are.

CUBRID Blog

PostgreSQL at a glance

Why You Should Know about PostgreSQL

History

Functionalities and Limitations

Roadmap

Internal Structure

Development Process

Products in Competition

Present Status and Trend

Conclusion

My impressions after giving talks at RIT++ and Percona conferences

RIT++

Percona

NoSQL Benchmarking

Why NoSQL?

Benchmarking Tests using YCSB

Cassandra and HBase Architecture

Cassandra

HBase

MongoDB Architecture

MongoDB Structure

RDBMS Architecture

Summary

Comprehensive Overview of Top 14 Content Management Systems

# 1: Drupal

# 2: Joomla

#3: modx

#4: WordPress

#5: DotNetNuke

#6: Umbraco

#7: Liferay

#8: Typo3

#9: CMS Made Simple

#10: Movable Type

#11: Plone

#12: eZ Publish

#13: Concrete 5

#14: Alfresco

See also

Our Experience of Creating Large Scale Log Search System Using ElasticSearch

Features of ElasticSearch

JSON-based Schemaless Storage

Multitenancy

Scalability and Flexibility

Distributed Storage

Installing

Quick Start

Installing Plugins

Main Configurations

Using REST API

Creating Documents and Indexes

Deleting a Document

Getting a Document

Search

Search API by Using URI Request

Search API by Using Request Body

Mapping

Put Mapping API

Get Mapping API

Delete Mapping API

How to Optimize Performance

Memory and the Number of Open Files

Index Optimization

Shards and Replicas

Configuring Cluster Topologies

Configuring Routing

Conclusion

References

Announcing CUBRID ALL-IN-ONE Windows Downloader

CUBRID Node.js Driver 1.1 is now available at NPM

What's New

Testing

What's next

Inside Vert.x. Comparison with Node.js.

Philosophy of Vert.x

What is the relationship and defference between Netty and Vert.x?

Comparison of Performance with Node.js

Vert.x Terminology

Verticle