Thus, in this article I will explain how JVM works, its structure, how it executes Java bytecode, the order of execution, examples of common mistakes and their solutions, as well as the new features in Java SE 7 Edition.

Virtual Machine

The JRE is composed of the Java API and the JVM. The role of the JVM is to read the Java application through the Class Loader and execute it along with the Java API.

A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. Originally, Java was designed to run based on a virtual machine separated from a physical machine for implementing WORA (Write Once Run Anywhere), although this goal has been mostly forgotten. Therefore, the JVM runs on all kinds of hardware to execute the Java Bytecode without changing the Java execution code.

The features of JVM are as follows:

• Stack-based virtual machine: The most popular computer architectures such as Intel x86 Architecture and ARM Architecture run based on a register. However, JVM runs based on a stack.
• Symbolic reference: All types (class and interface) except for primitive data types are referred to through symbolic reference, instead of through explicit memory address-based reference. 
• Garbage collection: A class instance is explicitly created by the user code and automatically destroyed by garbage collection.
• Guarantees platform independence by clearly defining the primitive data type: A traditional language such as C/C++ has different int type size according to the platform. The JVM clearly defines the primitive data type to maintain its compatibility and guarantee platform independence.
• Network byte order: The Java class file uses the network byte order. To maintain platform independence between the little endian used by Intel x86 Architecture and the big endian used by the RISC Series Architecture, a fixed byte order must be kept. Therefore, JVM uses the network byte order, which is used for network transfer. The network byte order is the big endian.

Sun Microsystems developed Java. However, any vendor can develop and provide a JVM by following the Java Virtual Machine Specification. For this reason, there are various JVMs, including Oracle Hotspot JVM and IBM JVM. The Dalvik VM in Google's Android operating system is a kind of JVM, though it does not follow the Java Virtual Machine Specification. Unlike Java VMs, which are stack machines, the Dalvik VM is a register-based architecture. Java bytecode is also converted into an register-based instruction set used by the Dalvik VM.

Java bytecode

To implement WORA, the JVM uses Java bytecode, a middle-language between Java (user language) and the machine language. This Java bytecode is the smallest unit that deploys the Java code.

Before explaining the Java bytecode, let's take a look at it. This case is a summary of a real example that has occurred in development process.

Symptom

An application that had been running successfully no longer runs. Moreover, returns the following error after the library has been updated.

The application code is as follows, and no changes to it have been made.

The updated library source code and the original source code are as follows.

In short, the addUser() method which has no return value has been changed to a method that returns the User class instance. However, the application code has not been changed, since it does not use the return value of the addUser() method.

At first glance, the com.nhn.user.UserAdmin.addUser() method seems to still exist, but if so, why does NoSuchMethodError occur?

Reasons

The reason is that the application code has not been compiled to a new library. In other words, the application code seems to invoke methods regardless of the return value. However, the compiled class file indicates the method that has a return value.

You will see this through the following error message.

NoSuchMethodError has occurred since the "com.nhn.user.UserAdmin.addUser(Ljava/lang/String;)V" method could not be found. Take a look at "Ljava/lang/String;" and the last "V". In the expression of Java Bytecode, "L<classname>;" is the class instance. This means that the addUser() method returns one java/lang/String object as a parameter. In the library of this case, the parameter has not been changed, so it is normal. The last "V" of the message stands for the return value of the method. In the expression of Java Bytecode, "V" means that it has no return value. In short, the error message means that one java.lang.String object has been returned as a parameter and the com.nhn.user.UserAdmin.addUser method without any return value has not been found.

Since the application code has been compiled to the previous library, the class file defined that a method that returns "V" should be invoked. However, in the changed library, the method that returned "V" did not exist, but the method that returned "Lcom/nhn/user/User;" has been added. Therefore, a NoSuchMethodError occurred.

Note

The error has occurred since the developer did not compile a new library again. However, in this case, the library provider is mostly responsible for that. There was no return value of the method as public, but it later has been changed to return the user class instance. This is an obvious method signature change. This means that the backward compatibility of the library has been broken. Therefore, the library provider must have reported to the users that the method has been changed.

Let's go back to the Java Bytecode. Java Bytecode is the essential element of JVM. The JVM is an emulator that emulates the Java Bytecode. Java compiler does not directly convert high-level language such as C/C++ to the machine language (direct CPU instruction); it converts the Java language that the developer understands to the Java Bytecode that the JVM understands. Since Java bytecode has no platform-dependent code, it is executable on the hardware where the JVM (accurately, the JRE of the same profile) has been installed, even when the CPU or OS is different (a class file developed and compiled on the Windows PC can be executed on the Linux machine without additional change.) The size of the compiled code is almost identical to the size of the source code, making it easy to transfer and execute the compiled code via  the network.

The class file itself is a binary file that cannot be understood by a human. To manage this file, JVM vendors provide javap, the disassembler. The result of using javap is called Java assembly. In the above case, the Java assembly below is obtained by disassembling the UserService.add() method of the application code with the javap -c option.

In this Java assembly, the addUser() method is invoked by the fourth row, "5: invokevirtual #23;". This means that the method corresponding to the 23rd index should be invoked. The method of the 23rd index is annotated by the javap program. The invokevirtual is the OpCode (operation code) of the most basic command that invokes a method in the Java Bytecode. For reference, there are four OpCodes that invoke a method in the Java Bytecode: invokeinterface, invokespecial, invokestatic, and invokevirtual. The meaning of each OpCode is as follows.

• invokeinterface: Invokes an interface method
• invokespecial: Invokes an initializer, private method, or superclass method
• invokestatic: Invokes static methods
• invokevirtual: Invokes instance methods

The instruction set of Java Bytecode consists of OpCode and Operand. The OpCode such as invokevirtual requires a 2-byte Operand.

By compiling the application code above with the updated library and then disassembling it, the following result will be obtained.

You can see that the method corresponding to the 23rd has been converted to the method that returns "Lcom/nhn/user/User;".

In the disassembled result above, what does the number in front of the code mean?

It is the byte number. Perhaps this is the reason why the code executed by the JVM is called Java "Byte"code. In short, the bytecode instruction OpCodes such as aload_0, getfield, and invokevirtual are expressed as a 1-byte byte number. (aload_0 = 0x2a, getfield = 0xb4, invokevirtual = 0xb6) Therefore, the maximum number of Java Bytecode instruction OpCodes is 256.

OpCodes such as aload_0 and aload_1 do not need any Operand. Therefore, the next byte of aload_0 is the OpCode of the next instruction. However, getfield and invokevirtual need the 2-byte Operand. Therefore, the next instruction of getfield on the first byte is written on the fourth byte by skipping two bytes. The bytecode shown through Hex Editor is as follows.

In the Java Bytecode, the class instance is expressed as "L;" and void is expressed as "V". In this way, other types have their own expressions. The table below summarizes the expressions.

Table 1: Type Expression in Java Bytecode

The table below shows examples of Java Bytecode expressions.

Table 2: Examples of Java Bytecode Expressions

For more details, see "4.3 Descriptors" in "The Java Virtual Machine Specification, Second Edition". For various Java Bytecode instruction sets, see "6. The Java Virtual Machine Instruction Set" in "The Java Virtual Machine Specification, Second Edition".

Class File Format

Before explaining the Java class file format, let's review an example that frequently occurs in Java Web applications.

Symptom

When writing and executing JSP on Tomcat, the JSP did not run, and the following error occurred.

Reasons

The error message above varies slightly depending on the Web application server, however, one thing is the same; it is because of the 65535 byte limit. The 65535 byte limit is one of the JVM limitations, and stipulates that the size of one method cannot be more than 65535 bytes.

I will present the meaning of the 65535 byte limit and why it has been set in more detailed manner.

The branch/jump instructions used in the Java Bytecode are "goto" and "jsr".

Both of the two receive 2-byte signed branch offset as their Operand so that they can be expanded to the 65535th index at a maximum. However, to support more sufficient branch, Java Bytecode prepares "goto_w" and "jsr_w" that receive 4-byte signed branch offset.

With the two, branch is available with an index exceeding 65535. Therefore, the 65535 byte limit of Java method may be overcome. However, due to various other limits of the Java class file format, the Java method still cannot exceed 65535 bytes. To view other limits, I will simply explain the class file format.

The outline of a Java class file is as follows: 

The above is included in "4.1. The ClassFile Structure" of "The Java Virtual Machine Specification, Second Edition".

The first 16 bytes of the UserService.class file disassembled earlier are shown as follows in the Hex Editor.

ca fe ba be 00 00 00 32 00 28 07 00 02 01 00 1b

With this value, see the class file format.

• magic: The first 4 bytes of the class file are the magic number. This is a pre-specified value to distinguish the Java class file. As shown in the Hex Editor above, the value is always 0xCAFEBABE. In short, when the first 4 bytes of a file is 0xCAFEBABE, it can be regarded as the Java class file. This is a kind of "witty" magic number related to the name "Java".
• minor_version, major_version: The next 4 bytes indicate the class version. As the UserService.class file is 0x00000032, the class version is 50.0. The version of a class file compiled by JDK 1.6 is 50.0, and the version of a class file compiled by JDK 1.5 is 49.0. The JVM must maintain backward compatibility with class files compiled in a lower version than itself. On the other hand, when a upper-version class file is executed in the lower-version JVM, java.lang.UnsupportedClassVersionError occurs.
• constant_pool_count, constant_pool[]: Next to the version, the class-type constant pool information is described. This is the information included in the Runtime Constant Pool area, which will be explained later. While loading the class file, the JVM includes the constant_pool information in the Runtime Constant Pool area of the method area. As the constant_pool_count of the UserService.class file is 0x0028, you can see that the constant_pool has (40-1) indexes, 39 indexes.
• access_flags: This is the flag that shows the modifier information of a class; in other words, it shows public, final, abstract or whether or not to interface.
• this_class, super_class: The index in the constant_pool for the class corresponding to this and super, respectively.
• interfaces_count, interfaces[]: The index in the the constant_pool for the number of interfaces implemented by the class and each interface.
• fields_count, fields[]: The number of fields and the field information of the class. The field information includes the field name, type information, modifier, and index in the constant_pool.
• methods_count, methods[]: The number of methods in a class and the methods information of the class. The methods information includes the methods name, type and number of the parameters, return type, modifier, index in the constant_pool, execution code of the method, and exception information.
• attributes_count, attributes[]: The attribute_info structure has various attributes. For field_info or method_info, attribute_info is used.

The javap program briefly shows the class file format in a format that users can read. When UserService.class is analyzed using the "javap -verbose" option, the following contents are printed.

Due to a lack of space, I have extracted some parts from the entire printout. The entire printout shows you the various information included in the constant pool and the contents of each method.

The 65535 byte limit of method size is related to the contents of method_info struct. The method_info struct has Code, LineNumberTable, and LocalVariableTable attribute, as shown in the "javap -verbose" print shown above. All of the values corresponding to the length of LineNumberTable, LocalVariableTable, and exception_table included in the Code attribute are fixed at 2 bytes. Therefore, the method size cannot exceed the length of LineNumberTable, LocalVariableTable, and exception_table, and is limited to 65535 bytes.

Many people have complaints about the method size limit, and the JVM specifications state that 'it may be expandable later.’ However, no explicit move toward improvement has been made so far. Considering the characteristic of JVM specifications that loads almost same contents in the class file to the method area, it will be significantly difficult to expand the method size while maintining backward compatibility.

What will happen if an incorrect class file is created because of a Java compiler error? Or, what if due to errors in network transfer or file copy process, a class file can be broken?

To prepare for such cases, the Java class loader is verified through a very strict and tight process. The JVM specifications explicitly detail the process.

Note

How can we verify that the JVM successfully executes the class file verification process? How can we verify that various JVMs from various JVM vendors satisfy the JVM specifications? For verification, Oracle provides a test tool, TCK (Technology Compatibility Kit). The TCK verifies a JVM specification by executing ten thousands of tests, including a many incorrect class files in various ways. After passing the TCK, the JVM can be called a JVM. 

Like TCK, there is JCP (Java Community Process; http://jcp.org), which proposes new Java technical specifications as well as Java specifications. For the JCP, a specification document, reference implementation, and TCK for a proposed JSR (Java Specification Request) must be completed to complete JSR. Users who want to use new Java technology proposed as JSR should license the implementation from the RI provider, or directly implement it and test the implementation with TCK.

JVM Structure

The code written in Java is executed by following the process shown in the figure below. 

Figure 1: Java Code Execution Process.

A class loader loads the compiled Java Bytecode to the Runtime Data Areas, and the execution engine executes the Java Bytecode.

Class Loader

Java provides a dynamic load feature; it loads and links the class when it refers to a class for the first time at runtime, not compile time. JVM's class loader executes the dynamic load. The features of Java class loader are as follows:

• Hierarchical Structure: Class loaders in Java are organized into a hierarchy with a parent-child relationship. The Bootstrap Class Loader is the parent of all class loaders.
• Delegation mode: Based on the hierarchical structure, load is delegated between class loaders. When a class is loaded, the parent class loader is checked to determine whether or not the class is in the parent class loader. If the upper class loader has the class, the class is used. If not, the class loader requested for loading loads the class.
• Visibility limit: A child class loader can find the class in the parent class loader; however, a parent class loader can find the class in the child class loader.
• Unload is not allowed: A class loader can load a class but cannot unload it. Instead of unloading, the current class loader can be deleted, and a new class loader can be created.

Each class loader has its namespace that stores the loaded classes. When a class loader loads a class, it searches the class based on FQCN (Fully Qualified Class Name) stored in the namespace to check whether or not the class has been already loaded. Even if the class has an identical FQCN but a different namespace, it is regarded as a different class. A different namespace means that the class has been loaded by another class loader.

The following figure illustrates the class loader delegation model.

Figure 2: Class Loader Delegation Model.

When a class loader is requested for class load, it checks whether or not the class exists in the class loader cache, the parent class loader, and itself, in the order listed. In short, it checks whether or not the class has been loaded in the class loader cache. If not, it checks the parent class loader. If the class is not found in the bootstrap class loader, the requested class loader searches for the class in the file system.

• Bootstrap class loader: This is created when running the JVM. It loads Java APIs, including object classes. Unlike other class loaders, it is implemented in native code instead of Java.
• Extension class loader: It loads the extension classes excluding the basic Java APIs. It also loads various security extension functions.
• System class loader: If the bootstrap class loader and the extension class loader load the JVM components, the system class loader loads the application classes. It loads the class in the $CLASSPATH specified by the user.
• User-defined class loader: This is a class loader that an application user directly creates on the code.

Frameworks such as Web application server (WAS) use it to make Web applications and enterprise applications run independently. In other words, this guarantees the independence of applications through class loader delegation model. Such a WAS class loader structure uses a hierarchical structure that is slightly different for each WAS vendor.

If a class loader finds an unloaded class, the class is loaded and linked by following the process illustrated below.

Figure 3: Class Load Stage.

Each stage is described as follows.

• Loading: A class is obtained from a file and loaded to the JVM memory.
• Verifying: Check whether or not the read class is configured as described in the Java Language Specification and JVM specifications. This is the most complicated test process of the class load processes, and takes the longest time. Most cases of the JVM TCK test cases are to test whether or not a verification error occurs by loading wrong classes.
• Preparing: Prepare a data structure that assigns the memory required by classes and indicates the fields, methods, and interfaces defined in the class.
• Resolving: Change all symbolic references in the constant pool of the class to direct references.
• Initializing: Initialize the class variables to proper values. Execute the static initializers and initialize the static fields to the configured values.

The JVM specification defines the tasks. However, it allows flexible application of the execution time.

Runtime Data Areas

Figure 4: Runtime Data Areas Configuration.

Runtime Data Areas are the memory areas assigned when the JVM program runs on the OS. The runtime data areas can be divided into 6 areas. Of the six, one PC Register, JVM Stack, and Native Method Stack are created for one thread. Heap, Method Area, and Runtime Constant Pool are shared by all threads.

• PC register: One PC (Program Counter) register exists for one thread, and is created when the thread starts. PC register has the address of a JVM instruction being executed now.
• JVM stack: One JVM stack exists for one thread, and is created when the thread starts. It is a stack that saves the struct (Stack Frame). The JVM just pushes or pops the stack frame to the JVM stack. If any exception occurs, each line of the stack trace shown as a method such as printStackTrace() expresses one stack frame.

Figure 5: JVM Stack Configuration.

− Stack frame: One stack frame is created whenever a method is executed in the JVM, and the stack frame is added to the JVM stack of the thread. When the method is ended, the stack frame is removed. Each stack frame has the reference for local variable array, Operand stack, and runtime constant pool of a class where the method being executed belongs. The size of local variable array and Operand stack is determined while compiling. Therefore, the size of stack frame is fixed according to the method.

− Local variable array: It has an index starting from 0. 0 is the reference of a class instance where the method belongs. From 1, the parameters sent to the method are saved. After the method parameters, the local variables of the method are saved.

− Operand stack: An actual workspace of a method. Each method exchanges data between the Operand stack and the local variable array, and pushes or pops other method invoke results. The necessary size of the Operand stack space can be determined during compiling. Therefore, the size of the Operand stack can also be determined during compiling.

• Native method stack: A stack for native code written in a language other than Java. In other words, it is a stack used to execute C/C++ codes invoked through JNI (Java Native Interface). According to the language, a C stack or C++ stack is created.
• Method area: The method area is shared by all threads, created when the JVM starts. It stores runtime constant pool, field and method information, static variable, and method bytecode for each of the classes and interfaces read by the JVM. The method area can be implemented in various formats by JVM vendor. Oracle Hotspot JVM calls it Permanent Area or Permanent Generation (PermGen). The garbage collection for the method area is optional for each JVM vendor.
• Runtime constant pool: An area that corresponds to the constant_pool table in the class file format. This area is included in the method area; however, it plays the most core role in JVM operation. Therefore, the JVM specification separately describes its importance. As well as the constant of each class and interface, it contains all references for methods and fields. In short, when a method or field is referred to, the JVM searches the actual address of the method or field on the memory by using the runtime constant pool.
• Heap: A space that stores instances or objects, and is a target of garbage collection. This space is most frequently mentioned when discussing issues such as JVM performance. JVM vendors can determine how to configure the heap or not to collect garbage.

Let's go back to the disassembled bytecode we discussed previously. 

Comparing the disassembled code and the assembly code of the x86 architecture that we sometimes see, the two have a similar format, OpCode; however, there is a difference in that Java Bytecode does not write register name, memory addressor, or offset on the Operand. As described before, the JVM uses stack. Therefore, it does not use register, unlike the x86 architecture that uses registers, and it uses index numbers such as 15 and 23 instead of memory addresses since it manages the memory by itself. The 15 and 23 are the indexes of the constant pool of the current class (here, UserService class). In short, the JVM creates a constant pool for each class, and the pool stores the reference of the actual target.

Each row of the disassembled code is interpreted as follows.

• aload_0: Add the #0 index of the local variable array to the Operand stack. The #0 index of the local variable array is always this, the reference for the current class instance.
• getfield #15: In the current class constant pool, add the #15 index to the Operand stack. UserAdmin admin field is added. Since the admin field is a class instance, a reference is added.
• aload_1: Add the #1 index of the local variable array to the Operand stack. From the #1 index of the local variable array, it is a method parameter. Therefore, the reference of String userName sent while invoking add() is added.
• invokevirtual #23: Invoke the method corresponding to the #23 index in the current class constant pool. At this time, the reference added by using getfield and the parameter added by using aload_1 are sent to the method to invoke. When the method invocation is completed, add the return value to the Operand stack.
• pop: Pop the return value of invoking by using invokevirtual from the Operand stack. You can see that the code compiled by the previous library has no return value. In short, the previous has no return value, so there was no need to pop the return value from the stack.
• return: Complete the method.

The following figure will help you understand the explanation.

Figure 6: Example of Java Bytecode Loaded on Runtime Data Areas.

For reference, in this method, no local variable array has been changed. So the figure above displays the changes in Operand stack only. However, in most cases, local variable array is also changed. Data transfer between the local variable array and the Operand stack is made by using a lot of load instructions (aload, iload) and store instructions (astore, istore). 

In this figure, we have checked the brief description of the runtime constant pool and the JVM stack. When the JVM runs, each class instance will be assigned to the heap, and class information including User, UserAdmin, UserService, and String will be stored in the method area.

Execution Engine

The bytecode that is assigned to the runtime data areas in the JVM via class loader is executed by the execution engine. The execution engine reads the Java Bytecode in the unit of instruction. It is like a CPU executing the machine command one by one. Each command of the bytecode consists of a 1-byte OpCode and additional Operand. The execution engine gets one OpCode and execute task with the Operand, and then executes the next OpCode.

But the Java Bytecode is written in a language that a human can understand, rather than in the language that the machine directly executes. Therefore, the execution engine must change the bytecode to the language that can be executed by the machine in the JVM. The bytecode can be changed to the suitable language in one of two ways.

• Interpreter: Reads, interprets and executes the bytecode instructions one by one. As it interprets and executes instructions one by one, it can quickly interpret one bytecode, but slowly executes the interpreted result. This is the disadvantage of the interpret language. The 'language' called Bytecode basically runs like an interpreter.
• JIT (Just-In-Time) compiler: The JIT compiler has been introduced to compensate for the disadvantages of the interpreter. The execution engine runs as an interpreter first, and at the appropriate time, the JIT compiler compiles the entire bytecode to change it to native code. After that, the execution engine no longer interprets the method, but directly executes using native code. Execution in native code is much faster than interpreting instructions one by one. The compiled code can be executed quickly since the native code is stored in the cache. 

However, it takes more time for JIT compiler to compile the code than for the interpreter to interpret the code one by one. Therefore, if the code is to be executed just once, it is better to interpret it instead of compiling. Therefore, the JVMs that use the JIT compiler internally check how frequently the method is executed and compile the method only when the frequency is higher than a certain level.

Figure 7: Java Compiler and JIT Compiler.

How the execution engine runs is not defined in the JVM specifications. Therefore, JVM vendors improve their execution engines using various techniques, and introduce various types of JIT compilers. 

Most JIT compilers run as shown in the figure below: 

Figure 8: JIT Compiler.

The JIT compiler converts the bytecode to an intermediate-level expression, IR (Intermediate Representation), to execute optimization, and then converts the expression to native code.

Oracle Hotspot VM uses a JIT compiler called Hotspot Compiler. It is called Hotspot because Hotspot Compiler searches the 'Hotspot' that requires compiling with the highest priority through profiling, and then it compiles the hotspot to native code. If the method that has the bytecode compiled is no longer frequently invoked, in other words, if the method is not the hotspot any more, the Hotspot VM removes the native code from the cache and runs in interpreter mode. The Hotspot VM is divided into the Server VM and the Client VM, and the two VMs use different JIT compilers.

Figure 9: Hotspot Client VM and Server VM.

The client VM and the server VM use an identical runtime; however, they use different JIT compilers, as shown in the above figure. The client VM and the server VM use an identical runtime, however, they use different JIT compilers as shown in the above figure. Advanced Dynamic Optimizing Compiler used by the server VM uses more complex and diverse performance optimization techniques.

IBM JVM has introduced AOT (Ahead-Of-Time) Compiler from IBM JDK 6 as well as the JIT compiler. This means that many JVMs share the native code compiled through the shared cache. In short, the code that has been already compiled through the AOT compiler can be used by another JVM without compiling. In addition, IBM JVM provides a fast way of execution by pre-compiling code to JXE (Java EXecutable) file format using the AOT compiler.

Most Java performance improvement is accomplished by improving the execution engine. As well as the JIT compiler, various optimization techniques are being introduced so the JVM performance can be continuously improved. The biggest difference between the initial JVM and the latest JVM is the execution engine.

Hotspot compiler has been introduced to Oracle Hotspot VM from version 1.3, and JIT compiler has been introduced to Dalvik VM from Android 2.2.

Note

The technique in which an intermediate language such as bytecode is introduced, the VM executes the bytecode, and the JIT compiler improves the performance of JVM is also commonly used in other languages that have introduced intermediate languages. For Microsoft's .Net, CLR (Common Language Runtime), a kind of VM, executes a kind of bytecode, called CIL (Common Intermediate Language). CLR provides the AOT compiler as well as the JIT compiler. Therefore, if source code is written in C# or VB.NET and compiled, the compiler creates CIL and the CIL is executed on the CLR with the JIT compiler. The CLR uses the garbage collection and runs as a stack machine like the JVM.

The Java Virtual Machine Specification, Java SE 7 Edition

On 28th July, 2011, Oracle released Java SE 7 and updated the JVM specifications to Java SE 7 version. After releasing "The Java Virtual Machine Specification, Second Edition" in 1999, it took 12 years for Oracle to release the updated version. The updated version includes various changes and modifications accumulated over 12 years, and describes more clear specifications. In addition, it reflects the contents included in "The Java Language Specification, Java SE 7 Edition" released with Java SE 7. The major changes can be summarized as follows:

• Generics introduced from Java SE 5.0, supporting variable argument method
• Bytecode verification process technique changed since Java SE 6
• Added invokedynamic instruction and related class file formats for supporting dynamic type languages
• Deleted the description of the concept of the Java language itself and referred reader to the Java language specifications
• Deleted the description on Java Thread and Lock, and transferred these to the Java language specifications

The biggest change of these is the addition of invokedynamic instruction. This means that a change was made in the JVM internal instruction sets, as the JVM started to support dynamic type languages of which type is not fixed, such as script languages, as well as Java language from Java SE 7. The OpCode 186 which had not been used previously has been assigned to the new instruction, invokedynamic, and new contents have been added to the class file format to support the invokedynamic.

The version of the class file created by the Java compiler of Java SE 7 is 51.0. The version of Java SE 6 is 50.0. Much of the class file format has been changed. Therefore, class files with version 51.0 cannot be executed in the Java SE 6 JVM. 

Despite these various changes, the 65535 byte limit of the Java method has not been removed. Unless the JVM class file format is innovatively changed, it may not be removed in the future.

For reference, Oracle Java SE 7 VM supports G1, the new garbage collection; however, it is limited to the Oracle JVM, so JVM itself does not limit any garbage collection type. Therefore, the JVM specifications do not describe that.

String in switch Statements

Java SE 7 adds various grammars and features. However, compared to the various changes in language of Java SE 7, there are not so many changes in the JVM. So, how can the new features of the Java SE 7 be implemented? We will see how String in switch Statements (a function to add a string to a switch() statement as a comparison) has been implemented in Java SE 7 by disassembling it.

For example, the following code has been written.

Since it is a new function of Java SE 7, it cannot be compiled using the Java compiler for Java SE 6 or lower versions. Compile it using the javac of Java SE 7. The following screen is the compiling result printed by using javap –c.

A significantly longer bytecode than the Java source code has been created. First, you can see that lookupswitch instruction has been used for switch() statement in Java bytecode. However, two lookupswitch instructions have been used, not the one lookupswitch instruction. When disassembling the case in which int has been added to switch() statement, only one lookupswitch instruction has been used. This means that the switch() statement has been divided into two statements to process the string. See the annotation of the #5, #39, and #53 byte instructions to see how the switch() statement has processed the string.

In the #5 and #8 byte, first, hashCode() method has been executed and switch(int) has been executed by using the result of executing hashCode() method. In the braces of the lookupswitch instruction, branch is made to the different location according to the hashCode result value. String "abc" is hashCode result value 96354, and is moved to #36 byte. String "123" is hashCode result value 48690, and is moved to #50 byte.

In the #36, #37, #39, and #42 bytes, you can see that the value of the str variable received as an argument is compared using the String "abc" and the equals() method. If the results are identical, '0' is inserted to the #3 index of the local variable array, and the string is moved to the #61 byte.

In this way, in the #50, #51, #53, and #56 bytes, you can see that the value of the str variable received as an argument is compared by using the String "123" and the equals() method. If the results are identical, '1' is inserted to the #3 index of the local variable array and the string is moved to the #61 byte.

In the #61 and #62 bytes, the value of the #3 index of the local variable array, i.e., '0', '1', or any other value, is lookupswitched and branched.

In other words, in Java code, the value of the str variable received as the switch() argument is compared using the hashCode() method and the equals() method. With the result int value, switch() is executed.

In this result, the compiled bytecode is not different from the previous JVM specifications. The new feature of Java SE 7, String in switch is processed by the Java compiler, not by the JVM itself. In this way, other new features of Java SE 7 will also be processed by the Java compiler.

Conclusion

I don't think that we need to review how Java has been developed to use Java well. So many Java developers develop great applications and libraries without understanding JVM deeply. However, if you understand JVM, you will understand Java more, and it will be helpful to solve the problems like the case we have reviewed here.

Besides the description mentioned here, the JVM has various features and technologies. The JVM specifications provide a flexible specification for JVM vendors to provide more advanced performance so that various technologies can be applied by the vendor. In particular, garbage collection is the technique used by most languages that provides usability similar to that of a VM, the latest and state-of-the-art technique in its performance. However, as this has been discussed in many more prominent studies, I did not explain it deeply in this article.

For Korean speakers, if you need more information on the internal structure of JVM, I recommend you to refer to "Java Performance Fundamental" (Hando Kim, Seoul, EXEM, 2009). The book is written in Korean so it is easy to read. I have referenced this book as well as the JVM specifications to write this article. For English speaking readers, there should be many books covering Java Performance topic.

By Se Hoon Park, Messaging Platform Development Team, NHN Corporation.

When there is an obstacle, or when a Java based Web application is running much slower than expected, we need to use thread dumps. If thread dumps feel like very complicated to you, this article may help you very much. Here I will explain what threads are in Java, their types, how they are created, how to manage them, how you can dump threads from a running application, and finally how you can analyze them and determine the bottleneck or blocking threads. This article is a result of long experience in Java application debugging.

Java and Thread

A web server uses tens to hundreds of threads to process a large number of concurrent users. If two or more threads utilize the same resources, a contention between the threads is inevitable, and sometimes deadlock occurs.

Thread contention is a status in which one thread is waiting for a lock, held by another thread, to be lifted. Different threads frequently access shared resources on a web application. For example, to record a log, the thread trying to record the log must obtain a lock and access the shared resources.

Deadlock is a special type of thread contention, in which two or more threads are waiting for the other threads to complete their tasks in order to complete their own tasks.

Different issues can arise from thread contention. To analyze such issues, you need to use the thread dump. A thread dump will give you the information on the exact status of each thread.

Background Information for Java Threads

Thread Synchronization

A thread can be processed with other threads at the same time. In order to ensure compatibility when multiple threads are trying to use shared resources, one thread at a time should be allowed to access the shared resources by using thread synchronization.

Thread synchronization on Java can be done using monitor. Every Java object has a single monitor. The monitor can be owned by only one thread. For a thread to own a monitor that is owned by a different thread, it needs to wait in the wait queue until the other thread releases its monitor.

Thread Status

In order to analyze a thread dump, you need to know the status of threads. The statuses of threads are stated on java.lang.Thread.State.

Figure 1: Thread Status.

• NEW: The thread is created but has not been processed yet.
• RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITING status due to the OS's resource distribution.)
• BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock.
• WAITING: The thread is waiting by using a wait, join or park method.
• TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.) 

Thread Types

Java threads can be divided into two:

1. daemon threads;
2. and non-daemon threads.

Daemon threads stop working when there are no other non-daemon threads. Even if you do not create any threads, the Java application will create several threads by default. Most of them are daemon threads, mainly for processing tasks such as garbage collection or JMX.

A thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.)

Getting a Thread Dump

We will introduce the three most commonly used methods. Note that there are many other ways to get a thread dump. A thread dump can only show the thread status at the time of measurement, so in order to see the change in thread status, it is recommended to extract them from 5 to 10 times with 5-second intervals.

Getting a Thread Dump Using jstack

In JDK 1.6 and higher, it is possible to get a thread dump on MS Windows using jstack.

Use PID via jps to check the PID of the currently running Java application process.

Use the extracted PID as the parameter of jstack to obtain a thread dump.

A Thread Dump Using jVisualVM

Generate a thread dump by using a program such as jVisualVM.

Figure 2:  A Thread Dump Using visualvm.

The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file.

Generating in a Linux Terminal

Obtain the process pid by using ps -ef command to check the pid of the currently running Java process.

Use the extracted pid as the parameter of kill –SIGQUIT(3) to obtain a thread dump.

Thread Information from the Thread Dump File

"pool-1-thread-13" prio=6 tid=0x000000000729a000 nid=0x2fb4 runnable [0x0000000007f0f000] java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)

              at java.net.SocketInputStream.read(SocketInputStream.java:129)

              at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

              at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

              at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

              - locked <0x0000000780b7e688> (a java.io.InputStreamReader)

              at java.io.InputStreamReader.read(InputStreamReader.java:167)

              at java.io.BufferedReader.fill(BufferedReader.java:136)

              at java.io.BufferedReader.readLine(BufferedReader.java:299)

              - locked <0x0000000780b7e688> (a java.io.InputStreamReader)

              at java.io.BufferedReader.readLine(BufferedReader.java:362)

• Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number).
• Priority: Represents the priority of the threads.
• Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.)
• Thread status: Represents the status of the threads.
• Thread callstack: Represents the call stack information of the threads. 

Thread Dump Patterns by Type

When Unable to Obtain a Lock (BLOCKED)

This is when the overall performance of the application slows down because a thread is occupying the lock and prevents other threads from obtaining it. In the following example, BLOCKED_TEST pool-1-thread-1 thread is running with <0x0000000780a000b0> lock, while BLOCKED_TEST pool-1-thread-2 and BLOCKED_TEST pool-1-thread-3 threads are waiting to obtain <0x0000000780a000b0> lock.

Figure 3: A thread blocking other threads. 

"BLOCKED_TEST pool-1-thread-1" prio=6 tid=0x0000000006904800 nid=0x28f4 runnable [0x000000000785f000]
   java.lang.Thread.State: RUNNABLE
                at java.io.FileOutputStream.writeBytes(Native Method)
                at java.io.FileOutputStream.write(FileOutputStream.java:282)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                - locked <0x0000000780a31778> (a java.io.BufferedOutputStream)
                at java.io.PrintStream.write(PrintStream.java:432)
                - locked <0x0000000780a04118> (a java.io.PrintStream)
                at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
                at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
                at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
                - locked <0x0000000780a040c0> (a java.io.OutputStreamWriter)
                at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
                at java.io.PrintStream.newLine(PrintStream.java:496)
                - locked <0x0000000780a04118> (a java.io.PrintStream)
                at java.io.PrintStream.println(PrintStream.java:687)
                - locked <0x0000000780a04118> (a java.io.PrintStream)
                at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:44)
                - locked <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState)
                at com.nbp.theplatform.threaddump.ThreadBlockedState$1.run(ThreadBlockedState.java:7)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - <0x0000000780a31758> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"BLOCKED_TEST pool-1-thread-2" prio=6 tid=0x0000000007673800 nid=0x260c waiting for monitor entry [0x0000000008abf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:43)
                - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState)
                at com.nbp.theplatform.threaddump.ThreadBlockedState$2.run(ThreadBlockedState.java:26)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - <0x0000000780b0c6a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"BLOCKED_TEST pool-1-thread-3" prio=6 tid=0x00000000074f5800 nid=0x1994 waiting for monitor entry [0x0000000008bbf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:42)
                - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState)
                at com.nbp.theplatform.threaddump.ThreadBlockedState$3.run(ThreadBlockedState.java:34)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - <0x0000000780b0e1b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

When in Deadlock Status

This is when thread A needs to obtain thread B's lock to continue its task, while thread B needs to obtain thread A's lock to continue its task. In the thread dump, you can see that DEADLOCK_TEST-1 thread has 0x00000007d58f5e48 lock, and is trying to obtain 0x00000007d58f5e60 lock. You can also see that DEADLOCK_TEST-2 thread has 0x00000007d58f5e60 lock, and is trying to obtain 0x00000007d58f5e78 lock. Also, DEADLOCK_TEST-3 thread has 0x00000007d58f5e78 lock, and is trying to obtain 0x00000007d58f5e48 lock. As you can see, each thread is waiting to obtain another thread's lock, and this status will not change until one thread discards its lock.

Figure 4: Threads in a Deadlock status.

"DEADLOCK_TEST-1" daemon prio=6 tid=0x000000000690f800 nid=0x1820 waiting for monitor entry [0x000000000805f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
                - waiting to lock <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
                - locked <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135)

   Locked ownable synchronizers:
                - None

"DEADLOCK_TEST-2" daemon prio=6 tid=0x0000000006858800 nid=0x17b8 waiting for monitor entry [0x000000000815f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
                - waiting to lock <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
                - locked <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135)

   Locked ownable synchronizers:
                - None

"DEADLOCK_TEST-3" daemon prio=6 tid=0x0000000006859000 nid=0x25dc waiting for monitor entry [0x000000000825f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
                - waiting to lock <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
                - locked <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
                at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135)

   Locked ownable synchronizers:
                - None

When Continuously Waiting to Receive Messages from a Remote Server

The thread appears to be normal, since its state keeps showing as RUNNABLE. However, when you align the thread dumps chronologically, you can see that socketReadThread thread is waiting infinitely to read the socket.

Figure 5: Continuous Waiting Status.

"socketReadThread" prio=6 tid=0x0000000006a0d800 nid=0x1b40 runnable [0x00000000089ef000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
                at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
                - locked <0x00000007d78a2230> (a java.io.InputStreamReader)
                at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107)
                - locked <0x00000007d78a2230> (a java.io.InputStreamReader)
                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93)
                at java.io.InputStreamReader.read(InputStreamReader.java:151)
                at com.nbp.theplatform.threaddump.ThreadSocketReadState$1.run(ThreadSocketReadState.java:27)
                at java.lang.Thread.run(Thread.java:662)

When Waiting

The thread is maintaining WAIT status. In the thread dump, IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue, then the thread status will not change.

Figure 6: Waiting status.

"IoWaitThread" prio=6 tid=0x0000000007334800 nid=0x2b3c waiting on condition [0x000000000893f000]
   java.lang.Thread.State: WAITING (parking)
                at sun.misc.Unsafe.park(Native Method)
                - parking to wait for  <0x00000007d5c45850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:440)
                at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:629)
                at com.nbp.theplatform.threaddump.ThreadIoWaitState$IoWaitHandler2.run(ThreadIoWaitState.java:89)
                at java.lang.Thread.run(Thread.java:662)

When Thread Resources Cannot be Organized Normally

Unnecessary threads will pile up when thread resources cannot be organized normally. If this occurs, it is recommended to monitor the thread organization process or check the conditions for thread termination.

Figure 7: Unorganized Threads.

How to Solve Problems by Using Thread Dump

Example 1: When the CPU Usage is Abnormally High

1. Extract the thread that has the highest CPU usage.
   
   [user@linux ~]$ ps -mo pid.lwp.stime.time.cpu -C java
   
        PID         LWP          STIME                  TIME        %CPU
   10029               -         Dec07          00:02:02           99.5
            -       10039        Dec07          00:00:00              0.1
            -       10040        Dec07          00:00:00           95.5

   From the application, find out which thread is using the CPU the most.

   Acquire the Light Weight Process (LWP) that uses the CPU the most and convert its unique number (10039) into a hexadecimal number (0x2737).

2. After acquiring the thread dump, check the thread's action.

   Extract the thread dump of an application with a PID of 10029, then find the thread with an nid of 0x2737.

   "NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable [0x49aa5000]
   java.lang.Thread.State: RUNNABLE
                   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
                   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
                   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
                   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
                   - locked <0x74c52678> (a sun.nio.ch.Util$1)
                   - locked <0x74c52668> (a java.util.Collections$UnmodifiableSet)
                   - locked <0x74c501b0> (a sun.nio.ch.EPollSelectorImpl)
                   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
                   at external.org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
                   at external.org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:708)
                   at external.org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
                   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                   at java.lang.Thread.run(Thread.java:662)
   

   Extract thread dumps several times every hour, and check the status change of the threads to determine the problem.

Example 2: When the Processing Performance is Abnormally Slow  

After acquiring thread dumps several times, find the list of threads with BLOCKED status.

" DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000]
java.lang.Thread.State: BLOCKED (on object monitor)
                at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
                - waiting to lock <0xe0375410> (a beans.ConnectionPool)
                at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
                at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)

"DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020]
java.lang.Thread.State: BLOCKED (on object monitor)
                at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
                - waiting to lock <0xe0375410> (a beans.ConnectionPool)
                at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
                at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)

" DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080]
java.lang.Thread.State: RUNNABLE
                at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570)
                - waiting to lock <0xe03ba2e0> (a oracle.jdbc.driver.OracleConnection)
                at beans.ConnectionPool.getConnection(ConnectionPool.java:112)
                - locked <0xe0386580> (a java.util.Vector)
                - locked <0xe0375410> (a beans.ConnectionPool)
                at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66)
                at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120)

Acquire the list of threads with BLOCKED status after getting the thread dumps several times.

If the threads are BLOCKED, extract the threads related to the lock that the threads are trying to obtain.

Through the thread dump, you can confirm that the thread status stays BLOCKED because <0xe0375410> lock could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock.

There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working, they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them, you will often see that some of the threads that were BLOCKED previously are in a different state.

The second reason is the abnormal connection. When the connection with DBMS stays abnormal, the threads wait until the time is out. In this case, even after extracting the thread dumps several times and comparing them, you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values, such as the timeout value, you can shorten the time in which the problem occurs.

Coding for Easy Thread Dump

Naming Threads

When a thread is created using java.lang.Thread object, the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object, the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application, if all the threads still have their default names, analyzing them becomes very difficult, because it is difficult to distinguish the threads to be analyzed.

Therefore, you are recommended to develop the habit of naming the threads whenever a new thread is created. 

When you create a thread using java.lang.Thread, you can give the thread a custom name by using the creator parameter.

When you create a thread using java.util.concurrent.ThreadFactory, you can name it by generating your own ThreadFactory. If you do not need special functionalities, then you can use MyThreadFactory as described below:

Obtaining More Detailed Information by Using MBean

You can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps, by using ThreadInfo.

You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo, and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time.

In Conclusion

In this article I was concerned that for developers with a lot of experience in multi-thread programming, this material may be common knowledge, whereas for less experienced developers, I felt that I was skipping straight to thread dumps, without providing enough background information about the thread activities. This was because of my lack of knowledge, as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers.

By Tae Jin Gu, Software Engineer at Web Platform Development Lab, NHN Corporation.

Every day, many developers and database administrators struggle to find ways to complete their tasks. One such task is statistics. Let's say that an operator (or a producer) asks you to retrieve some statistics data. It will be a relatively easy task if the size of the data is less than a few million rows, or if your organization has a dedicated backup database from which you can retrieve statistics.

But what if you have to retrieve a large amount of data without a backup database?

This will be a challenge for most of us. The Monitoring System Development Team has long been searching for better ways to process work involving statistics, and finally found a solution which involves using MongoDB and Hadoop. In this article, I will show you what we have done and experienced.

Introduction

The team is developing and operating monitoring systems such as xMon (these are common names of the monitoring systems used at NHN: uMon, kMon, cMon, jrMon and ocMon). The storage capacity of the Oracle database used by xMon is 20 TB. This Oracle database stores a large amount of data, such as user-generated content (UGC) metadata (the body or multimedia data are distributed and saved in OwFS and the video infrastructure) that have occurred on each NAVER service and the related log data for the past six months. The figure below shows the cycle of UGC generation and destruction that generates a log. The company has been using three Oracle DBMS systems with 128 GB memory to process 20 TB of data.

OwFS is a distributed file storage system developed by NHN . Each storage is allocated and distributed in the unit of Owner.

Naver is the major search engine service in Korea.

Figure 1. UGC Generation and Destruction Cycle.

The environment in which the Oracle DBMS was being used was getting worse as time went by. The amount of UGC was exponentially increasing. We adopted the policy of archiving only six months worth of data in the database, but the amount of data for that six months was getting bigger and bigger, creating a problem for us. Furthermore, the cost for additional Oracle DBMS storage could not be ignored. The table and figure below show the status and trend of the storage usage of xMon.

xMon Storage Usage

Firgure 2: xMon Storage Usage Trend.

After experiencing some difficulties, the team had to adopt a better system than Oracle DBMS because of the following two key requirements.

• Requirement 1: Be able to extract the user unit index by utilizing the existing log data.
• Requirement 1-1: The index must be extracted at least once a day.
• Requirement 2: The extracted user unit index must be immediately available online.
• Requirement 2-1: The index extraction must be implemented as a query function in the existing system.

To satisfy the first requirement, the index, which is a form of statistical data, must be extracted from the data as soon as possible. In the company's old system, statistical data was extracted in the following way: a developer would write an SQL and send it to a database administrator; then, the administrator would verify and execute the SQL and send the result back. As you can imagine, this process was far from speedy.

Requirement 2 can be summarized as "Provide index data that can be queried by a Web browser." Under normal circumstances, meeting this requirement is trivial; but when the size of data to be processed is somewhere around the realm of 4 TB, adding the function to the existing Oracle database is a problem. Adding index data to the storage would increase the load, which was already hitting the ceiling due to the load of processing the ever-growing UGC metadata and log data.

Data Aggregation and Extraction by Utilizing Map-Reduce

The required table for index extraction is a generic log table with 30 to 60 columns. This simple schema is logically related to the master table, but there is no actual relation. With this information at hand, one could easily conclude that utilizing DBMS or the Map-Reduce method would be the answer.

In my first attempt, I extracted the index from Oracle by writing an SQL. Adding a query function to the system is not much of a problem (if performance is of no concern) when there are already dozens of PL/SQLs and aggregation tables for a variety of statistics data. I divided the date into small chunks in my SQL. This is called the divide-and-conquer approach. It was exactly a week later when I got a call from the database admin. The admin. told me that the SQL in question had been executed for 860 minutes and the CPU usage was up to 60% (30-40% is the usual). The admin stopped the process. With that, my first attempt ended. Unfortunately, my first trial was a total failure. The following is part of the SQL code that is actually used in the field to aggregate the size of unreviewed data for last 1 month.

So, I picked Map-Reduce as an alternative. Map-Reduce works in the same way as the Sort/Merge functions of RBDMS. Map-Reduce is suitable for a batch data analysis for OLAP (On-line Analytical Processing), which tends to deal with a large amount of data. Most people would think of Hadoop when I say Map-Reduce, but other DBs such as MongoDB, HBase, CouchDB and Hypertable offer Map-Reduce as well. Among these, I chose two DBs to test and verify: Hadoop for its time-proven performance, and MongoDB for its document-orientedness, which makes saving logs in a sequence an easy task.

I extracted the data first because I couldn't access the Oracle if I were to verify the Map-Reduce of Hadoop. I used the Kettle of Pentaho, an open-source ETL (Extract, Transform, Load) solution, to extract the data stored in Oracle. For testing purposes, I used HDFS and Map-Reduce only, two of the many features of Hadoop.

I configured three different environments for testing. For Hadoop, I configured the standalone operation and the fully-distributed operation using three DataNodes and one NameNode. Then, I compared the resulting performance with the Map-Reduce of MongoDB.

For your information, the system specifications of the testing machines (each machine is identical) are as follows:

• CPU: Intel(R) Xeon(R) CPU L5420@2.50GHz * 2 (8 cores)
• Memory: 32 GB
• Disk: 814 GB (300 GB * 6 ea, RAID 0 + 1)
• OS: CentOS 5.2 x86_64

I used one server for the standalone operation of Hadoop, 4 for the fully-distributed operation, and 5 for MongoDB. I separated the name node and the data node in the fully-distributed operation of Hadoop. This separation was necessary because the Job Tracker that executes Reduce task and the main controller run on the name node, while the Task Tracker that actually executes the Map task runs on the data node. MongoDB provides functions that are similar to Map-Reduce of Hadoop. I will explain more about MongoDB later.

The figure below shows the cluster configuration of Hadoop, including the data node and the name node. See Figure 6 below for the shard configuration of MongoDB.

Figure 3: Hadoop Cluster Configuration.

As shown in the table and figure below, standalone operation is suitable for processing small data, while fully-distributed operation is good for processing large data. Processing small data with the fully-distributed operation is undesirable because the time it takes to collect distributed data in the same nodes during a Reduce operation outweighs the advantages of distributing data to each node using Map. In this test, the fully-distributed operation of 10 million rows of data outperformed other test conditions. The test result varies depending on the amount of data and the system specification - you must test the configuration before applying it to the system.

Comparison of Map-Reduce Performance between Hadoop and MongoDB

Figure 4: Comparison of Map-Reduce Performance between Hadoop and MongoDB.

Note that you must compensate for the difference in the number of servers used for the standalone operation and the fully-distributed operation by adjusting the amount of data to be processed. Also note that I did not consider using 4 systems for the standalone operation, because doing so would require a separate data split and collection process, which would require additional development resources. Besides, it is simpler to scale in the fully-distributed operation than it is in the standalone operation, because you can just add a data node without modifying the data split and collection process.

I have tested MongoDB as well, in the hope that I could implement both the aggregation system and the index storage system with MongoDB if its performance was similar to that of Hadoop. To my dismay, the Map-Reduce of MongoDB is quite subpar compared to that of Hadoop, as shown in the table above. In short, MongoDB is roughly four times slower than Hadoop in fully-distributed mode. A newer version of MongoDB has become available since the test, but I doubt that it would show a noticeable difference in performance from earlier versions.

Productivity - What Makes MongoDB Stand Out

When processing small data, MongoDB is still useful. This is because its productivity outweighs its lack of performance when processing small data. The following is the Map-Reduce method using the Java API in Hadoop.

The following is a list of internal commands used in the Map-Reduce of MongoDB:

Obviously, the MongoDB commands are far simpler than in Hadoop. This is because the commands of MongoDB take advantage of functional language, such as Scala and Ruby. Of course, Hadoop could be optimized with a script language such as Python.

Despite the outstanding performance of Map-Reduce in Hadoop, debugging it could be a challenge, because the data is displayed across multiple, distributed nodes. Maintenance could also be a hassle, because a jar archive must be created and executed when compiling code. But there is some good news - writing the Map-Reduce function is an easy task if you fully understand the underlying concepts. Although you could go the extra mile to fine-tune its parameters, I will not cover that in this article.

To recap, I tested the data aggregation performance for the following four items:

1. Oracle DBMS;
2. the standalone operation of Hadoop;
3. the fully-distributed operation of Hadoop;
4. and the Map-Reduce of MongoDB.

Out of the four contenders, the fully-distributed operation of Hadoop proved to be the winner for its speed and scalability.

Selecting Aggregation Information Storage: MongoDB vs. Cassandra

The test I have performed aimed to find the most efficient Map-Reduce alternative to replace the existing the Group by operation of RDBMS. Now it is time to select a storage to store, manage and provide the aggregation data. Oracle was eliminated from consideration from the start, because its scalability was limited.

I have chosen two test subjects, based on popularity. The first subject is Cassandra from Facebook, which is known for its column family form. According to an article, Facebook has developed an HBase-based real-time analysis system that can handle some 20 billion events a day. That means Facebook uses HBase as their Map-Reduce and storage. I wish I could benchmark HBase in my article.

The other test subject is MongoDB, which I paired with Hadoop for comparing the Map-Reduce performance. MongoDB is an intuitive document-oriented data model that offers flexibility in field management, one of the advantages of schema-free databases. In addition, MongoDB offers a full-text indexing feature, enabling a quick search against all fields in all documents or an indexing of a specific field. MongoDB has a reputation for good performance and reliability.

It is obvious that the read performance of Cassandra is inferior to that of MongoDB, as shown in the table below. Actually, there is a whole article written by my colleague from Storage System Development Team at NHN called NoSQL Benchmarking. Refer to it for deeper results on performance comparison between Cassandra and MongoDB for both READ and WRITE.

After reviewing the code and related documents, I have concluded that the philosophical difference between the two solutions led to the difference in performance. Since MongoDB is a document-oriented model, it processes one row of the database with a single insert operation. On the other hand, Cassandra, a column-based model, processes one column of the database with a single insert operation. For this reason, Cassandra is slower than MongoDB. The data used by the test consists of 30 columns. I assumed that the performance would be different by about 30-fold. But the actual performance showed a difference of 300 times.

The following code shows that the insert operation must be repeated as many times as the number of columns.

To confirm that the column-based insert operation caused the bad write performance, we have reduced the number of columns in the test. By combining the other columns except for key, we reduced the number of columns to two. As shown in Table 3, the 100,000 write operation performance of Cassandra was significantly faster when the number of columns was two than when it was 30. However, it could not still surpass the write performance of MongoDB.

Table 3 Comparison of Write/Read Performance of MongoDB vs. Cassandra

For reference, the schema structure used for Cassandra performance test is as shown in the following figure. We have freely configured the columns in order to include all aggregation information related to one user (in the unit of user_id). Designing it as a hierarchical structure made it easy to add data items to each layer. It allowed a variable schema design, which we thought would be a benefit. However, the test result showed poor performance, so we had to discard it.

Figure 5: Cassandra Schema.

When the fully-distributed operation was provided by Cassandra, the read performance of the same key will be better than it is now. However, the test considering N nodes was excluded from the scope of this test due to our internal situation. Therefore, for consistency in comparison, we have configured MongoDB as a standalone operation.

For reference, MongoDB supports write-ahead journaling from version 1.7.5 for recovery from errors and durability of the storage engine. If this journaling option is used, the stability will improve, but the performance measurement result of MongoDB will get worse. Therefore, this option has been excluded from the test scope, as the aggregate data allows easy recovery.

Separately from the above test result, there was another important reason why we have selected MongoDB; it provides the composite index function. To utilize various inquiry conditions for aggregate data, we needed to set multiple indexes. MongoDB can create indexes for data of one million cases in just one second. In addition, users familiar with the relational database can more easily access the solution, since it is conceptually similar to SQL. On the other hand, to use Cassandra, users must understand the key-value based tree structure.
Based on the performance test of MongoDB and Cassandra, we selected MongoDB by considering the batch write performance and performance of reading various columns. The write/read performance test was carried out based on standalone operation; however, the final configuration of MongoDB was determined as five servers as shown in the following figure.

Figure 6: MongoDB Cluster Configuration.

We underwent two trials configuring as shown above. First, we configured the cluster with four mongod config servers, excluding one arbiter server, but the cluster did not operate. The configuration document specified "one or three servers must be used to configure a cluster," but we missed that point. In addition, when the Shard configuration across several servers is incorrect and there is a network failure, data integrity may not be kept. To solve the problem, an additional arbiter server (cbat005 in the figure) must be configured.

Final Hadoop & MongoDB Based Aggregation System

The following figure shows the final configuration of the system. The system may be configured with the Hadoop server and MongoDB in the standalone operation. However, we configured it to enable easy expansion, as the data volume is continuously increasing. Data flows in the order of Oracle, ETL, Hadoop, and MongoDB.

Figure 7: System Configuration.

The operation scenario of the above configuration is illustrated as shown below.

1. Select the target data to aggregate from the database.
2. Extract data by using ETL tool.
3. Balance and load the data to the HDFS DataNode of the Hadoop Cluster.
4. Execute Map-Reduce in the NameNode of the Hadoop Cluster.
5. The aggregation result is transferred and loaded to the MongoDB by using the transfer application. At this time, L4 is used for load balancing.
6. Data loaded onto the MongoDB is provided to users via the external modules.

Among the above processes, a natural data flow is shown from the first process to the third process, but in the fourth process, aggregation is executed by using Map-Reduce. In the fifth, the aggregation result is saved through the API provided by the MongoDB. When MongoDB is accessed, it is recommended to distribute load through L4. In addition, the server list (mongos server list) may be described while writing clients. Or, it may be possible to use RequestBroker. When the first to the fifth processes are complete, the user can receive data. As a supplementary explanation of the process to store data in the MongoDB, the client server determines in which server Shard data will be stored through the config server. Data are stored in the master server of Shard and copied to the replication server. MongoDB specifies the arbiter in order to select the master server from among the remaining servers when there is any failed distributed server.

Conclusion

We have accomplished the first goal by configuring the system as described. We recommend you consider the following if you are planning to introduce a Map-Reduce or NoSQL solution.

If the data volume is not particularly high and Map-Reduce must be applied, configure the environment to Hadoop standalone operation. If the data volume is large, configure the fully-distributed operation consisting of at least three DataNodes. However, if you are using MongoDB and the data volume is not particularly high, MongoDB Map-Reduce will be useful.

For NoSQL, if multi-index is required, choose MongoDB. If there are many changes in data items and many unique accesses, Cassandra will be better. Since we have not yet tested Hbase, an additional review of that is also required.

After accomplishing the first goal, we are considering the second goal. We are trying to establish long-term measures that can effectively process the continuously increasing Oracle data. To solve the problem, we think that the log data for which coupling to the master data is low and inquiry frequency is also low must be moved to the NoSQL storage. If this problem can be solved using the NoSQL solution, we expect that we can reduce the costs of using relatively-expensive Oracle storage.

The NoSQL solution is being advanced according to various needs. The biggest advantage of the NoSQL solution that I have experienced is Scale-out. And the relatively-fast write/read performance is a bonus. However, there is no such thing as a free lunch; we can lose our precious time in development.

I wanted to share our NoSQL experience in the hope that it can be helpful for you. This is the first article I have posted on this site, so I'm concerned that my poor writing skills may not be sufficiently effective to convey our experience. I hope that more information on the NoSQL solution can be shared in the future.

By Jeong Hyun Lee, Web Service Development Team, NHN Corporation.

Let’s look at what Nginx is and how it differs from Apache Web Server.

In the beginning of 21^st century as the use of internet became more active, people began to shed more interest on web servers capable of processing more requests. Later the C10k problem emerged in this era, in other words to enable each web server to be capable of processing 10,000 clients at the same time. Therefore a need to develop a better Network I/O and Thread management technology was emerged. During this period, far less-developed technology was applied to Linux than previous ones such as epoll and NPTL.

epoll was designed for better process of IO events on Linux. kqueue (FreeBSD) and IOCP (Windows) are similar. NPTL is a thread library of Linux loaded on Kernel 206 and up (based on safe version). It provided enormous improvement in thread performance.

It is fair to say that the emergence of several technology was not the result of an effort to tackle C10k problem (as a common problem) but "The C10k problem" succeeded in putting together efforts to enhance performance of network server development. As a result various web servers such as Nginx, Lightpad and Cherokee made with new technology were in the market.

Nginx was first developed by a Russian guy called Igor Sysoev in 2002. After two years in 2004 it made its debut in the forms of HTTP server and reverse proxy/IMAP/POP3 server. To respond to C10k problem, Nginx adopted event-driven (asynchronous) structure which is not a conventional way (as in Apache) of processing single client in a single thread. Other than Nginx, various newly developed web servers such as Lightpad, Tornado, Magnum and Aleph have adopted Event-driven structure too.

Event-driven Architecture (EDA)

EDA method is capable of handling more number of clients with less threads compared to conventional way (one thread for one client). In Apache each thread is dedicated to handle one client. So, there will be many cases when a certain thread needs to be on hold due to I/O problems until certain data is read from persistence layer in "accept" and delivered to the client. There should be the same number of threads as that of clients.

However, with EDA a method enables event (i.e. when event needs to be created or response result is completed) to be processed every time it takes place by designating each state. In other words, with less thread (or even without any thread) CPU can be used more efficiently.

Nginx provides almost all the functions Apache web server does. For example:

• handling of static files
• reverse proxy
• load balancing
• SSL support
• FastCGI
• Virtual Host
• FLV Streaming
• MP4 streaming
• Web page access authentication
• gzip
• URL Rewriting
• Custom Logging
• SSI
• WebDAV

Nginx Use

According to a research done in April, 2011 by W3Techs.com, Nginx takes up 6.9% in web server market and is on a steady increase. The graph below shows the market share of Nginx in recent 1 year.

We can see that many websites process massive traffic using Nginx. Alexa.com is well known for releasing website ranking according to its traffic. The following list, released on April 18, 2011, illustrates websites from Top 500 which use Nginx.

Characteristics of Nginx

Non-blocking event-driven method

The most notable characteristic of Nginx is that it uses non-blocking event-driven method. All the network connection operates in non-blocking way. Some socket interfaces return results immediately, while some are blocked in certain cases. Socket() and setsockopt() returns results without blocking, whereas connect(), send(), recv() and close() might experience some blocking in certain cases. Nginx calls above mentioned functions only after confirming there will be no lags. In order to prevent blocking, as for connect(), socket is changed in advance to non-blocking using ioctl().

As for send() and recv(), epoll is used to confirm blocklessness. Codes for calling send() or recv() are composed in event-driven format. Each event is composed of a socket, a socket state and an operating function. Almost all of internal Nginx is operated in event-driven way since web server processes request from web browser through network.

Single thread

Nginx is operated by pre-set number of worker processes. Each process operates as a single thread. Also, the need for multi-thread is relatively low since with the use of Non-blocking event-driven allows a single worker process to handle requests by multiple clients.

Advantage of Nginx

High Performance

Generally Nginx is known to record higher performance than Apache. However, we cannot simply jump to conclusion in terms of better performance since it varies greatly according to workload. Below chart indicates a system function that is called to serve a single favicon.ico using starce. As you can see in the table below Apache called 31 system functions whereas Nginx called 16. Although the system function used here cannot accurately indicate the performance, we can still expect relative performance.

Handling many clients with less process

Disadvantage of Nginx

Difficult to create module

Does not support HTTP/1.1 in communication with Backend

Things to note while using Nginx

• Nginx has a non-blocking event-driven structure. Such structure is desirable for network transmission and reception. However, it could generate blocking when file is being sent and received. But the latest Linux versions support Asynchronous Input-Output (AIO), and Nginx began to support AIO since version 0.8.11.
• Nginx uses AIO function through eventfd() system function of Linux. However it only works when AIO of Linux is set as DirectIO. Thus, using AIO on Linux disables buffer cache of OS.
• As in the case of reading the same file over and over again, it reads from disk in the first place however from the second time it can rapidly read stored file on buffer cache. No matter how many times the file is being read, using DirectIO makes it to be read from disk consistently.
• If a DBMS uses independent buffer, DirectIO might be efficient. However in other cases DirectIO could decrease file input/output performance.
• If most of the files that Nginx read exist in OS buffer cache, most of file I/O become non-blocking. However if Nginx gets to serve files which do not existent in the buffer cache, the worker process becomes blocked due to file I/O. In this case, you need to increase the number of worker processes in order to prevent the worker process from becoming blocking.

CUBRID 8.4.1 Beta is available for download right now!

Thanks to all of those who have reported bugs, CUBRID developers have done really great job on improving both the performance and stability of the database server. While performing tests on large amount of transaction, performance of INSERT and UPDATE was improved over 70%.

Also, according to your requests we have added many new functions and SQL syntaxes to make CUBRID more compatible with MySQL. Moreover, regular expressions are now supported in conditional search, broker parameters are added, and monitoring features have been improved. In addition, approximately 50 major and minor bugs have been fixed.

Below I will list new features and major changes in this new release.

Improved Performance of INSERT and UPDATE more than 70%

(y-axis refers to the number of queries per second)

In CUBRID 8.4.1 we have improved the internal implementation of the Engine to enhance the performance during simultaneous processing of transaction logs, writing memory buffers to a disk, and performing replication for HA. Consequently, the performance of INSERT and UPDATE operations was dramatically improved over 70% in comparison with 2008 R4.0 Patch 2 version.

The following table illustrates the overall performance increase gained in CUBRID 8.4.1.

The test has been conducted on large amount of transactions based on YCSB (Yahoo Cloud Serving Benchmark). For more information on performance test, refer to section 2.2.2 in CUBRID 2008 R4.1 QA Completion Report.

New SQL Functions and Regular Expression Support

• In this version we have added new SQL functions: ADDTIME, ASCII, BIN, CONV, FIND_IN_SET, HEX, and SERIAL_NEXT_VALUE (serial_name, number) functions, which can be used to obtain multiple serial values with a single call.
• Added the REGEXP operator in order to allow conditions to be searched with regular expressions.
• DATE, DATETIME, and TIMESTAMP data types now support '0000-00-00', '0000-00-00 00:00:00' lower bound values. This provides more compatibility with MySQL's SQL syntax.

More Features

The full list of new features, improvements, and fixes can be found the in CUBRID 2008 R4.1 Release Note.

Simply put, phpMyEdit is an open source instant database table editing tool. It generates PHP code for you that you might need to display or make the necessary changes in the database tables through your web site interface. What you need is just copy the generated code and paste it into your web application's back-end code. You can customize it to fit your web site layout. It will make sure everything looks like as if you created this code yourself.

So this video will explain you what phpMyEdit is, how it works, and how to use it. You will learn how to use the code generated by phpMyEdit to do certain database operations. phpMyEdit includes a huge set of table manipulation functions (record addition, change, view, copy, and removal), table sorting, filtering, table lookups, and more.

Fortunately, now phpMyEdit, a powerful database table editing tool, is available also for CUBRID. You can download it from CUBRID SF.net project site http://sourceforge.net/projects/cubrid/files/CUBRID-AppsTools/phpMyEdit/phpMyEdit-5.7.1-cubrid.zip/download.

It will save you a great deal of your precious time when developing a PHP code which needs a database interactions. For more information, visit phpMyEdit Official Community Web Site at http://www.phpmyedit.org/.

Later we will be uploading more videos. If you want to be up to date and follow our video tutorials, subscribe to our CUBRID Blog or CUBRID Trainings Channel, follow us on Facebook or Twitter.  We will make sure you get the latest news and updates about CUBRID.

Last month we announced the first beta version of official ADO.NET driver for CUBRID. Today, we are happy to announce the stable 1.0 version release of the driver.

• The ADO.NET driver, developed by CUBRID community contributors, has now over 70 test scenarios.
• The driver is released with 100% fully-managed code under the terms of BSD license.
• All main classes of ADO.NET model have been implemented.
• Extended classes implementation (SchemaProvider, CommandBuilder, etc.).
• It implements the (socket) communication protocol of the latest version of CUBRID 8.4.0.
• Supports BLOB/CLOB data types and CUBRID Collections.
• Localization support is built in. For now, only English messages are released. Feel free to contribute the translation of your own language.
• Various source code, demos, examples (UI, VB.NET, NHibernate) are available on the project Wiki page.

Download

You can download CUBRID ADO.NET driver from http://www.cubrid.org/?mid=downloads&item=ado_dot_net_driver&os=windows.

There are 3 ways you can install the driver.

1. The easiest way is to use the auto-installer (available on the downloads page).
2. Download the compiled library.
3. Install from the source code.

For detailed instructions of each installation method refer to CUBRID ADO.NET Driver Installation Guide and ADO.NET Tutorials.

The complete driver documentation is available at http://www.cubrid.org/manual/api/ado.net/1.0/Index.html.

Additional resources

• v1.0 Release Note
• CUBRID ADO.NET Driver project page
• Issue Tracker

Difficulty of Distributing Static Files

When static files such as the CSS and JavaScript are allowed to be cached on the client side by a browser, users can experience an increased speed and the server can be relieved from the burden of serving those resources. However, once the content of these static files changes, instead of using files stored in the cache, the browser must download the latest versions from the Web server. The most common method used to achieve this is by adding a query string-type timestamp to the name of a static file like common.css?t=20110405. The timestamp value used here can represent either the build time of the file, or the start time of the Web server, or the revision number of a repository. However, these solutions may cause some issues. I will explain them below.

Figure 1: Server Before and After Distribution.

Inconsistent distribution

For example, if a user accesses a Web page and obtains an HTML file from Server A (post-distribution) but CSS files used in this HTML files are served from Server B (pre-distribution), the user will end up using the new HTML file and the old CSS file, which may result in broken layout of the page or site malfunction.

1. To prevent this kind of problem, it is necessary to change the name of a newly distributed file making sure that the old HTML file refers to the old CSS file, and the new HTML file refers to the new CSS file.
2. If several Web servers are used for file distribution, the static files may be distributed in advance to prevent CSS or JavaScript files from being distributed to only some of the servers.

By using the method above, most problems that can occur while distributing static files can be prevented.

Decreased Performance

However, if the number of static files referred by a Web page is large, the performance may suffer.

1. For instance, in case a file has not been cached on the client side, it must be downloaded whenever it is requested.
2. On the other hand, if a file has been cached, a "304 Not Modified" will be received whenever a user accesses the server, incurring a DNS resolve cost and a socket connection cost.
3. Even when you merge the static files to reduce their number, unnecessary blanks and long variable names tend to increase the size of the transferable data.
4. More than that, the size of the transferrable data will grow when static files are distributed to the Web server, as cookies included in an HTTP request message will be transferred as well.

Proposed Solution

Though it would seem tempting to have a single person manage all of the static files written during the development process, it often proves inefficient due to an increased management cost. Therefore, the distribution process must be automated to the greatest extent possible, and the following rules must be kept for the automatic and efficient distribution of static files.

1. Add the timestamp value to the query string.
2. Manage the versions of CSS JavaScript files.
3. Reduce the number of files.
4. Minify file content.
5. Distribute CSS and JavaScript files from a cookie free domain.

Optimization by Using Apache Web Server Configuration

1. Compress the static files using the deflate module of the Apache Web server and transfer the compressed file. Add the deflate module configuration to the Apache configuration file, as shown below:

# Add the deflate module
LoadModule deflate_module modules/mod_deflate.so

# Compression level
DeflateCompressionLevel 1

# Set the compression level based on MIME type
AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml text/javascript

# Add the MIME type 
AddType text/javascript .js
AddType text/css .css



2. Then, add the expire module to allow the browser to cache the static file.

# Add the expires module
LoadModule expires_module modules/mod_expires.so

ExpiresActive On

# Set the default expires to one year
ExpiresDefault "access plus 1 years"

HOW TO manage static files

Assume that JavaScript files are located in /web/js directory and main.js and detail.js files referred in a Web page are located in that /web/js directory.

Figure 2: Directory Structure.

For an easy version control, mark each JavaScript file with its version (ex: main_v1.js, detail_v1.js).

How can I add a JavaScript file without describing the version and timestamp on each Web page?

1. ... scan the subdirectory files including /web/js;
2. and create a list of files with the highest version;
3. then mark the file path by using the custom tag as shown below:
   
   <link:js prefix="/js/main.js"/>

The following code will be displayed in the HTML:

To add new content or modify the existing one that does not support backward compatibility, get a higher version (ex: main_v1.js  main_v2.js). The script will load the latest version.

What are the potential issues which can arise when using this solution?

If static files have been distributed in advance, all Web servers will have both main_v1.js file and main_v2.js file. Therefore, no problem would occur during the distribution.

On the Web server after distribution, a new Web page will be linked to a new JavaScript file (main_v2.js), and on the Web server before distribution, the existing Web page will be linked to the existing JavaScript file (main_v1.js).

During the Web page development process, the development versions of static files are used (external URL). What should I do in this case?

Modify the custom tag to describe the development URL like:

While developing the Web application, use devOnlyUrl. After completing the development, use the highest version of the main.js file. Of course, there should be a way to set the development status of the file.

If a CSS or a JavaScript file is provided from a domain that does not use cookies, will there be any problem?

If a user-defined cookie is used in the JavaScript file, that file must exist in the path within the Web server. To achieve this, an additional setting such as needCookie is required in the custom page.

HOW TO merge static files

So far, I have described how to manage static file versions and how to describe files being developed. Now, I will explain how to bind the main.js file and the detail.js file to one core.js file to reuse their number. Write the configuration file as shown below:

How do I manage the versions of the created files?

1. If you have bound main_v1.js and detail_v1.js files to create one core_v1.js file, and there is no change in those files, keep the file name as 'core_v1.js'.
2. When the content or the version of the main_v1.js or the detail_v1.js file is changed, the core_v1.js file must be changed to the core_v2.js file. To do this, you must manage the name, size, and checksum of each file that is part of the file to be created.

Now, you should minify the bound files. There are a lot of libraries related to optimization, but I will use the most popular one, YUI Compressor.

When should I create or optimize the files? At runtime or at build time?

To provide static files from CDN (Content Delivery Network) or a separate server, I recommend creating or minifying the file at build time. I have implemented a file creation and minification function as a Maven plug-in for that specific purpose. The following code is an example of the pom.xml file:

Distribution with BDS

NHN distributes Web applications using its in-house developed Build Distribution System (BDS). The following figure illustrates the existing deployment process in a simple manner.

Figure 3: Deployment with BDS.

The following figure shows the process of CSS and JavaScript file compression and the existing deployment process.

Figure 4: Deployment for Minifying CSS and JavaScript File.

1. Compressed file creation: When you build a Web application using Maven, create a compressed file by reading the compression.xml file.
2. Version file creation: By using the information described in the compression.xml file, create a version file and save it in the version file repository (the repository described in the pom.xml file).
3. CSS and JavaScript file copy: By using the PostScript function of BDS, copy the CSS file and the JavaScript file to the CDN or the cookie free domain server.

How to Configure and Use

1. Create a compression.xml file and save it in the web/WEB-INF directory.
2. For Web projects that use Lucy, set the Lucy plug-in as shown below:

<lucy:plug-in name="compress" class="com.naver.shopping.common.compress.web.CompressPlugIn">
        <lucy:param name="configLocation" value="/WEB-INF/compression.xml"/>
        <lucy:param name="devOnly" value="false"/>
        <lucy:param name="baseUrl" value="http://static.shopping.naver.net/f"/>
</lucy:plug-in>

For Spring MVC-based Web projects, register Bean as shown below:

<bean class="com.naver.shopping.common.compress.web.CompressBean">
         <property name="configLocation" value="/WEB-INF/compression.xml"/>
         <property name="devOnly" value="false"/>
         <property name="baseUrl" value="http://static.shopping.naver.net/f"/>
</bean>



3. Add the Maven plug-in settings to the pom.xml file.
4. Describe the following in the jsp file:

<link:js prefix="/js/lib.js"/>
<link:js prefix="/js/ac.js" needCookie="true"/>
<link:js prefix="/js/notexists.js"/>

What I have described in the above steps will result in the following. Among the static files, the JavaScript file that requires cookies is served from the Web server itself, and the CSS file and the JavaScript file that does not require cookies are served from the cookie free domain or the CDN.

The /js/notexists.js file is not defined in the compression.xml file. In this case, the file is returned with the file name and timestamp of the highest version that starts with notexists under the /js directory.

Conclusion

In this article, I have briefly explained how we distribute and deploy static files at Knowledge Shopping service department here at NHN. The method I have described is not new. But I am making, what used to be an insider-only tip, public so that more people are aware of a better way to distribute static files and share their thoughts.

I would be really glad if you shared in the comments below how you distribute static files. Perhaps there are better ways to do this.

By Young-eun Oh, Shopping Service Development Team, NHN Corporation.

The driver supports all Perl::DBI methods, including:

• Driver handlers
• Database handlers
• Statement handlers
• Statement handler attributes
• LOB data manipulation methods

The full list of available functions and latest changes can be found in Release Notes.

The Perl driver for CUBRID is available from CPAN and can be installed using the following commands in the terminal:

For more details see Installation Instructions.

The source code of CUBRID Perl driver is available at http://www.cubrid.org/?mid=downloads&item=perl_driver&os=detect under the terms of BSD open source license.

Additional resources

• 8.4.0 Release Note
• Development Notes
• CUBRID Perl driver project page
• JIRA Issue Tracker
• Questions and Answers

SSD is faster than HDD. Then, would it be better to switch the storage medium from HDD to SSD? Let's learn about the DBMS buffering first, before considering the switch. 

Data Buffering

SSD has faster reading/writing speed than HDD. So replacing HDD with SDD will speed up the performance of a number of programs, including the OS. The performance improvement in the DBMS by replacing the storage medium from HDD to SSD, however, is minimal to none.

This is because data buffering (page buffering) is used in the DBMS. Data buffering is a sort of caching, and can decrease the storage medium I/O. In an environment where data buffering is running efficiently (i.e. the hit ratio is high for the caching system), the I/O performance of the storage medium become relatively less important.

So, if you want to improve the performance by switching from HDD to SSD, you should check if the DBMS' data buffering is operating properly before making the decision. You may get the improvements you wanted just by data buffering and without switching the storage medium to SSD.

In this article, we will look at the data buffering efficiency in both HDD and SSD environment and how the TPS changes accordingly, to find out the efficient way to configure the data buffering size and the best way to select the storage medium. 

HDD Structure and IO Features

HDD records data on a revolving platter (magnetic disc). HDD is composed of several platters, which all revolve together instead of revolving independently from one another. As you can see from Figure 1 below, the concentric circle in the middle of the platter is called a track, and the track found in the same location as several platters (same track number) is called a cylinder. The head reads and writes magnetic data on the platter, and since data can be recorded on both sides of the platter, each platter has two heads. This head is attached to the end of the disc arm. 

Figure 1:  HDD's Track/Cylinder, Sector and Heads.

Data is recorded by block in a consecutive order on the cylinder of the rapidly spinning platter. This allows easy access to the consecutively recorded data with a single movement (platter rotation and disc arm movement).

Figure 2:  Inside the HDD.

So, the IO time for a HDD refers to the total "seek time where the disc arm moves to the relevant cylinder," "rational latency where the platter rotates until it reached the relevant block (sector)" and "data transfer time where the head reads or writes data." 

As the seek time and rational latency are much higher than data transfer time (seek time >> rational latency > data transfer time), the OS generally uses the optimized IO scheduling type, which minimizes the head movement and the platter rotation.

However, IO scheduling effects cannot always be counted upon because the application's IO request for OS is very frequent and the OS cannot predict the timing.

When creating an application, make sure that it requests a small number of large-volume read/write operations rather than frequently requesting a large number of small-volume read/write operations. Doing so can increase the system performance by allowing an effective IO scheduling at the OS level.

CUBRID lumps disc IOs by page. In addition, IO requests that do not affect response time are internally sorted and processed by their page numbers so that the OS could minimize the seek time and rational latency for a faster scheduling. The difference in performance depends on whether to apply this or not.

In HDD, time for the head to read and write data is relatively very short. Because of this, when it comes to improving disc IO performance, shortening seek time and rational latency becomes crucial. Sometimes the sequential writing shows better performance than SSD.

The Solid State Disk Structure and IO Features

Solid State Disk (SSD) is a disc that uses flash memory as the storage medium. Flash memory is a type of EEPROM that can read, write or erase the memory content electronically.

Figure 3:  Inside the SSD.

When the memory is initialized, all bits are set to 1, whereas certain bits can be changed to 0 if programmed by page. Though the programmed pages can be reprogrammed, changing the bit from 0 to one is impossible while changing from 1 to 0 is possible. 

Because of this, in order to be used globally, you must erase and initialize all bits in a specific area to 1 before any data can be recorded again in same area. 

For HDD, it takes about the same time for the head to read and write data, but SSD has different reading and writing performance. Though this varies with each product, if the page reading time is about a few hundred nanoseconds (ns), the writing time is in microseconds (us) and erasing time is in milliseconds (ms).

Erasing an area happens simultaneously, and it usually takes the same amount of time to erase the entire chip or just a few pages. So, it is usually organized to process erase by sector (or block), which is the collection of several pages.

For a flash memory, there is a limit on how many times a certain sector can be erased. (The limit is usually about tens of thousands to hundreds of thousands.) Because of this, physical block addresses are mapped differently from logical block addresses so that there will be no frequent renewals in certain areas like FAT, and the number of erasing is set evenly in each sector. This process is called wear leveling.  

An additional effect of wear leveling is that, if the content of a certain page needs to be moved, the erasing and writing tasks need to be processed, but if the content needs to be moved to, and written in a different physical location (and if that page has been erased already), then writing can be performed without erasing. However, if empty spaces cannot be secured at the right time, this may also reach its limit.

For these reasons, IO of SSD is very different from HDD. When reading, as SSD only requires data transfer time, unlike HDD which also requires seek time and rational latency, a constant response speed can be guaranteed even for random reading.

But when writing, if there is no empty spaces in the middle of wear leveling, then it will be erased by sector while IO tasks for the corresponding sector will be all put on hold. 

DBMS - Data Buffering

The data buffering of DBMS is a type of caching. As the memory is fast and the disk is much slower in comparison, and recently and frequently used data may be used more often for a certain period of time in the future, frequently used data are stored to be read from the memory instead of the disk.

Buffering from DBMS can be processed with consideration of the transactions, which is different from disc cache of the OS. Additionally, though it was built based on LRU Caching Algorithm, by separating hot zone and cold zone, LRU management cost is cut down dramatically.

The overall processing time for an IO request from DBMS can be explained as the followings.

total access time = A * HR + B * (1 - HR)

                                 A = Access time of memory
                                 B = Average access time of storage
                                 HR = Biffer Hit Ratio

Therefore, the difference of processing time of HDD and SSD, B * (1-HR), will decrease as the hit ratio gets bigger. Common DBMSs have a hit ratio of 95% or higher. 

Test Environment and Scenario

This is the specifications of the DBMS server used for our tests:

• CPU: Xeon® CPU L5650 (2.27GHz 2*6 cores)
• Memory: 16G bytes
• HDD: 300G * 2
• SSD: 64G * 4
• OS: CentOS 5.3 x86_64
• DBMS: CUBRID 2008 R4.1 (8.4.1.0516) (64bit release build for linux_gnu)

YCSB (Yahoo! Cloud Serving Benchmark) program, created by Yahoo to benchmark cloud service storages, was used as the load generator. You can set different query patterns or distributions for analysis. In this test, the latest distribution (modified zipfian distribution and a high density pattern) sends the select request to DBMS. 60 transactions were processed at the same time for this test.

We measured TPS, hit ratio and the number of IO requests processed per second, as we changed the storage medium from HDD to SSD in the same equipment, and the data buffer from 2G to 12G, 1G at a time.

Test Results

The below are the results from this test.

Graph 1:  TPS Progress by Buffer Size.

As the data buffer for DBMS is increased, the TPS is increased as well, for both SSD and HDD. As HDD is relatively slow, once the page buffer size is increased for HDD, the TPS is increased as well. 

The graph for SSD demonstrates a simple proportional relationship outside of sections 2G - 4G in the middle.

On the other hand, the graph for HDD shows an S-shaped curve, with an inflection point around 10 G.

Graph 2 below is the collection of stat dumps from CUBRID, which shows the progress of IO requests made from DBMS to the OS.

The result of the IO request when using SSD shows an exponential function shape that simply decreases according to the buffer size. On the other hand, when using HDD, it shows a graph with an inflection point similar to the TPS graph, except that the inflection point is around 6G.

Graph 2: IO request Progress by Buffer Size.

The hit ratio graph of the DBMS data buffer is as shown in Figure 3. As it was tested with the same distribution, SSD and HDD show the same shape. And though the hit ratio is the same, due to the difference in process speed, IO request shows a different shape, as in Graph 2.

Graph 3: Hit Ratio Progress According to the Buffer Size.

The graph has the shape of an exponential function with its tangent at 100%, and its incline becomes more gradual as the buffer size increases.

Result Analysis and Implications

As the test results indicate, the performance of DBMS is very sensitive to the hit ratio of the data buffer. 

Therefore, to improve the performance of DBMS, it is crucial for the data buffer to maintain an adequate hit ratio.

The data buffer should be configured to the appropriate size to include the main working set. Also, the DBMS schema and application pattern should be designed to compose the main working set.

If a good hit ratio cannot be maintained in this way, then consider switching to SSD to maintain the performance above a certain level. 

Let's analyze this proposition in detail, using the test results.

The application used for the test was zipfian distribution, which is a model with a well-composed working set.

Now, the DBMS data buffer must be composed to adequately cover the working set. The graph for hit ratio changes by data buffer is the same shape as Graph 3.

The incline of the graph changes according to the distribution of the working set, but tends to have the same shape as Graph 3 (exponential function with 100% as the tangent). In this case, the buffer size and the cost are directly proportional, but the incline of the hit ratio decreases. Therefore, the data buffer's extension cost and performance ratio start to decrease.

So, 8G, where the graph's incline becomes more gradual, should be the right data buffer size.

Intuitively, the TPS can be predicted to be proportional with the hit ratio graph, but as you can see in Graph 1, it shows different patterns depending on the IO features of the storage medium.

As SSD "reads" fast, the TPS shows a gradual decline.

HDD, however, shows a pattern with an inflection point. In Graph 1, the areas in the square show that the performance ratio for SSD and HDD are proportional, but after that area, the TPS performance of HDD starts to decline steeply, and then declines gradually.

This phenomenon can be linked with the IO request development in Graph 2. The HDD is more than able to process the IO request that increases with the smaller hit ratio in section A (between 10 G~12 G). So the TPS graph above only shows the difference in physical performance compared to SSD.

But once the buffer size becomes equal to or smaller than 10 G (section B), HDD cannot process the increased IO requests within the set time and starts to stall. The slowdown in IO requests causes a steep decline in the TPS graph.

This would be the state in which the IO scheduling from the OS is not functioning well, as we have seen before. 

On the other hand, as the data buffer grows smaller (section C), the number of processing IO requests starts to increase within the set time, which means the OS' IO scheduling is working efficiently as the accumulated IO requests increase. This causes the TPS drop to flatten. However, even when IO requests increase, the hit ratio decreases, and the overall TPS decreases as well.

Gathering all results, the hit ratio starts changing gradually around 8G, but TPS incline grows steeply and reaches the inflection point at 10G. Because of this, the data buffer's performance improvement compared to the cost is maximized around 8G - 10G.

So in such environment, it would be more efficient cost-wise to install the data buffer to around 10 G instead of changing the storage medium to SSD.  

By Ki Yeul Lee, Senior Engineer, DBMS Development Lab, NHN Corporation.

Today CUBRID revealed its latest internal performance test results conducted using the latest CUBRID 3.0 Beta and the latest MySQL 5.1.47 (innoDB) database servers. The results are quite surprising for both database systems.

In this experiment CUBRID and MySQL database systems are tested to reveal their performance results in two different situations:

1. when the databases run on the HDD (Hard Disk Drive) equipped server
2. and, when the databases run on the SSD (Solid State Disk) equipped server

The test confirmed that TPS levels of CUBRID and MySQL database systems increase on SSD equipped machines. During the I/O Bound workload CUBRID had 4.2 times increase in TPS, while MySQL had 2.8 fold improvement.

Follow the link below to read the full report

http://www.cubrid.org/ssd_performance_test

Today we have several good news we would like to share with all of you! Most of the times we post the latest activities of our community through Twitter and Facebook. So make sure you follow us. For the last two months there have been quite an impressive amount of work accomplished at CUBRID Community. We have pushed the CUBRID gem to RubyGems.org, updated our tutorials for JDBC & Java Programming, released the Python Connector, published PHP API to Official PHP.net PECL Repository, released the CUBRID 3.0 Beta, and started Joomla CMS porting. Let's see in more details...

CUBRID-Ruby Gem

CUBRID account at RubyGems.org was created on July 11th, 2010. The CUBRID Connector testing has been started since then. Windows OS compatible cubrid-0.6-x86-mswin32-60.gem was first pushed to RubyGems.org on July 11th. Later on July 27th with slight changes in the extconf.rb gem installer file the cubrid-0.61-x86-mswin32.gem has been refreshed. Then, we have created the detailed tutorial and API documentation for the Connector. And finally announced to the community about these changes.

Later we received some user comments to host the CUBRID-Ruby source code on Github. After some conversation we decided to create Github account and upload the source code of the connector. On August 9th, 2010 we pushed cubrid-0.64.gem to Rubygems.org which is now cross-platform, i.e. can be installed both on Linux and Windows. Use the following command in the terminal to install the CUBRID gem for Ruby.

Java Tutorial

The main database connector for CUBRID is the JDBC Driver. Most external applications like CUBRID Manager connect to CUBRID through JDBC. To help users get started with Java Programming for CUBRID faster, we have created a tutorial with several detailed examples users can just copy and paste to get them up and running. Beside that there are numerous other manuals and documentations CUBRID users can refer to when building applications around CUBRID.

Python Programming with CUBRID

Python is an established programming language. It has been used by many professional web developers worldwide to create sophisticated web applications. Now Python developers can use the CUBRID-Python Connector to create the Web applications which use CUBRID as a back-end database management system. We encourage users to download the Python Library and read the API Documentation and Tutorial.

CUBRID API on PHP.net

This is probably one of the most important moves we make toward making the CUBRID Database more open to global users. We have contacted with the PECL members and received the approval for a new account for CUBRID. After several comments from PECL users we have made quite a lot modifications to the PHP API for CUBRID. Entire code has been reviewed and rewritten to comply with latest Zend Engine 2 API.

• Added more than 20 new APIs, see the PHP API Documentation for more details.
• Added two new schema types for cubrid_schema: CUBRID_SCH_DIRECT_SUPER_CLASS and CUBRID_SCH_PRIMARY_KEY.
• More data types support for cubrid_bind, the following types are added: CHAR, VARCHAR, BIGINT, MONETARY, VARBIT and DATETIME.
• Fix a bug in cubrid_get: If parameter attr is given and its type is string, and the function returns successfully, then there will be memory leak.

On July 19th, 2010, we have announced the latest CUBRID 3.0 Beta release. Overall we have fixed 42 bugs, and introduced 62 new SQL syntax extensions and other enhancements. Now users can leave most of their MySQL compatible SQL statements as they are without any modifications. With CUBRID 3.1 and 3.2 we plan to reach 95% MySQL syntax compatibility, thus providing users not only the increased performance but also the ease of porting their Web applications to CUBRID. As a matter of fact, we have started porting the most popular Content Management System in the world - the Joomla! CMS. Below you can see the current status and the brief explanations about how we leverage the CUBRID MySQL compatibility and optimizations for Web applications to make Joomla CMS porting faster and easier.

Porting Joomla! CMS to CUBRID

These days we are porting the most popular content management system – the Joomla! CMS. Until now none of other database system providers managed porting Joomla to seamlessly work with their system. Previously there were attempts by Postgres and MS SQL communities but stalled before reaching the implementation phase, while Joomla porting for CUBRID already has the feasible results. We estimate that by the end of this month we might have the Joomla alpha release for CUBRID. The project contributors mention that Joomla will have higher performance on CUBRID because of CUBRID's numerous Web optimizations. At this moment we have completed several large tasks like SQL parsing, Joomla Installer. Only a few details remained to get polished. Hope soon we will have another tool to make our users even happier!

You can join the discussions forum of Joomla for CUBRID at SF.net

https://sourceforge.net/projects/cubrid/forums/forum/1030991/topic/3803020

As PHP is something almost all websites are being created in, in this article you will learn how to create PHP applications with CUBRID Database running on the back-end.

This video is a part of Development Series of CUBRID Trainings Channel. You will learn what kind of support CUBRID provides for developing Web applications in PHP, how to install and configure PHP Extension, which is used to establish connections with CUBRID Database. Additionally, in this video tutorial you will see the examples how to request the data from CUBRID Database, how to fetch them, and display in the browser. Here I want to remind that last week CUBRID 2008 R3.0 Beta version has been released, which brings lots of MySQL Compatible features such as identical SQL syntax, operators, and same SQL embedded functions. Now on users do not have to worry much that they need to change the entire SQL queries to port their Web applications to work with CUBRID Database. Now almost all your code remains as it is. For more, read How to Port MySQL compatible PHP apps to CUBRID.

In the continuation of this video tutorial, you will learn where to download the PHP Library for CUBRID as well as its source code, how to enable CUBRID Database Extension in php.ini configuration file. As an example you will see how to verify that PHP extension has successfully been configured. We spent sleepless nights to ensure that our users have the great ease-of-use and convenience in developing applications with CUBRID Database. Thus, connecting to CUBRID database is as easy as replacing the "mysql" prefix in "mysql_connect" function to "cubrid", as a result calling cubrid_connect.

The video will last less than 10 minutes, so afterwards we highly recommend you to visit PHP Programming with CUBRID page we have specifically created for PHP Developers. This is the primary page you will be looking for PHP tips and links you will need throughout your PHP Development with CUBRID Database.

See the related blogs:

• PHP from Zero to Hero!
• Installing WordPress for CUBRID
• phpMyEdit for CUBRID

In order to successfully install and use most Web applications like well-known Wordpress or phpBB, it is required to have PHP installed besides database and web servers. Below is a list of these step by step tutorials. Let's get started!

1. There are three software we have to install before we can use any application written in PHP. One is a Database Server. Second - Web Server. Third is PHP. Since you are going to run your website with some web applications, it is recommended to install CUBRID as your Database Server. CUBRID is developed and highly optimized for web applications. It is exactly what you need.

   For this purpose we have create very easy-to-understand quick tutorial which will explain where you can download CUBRID from and how to install it and prepare for the Web.

   Read Getting Started with CUBRID.

2. Second, you need to install a Web Server. In our case we will use most popular and widely used Apache HTTP Web Server. In this tutorial you will learn how to download, install and quickly configure your Apache Server.

   Read Preparing Apache Web Server.

3. Third, you need to install PHP engine itself. PHP installation is very easy, indeed. The latest version provides easy installation tool which will automatically adjust your Apache Web Server's configuration file to correctly run PHP.

   Refer to Install PHP article to get PHP installed.

4. At this moment you should have almost everything you need. However, there is one more thing before we get started with Web application's installation. In order to be able to connect to CUBRID Database Server using PHP, there should something like a bridge to let them communicate with each other. This bridge is called PHP Connector or CUBRID PHP Library. This type of PHP Connector is typical to every database server. In our case CUBRID PHP Library will provide different kinds of PHP functions which you can use to let your web application communicated with CUBRID Database Server to retrieve data from the database, insert, update, and accomplish lots of other tasks.

   The tutorial below will show where to download CUBRID PHP Library from and how to install it. In fact, there is no need to install. It's a matter of copying the Library file to PHP extensions directory. It will also explain how you can configure the library for your needs.

   Read Install PHP Library for CUBRID.

5. Once you install everything, you might want to test your PHP. In the tutorial provided below you will see several PHP examples which you can copy & paste, then run them. Adjust any code according to your needs.

   Read PHP Sample Codes.

6. If you have already developed your PHP application for other databases and would like to use them with CUBRID, you might want to port them. This tutorial will explain in details all the CUBRID conventions, data types and SQL syntax which are important to be followed when porting PHP applications. Here you will see practical examples of how to port MySQL compatible PHP applications to CUBRID.

   Read Port MySQL Compatible PHP Applications to CUBRID.

7. Besides porting PHP applications to CUBRID you will highly likely need to migrate your data from other databases to CUBRID. If so, we have written very comprehensive and at the same time very easy-to-understand tutorial for Data Migration. In addition, this blog provides a video lesson for the same tutorial which will show a step-by-step migration process.

   Read and watch Video Tutorial: How to use Apache DdlUtils & Scriptella to migrate databases to CUBRID.

8. All these tutorials are aggregated and explained in PHP Tutorial.

If you think there is something else we could have covered, please let us know. We will do our best to make your CUBRID deployment easy and fast!

As most of you know, CUBRID was born and has been actively developed and maintained in South Korea for over 15 years. This creates a question why very few people know about CUBRID? The reason is CUBRID DBMS has been developed and maintained only inside of Korea all these years. The support and documentation was available only in Korean.

Then, we became an Open Source database server engine provider two years ago, late in 2008. We built CUBRID system in English, Chinese, and Japanese. After that, CUBRID enters the year of 2010 with a big news. It announces it will go abroad and create a Global Open Source Community around CUBRID. We opened our global community web site recently on CUBRID.org. Added CUBRID.blog, CUBRID.wiki, and our new CUBRID.forum.

Please get registered and engage in our CUBRID Open Source Community Collaboration. We will do our best to keep track of all issues you enter here. We will open dedicated forum branches for each type of question, like MySQL Compatibility branch or PHP, or Web applications. If you have any suggestions, feel free to post your request/advice/need/issue in the appropriate forum branch. We will make sure our DBAs and our large Open Source Community of Korea gets to you and addresses your requests quickly.

Wish you all productive work and easy deployment with CUBRID!

Have fun with the new CUBRID forum!

Once a user has installed CUBRID database server on their machine, the first thing they need to do is to connect to the database server, and then create the necessary databases, table, and so forth. There are several ways for CUBRID users to establish the successful database connection. First, you can use CUBRID Manager, a GUI database administration tool. Second, there are various third party applications such as Eclipse QuantumDB plug-in which you can also use just like CM (CUBRID Manager) to administer your databases. QuantumDB connects to CUBRID using JDBC driver. To learn more about third party applications, read CUBRID Administration Tools. Third, and the classic way to create and manage the databases, alter tables and grand user permissions, and briefly everything else you can handle with CSQL – a command line SQL interpreter. So, today we will talk about CSQL.

What is CSQL?

As I have already mentioned CSQL is a command line tool to administer CUBRID database server and run SQL statements. It is a basic path through which a user can communicate with CUBRID.

Why CSQL?

CSQL is mainly used for the following two purposes:

1. To execute SQL commands.
2. As a path for receiving tasks performed by a DBA for administration.

While the first purpose can be satisfied using the various 3rd party tools, the second purpose can only be achieved using CSQL provided solely for CUBRID.

What is the difference between CSQL and CUBRID Manager clients, or other various 3rd party tools?

CSQL connects to the DB without going through the Broker. For this reason, any problems occurring in the Broker will not affect the CSQL connection at all.

Now, let's look at the basic approach of connecting to CUBRID using CSQL.

First, in order to connect to any CUBRID database we have to use csql command with a given user account (if any) and a database name.

Shell > csql database_name

CSQL provides many options you can use when launching it. For details on these options, see CSQL Startup Options.

Shell > csql demodb

This is how a user has to connect to demodb, a demonstration database provided by default in CUBRID DBMS with the public account (empty password). When you connect to a database, you will see the following window.

At any time in order to learn what command you have to run for in certain cases, you type ;help command. There are several rules when running CSQL commands.

1. All CSQL commands must come after a semi-colon (;). For instance, in order to request help, you type ;help.
2. After executing any query, go to a new line and execute ;x or ;ru to see the result.
   1. ;x deletes the query in the buffer after execution,
   2. while ;ru leaves the query in the buffer to be reused later after the execution.
3. To reuse queries, use ;edit. This will start a vi editor, where you will see the queries that have been executed previously.

CSQL is a very powerful command line interpreter. You are recommended to read our online Wiki CSQL Interpreter Tutorial. You will learn about reading, saving, and appending SQL statement, Shell command execution/registration, transaction processing, outputting schema information/syntax rules and examples, parameter setting, etc, which are beyond those provided by CUBRID Manager or other simple query editor tools.

If you have any questions regarding CSQL, please post them on our CUBRID Forum.

If you want to be up to date and learn more about CUBRID, subscribe to our CUBRID Blog or CUBRID Trainings Channel, follow us on Facebook or Twitter.  We will make sure you are up to date with every news and updates about CUBRID.

Today is a Big day at Naver.com, Korea's No.1 Search Portal with 34 million subscribers and 17 million daily unique visitors. Today Naver launches its new Online Word Processor Service backed up by CUBRID Database Server in an Open Beta state. The Word Processor service runs on CUBRID 2008 R2.2 with High-Availability feature ON. There is an approximate estimate that CUBRID Database will process several million batch requests every day.

NaverWord is now available within Naver's most famous nDrive Web Service for storing and managing personal files. Users can now create Word Documents with advanced Tables, Charts, and Images, store them online, and access them from almost anywhere they want, including iPhone and Android phones, PC, Mac, and iPad.

Another famous online service which uses CUBRID as a back-end database management system is A-War Online Game. The below is a Game Trailer created for a Grand Open Ceremony day.

A short notice!

He says that "when choosing a DBMS for critical Web services, they do not go for the reputation or the brand". Mr. Kim emphasized on the "importance of the performance, of how the solution can be optimized for their services". "CUBRID has many features designed specifically for the Internet services, which is why we chose CUBRID," says Mr. Kim Kee Jeong, a team leader at NHN I&S.

To see the full text of the interview, visit http://www.cubrid.org/interview_nhn_information_services.

In 2009, VMware acquired SpringSource, "the innovator and driving force behind some of the most popular and fastest growing open source developer communities, application frameworks, runtimes, and management tools", for approximately $362 million in cash and equity plus the assumption of approximately $58 million of unvested stock and options.

There were quite hot discussions around why VMware would acquired Spring. It seems that it's high time to look into the relationship between Spring and Cloud, as Cloud computing is on the rise.

Spring and Cloud

In the same news release SpringSource, the developer of the Spring framework, and VMware, its parent company, claim that Spring is a technology suitable for the Cloud environment. Currently, Cloud platforms related to Spring include vFabric, Cloud Foundry, VMForce and Google App Engine.

• vFabric

  A solution for private Cloud offered by VMware, which provides a technology stack consisting of, among other things, the virtualization solution of VMware and the middleware, framework and development tools of SpringSource.

• Cloud Foundry

  A Cloud service operated by SpringSource, which uses Amazon’s AWS and provides middleware including Tomcat.

• VMForce

  A Cloud service offered jointly by VMware and Salesforce.com, which uses Force.com database and encompasses VMware’s solutions such as Spring, Tomcat and vSphere.

• Google App Engine

  As the result of a partnership between Google and VMware, SpringSource’s development tools now support Google App Engine. A plug-in is provided, which allows the development of an application that uses Google App Engine at Spring Roo.

At the SpringOne conference held in October 2010, the promotion of Cloud was notable. Spring was combined with diverse Cloud solutions, including vFabric, CloudFoundry, VMForce and even Google App Engine.  Two of the three platinum sponsors for SpringOne 2010 were Google and Salesforce.com.

"Spring is just an application framework. What does it have to do with Cloud?" - you would ask. "Aren't the core technologies of Cloud the virtualization and the distributed storage? Why would not Struts or Google Guice run in a Cloud environment instead of Spring, if JVM is correctly in place? These days, any technology that involves multiple servers is called Cloud. Isn’t Spring just one of these?" All we know at this moment is that these questions are not groundless.

Instead of dismissing such moves by major vendors as part of their promotion strategies, it will be useful in understanding the trend in technologies if we look into why VMware, Google and Salesforce.com are trying to accommodate Spring.

Last 10 years and profitable models

As it is widely known, Spring started as an alternative technology to the Java EE standard in the early EJB era.

Rob Johnson, the founding father of Spring, addressed the challenges in the early EJB era and developed his own framework, whose code was later disclosed in his book “Expert one to one J2EE development.” The code received such a positive response from developers that it became an open-source project. The Spring framework became popular enough for nine out of the ten top global banks to adopt it, and as a result, core developers founded a business entity to receive venture capital.

SpringSource must have thought a lot about its profit model since its foundation.  How can a mere application framework bring in money? The company once considered devising something called an enterprise service, in which minor patch versions would be offered to license purchasers.

SpringSource started to use M&A to secure products other than the framework. The first firm it had acquired was Covalent, where the core developers of Tomcat and Apache HTTPD are employed. The decision to acquire Covalent must have come out of a recognition that despite a pretext to integrate framework and middleware, a framework alone would not be enough for profit generation.

To everybody’s surprise, SpringSource itself was acquired by VMware for about 420 million dollars in 2009. The price was startling, as it was worth 530 thousand servers costing one thousand dollars each, but more startling was the buyer, VMware.  It was not a middleware vendor like Oracle or IBM, which Java developers are familiar with, but a virtualization solution vendor, which on the surface seems distant from Java.

Rod Johnson had looked for a business model with scalability. A business model based on consulting and training only would end up being a profit structure based on labor cost. So a model to sell products and services was required. Yet the big difference between the commercial version and the open source version would lead to criticism that the company was treating the open-source community poorly. SpringSource was offering Tomcat reinforced with tc Server, and if bugs are found in Tomcat that did not exist in tc Server, it would face huge criticism. Such restrictions in product selling probably made Rod Johnson decide to go for Cloud service, which is a scalable business model. That is why VMware was the buyer.

Apart from the business background, it is necessary to look into whether combining Spring and Cloud makes technological sense and what efforts had to be made for the combination.

Expansion of Spring portfolio and Cloud

Spring has picked up speed in evolution since its acquisition by VMware. Looking at projects, such as Spring-Android and Spring-Hadoop these days, it is hard to think of an area in which Spring has not had a hand.

Technologies associated with Cloud can be classified as follows:

First, there are technologies whose middleware for processing large volume was taken over from Spring directly. RabbitMQ, an AMQP (Advanced Message Queuing Protocol) based MQ solution, and GemFire, a datagrid technology, are two examples. Asynchronous processing and data cache will be more widely used in a Cloud environment, where large volume processing takes place frequently.

Second, there are technologies which provide API wrapping for distributed storage.

There are many projects for storage ongoing, under a project called Spring-data. Redis, Riak, CouchDB , MongoDB, Neo4j and more are already supported. There is no problem, of course, in using such storage without support from Spring. The objective of such projects is to ensure that the storage API facilitates convenient and consistent use of the API with Spring’s setting and programming method, just as Spring-jdbc is used instead of the direct use of JDBC. Storage APIs will not be as inconvenient as JDBC, but there will be some room for improvement, including a reduction in repeated Try-catch caused by Template-callback pattern.

Cache abstraction in Spring 3.1 facilitates the convenient use of distributed cache solutions. It allows designation of the objects to cache and the keys with Annotation, which is similar to the transaction processing method. Implementations for EhCache are provided by default, and diverse caches can be used to this specification once tuned to the SPI (Service Provider Interface).

Third, there are monitoring technologies. In a Cloud environment, many server and middleware instances are actually used, even if the user does not pay much attention to the physical server. The use of resources may need to be checked, since you only pay for what you use. A quick understanding of the use pattern is also useful in determining the cost and benefit. That is why monitoring is much more highly emphasized than in traditional systems. SpringSource’s acquisition of Hyperic, a monitoring tool, and development of tools like Spring-insight, can be understood in such context. In Spring-insight, Spring-specific information including Controller information for Spring MVC is provided, which is probably to highlight the benefits realized with Spring.

MQ data storages, cache and monitoring technologies can be installed and used in the existing environment, but can also be included in the technology stack offered as part of a Cloud service. Default solutions are an important part of the competitive edge, even in Cloud service. Amazon’s Cloud includes SQA (Simple Queue Service) as MQ and SimpleDB as storage, while Google App engine has Memcache as cache and BigTable-based DataStore as storage. vFabric, a Cloud solution offered by VMware, uses Gemfire for session clustering for the tc Server, and carries out monitoring with Hyperic.

A large portion of the recent expansion of Spring is related to technology investment in line with the cloud era.

Spring and Cloud portability

At SpringOne 2010, there was a demonstration in which an application running in VMware’s Cloud was run in Google's App Engine in exactly the same way. This is not surprising in a way, considering the portability of JVM. What is the implication of this?

One who intends to adopt Cloud should take into account what will happen over time. For example, an application may need to be transferred from a legacy server to Cloud, from Cloud to another Cloud, or from Cloud to a non-cloud. The greater the part of the application is dependent on a specific running environment, the more burdened the owner of the application will feel. Tying an application to a specific Cloud may seem to be an advantage for Cloud vendors, but it will hinder expansion of the user base because it can deter prospective users from adopting Cloud in its early stages. Existing applications will not be uploaded if it is expensive to move to Cloud. The user will be at a disadvantage in negotiation over pricing if it is difficult for him to exit even after moving-out. Consequently, when trying to expand the market, it will be to the advantage of Cloud vendors to emphasize that Cloud users can upload existing applications with minor modifications, and move out easily.

Meanwhile, will the Cloud environment accommodate existing applications as they are? This is not an easy task even for WAS, which Web applications are loaded onto. In Cloud services, only a small number of WASes are available. Google App Engine uses a variant of Jetty and Cloud Foundry, VMForce and vFabric, all of which are operated by VMware, and needless to say, use tc Server, a commercial version of Tomcat.  Moreover, JVM and WAS used in Cloud have limitations in their functions.

In Google App Engine, no file can be written directly, and no thread or socket can be created. In Servlet spec, the method used to call ServletContext.getNamedDispatcher and find out the name of the default Servlet does not work properly, either. All of this means that elements with the risk of security breach or abuse are not supported. When a legacy system is transferred to a Cloud environment, it is highly likely that the two WASes in the legacy system and in the Cloud are different. This is even more likely if the existing WAS is a Java EE server. Moreover, the function to call from WAS itself or the application will be restricted as well.

Here, Spring can be a big help for Cloud vendors. It can compensate for the weakness of a Cloud service, which offers WAS that supports WAS designated for the Cloud user, especially one that supports Servlet spec only. For example, life cycle management, dependency injection and declarative transaction for objects can be written with Tomcat only when there is no server that supports the JavaEE spec. Even when a JavaEE server is available, the data source and transaction service, among others, offered by the server, can be used by way of Spring. Indirect use of Java EE spec via Spring will require only minor modifications to applications at the time of switching to Tomcat or Jetty as a new WAS.

For example, you just need to change data source lookup with JNDI to DBCP, and instruct the Bean declaration to use DataSourceTransactionManager instead of JtaTransationManager. For this reason, applications using Spring have higher inter-WAS portability and a wider choice of WAS. It can be said that Spring acts as a buffer zone for WAS.

At this point, one can argue that though it is true that Spring reduces dependency on middleware such as WAS, it increases dependency on the framework itself, which in turn will reduce portability. Given such two options only, I would choose dependency on the framework rather than on WAS. A framework can co-exist with others on top of one single WAS, making it easy to change gradually. It also makes sense to have applications dependent on a framework instead of WAS in terms of flexibility. This is because version upgrade or library change for framework is easier than upgrading WAS. When it comes to changing jar files, for example, framework upgrade is done simply by changing a few lines of version declaration at pom.xml of Maven. WAS upgrade, in contrast, is burdensome compared to framework upgrade unless there is the automatic installation of AWS in place. In addition, Spring offers many methods that allow the writing of codes not dependent on it. A standard annotation such as @Inject is an example. The proper use of these methods can help reduce dependency on framework.

Salesforce.com, Google and VMware, the owner of SpringSource, all emphasize Cloud portability through Spring. The reason they speak with one voice while competing with each other is probably that for now, the initial market expansion is the most important priority. They also seem to share a belief that allowing transfer among services, instead of binding users to their own platforms, will be more beneficial in the long run. It can also be understood that all of them are confident in their core competencies, including infrastructure technologies.

Spring as a technology portal

Another reason why Spring and Cloud are linked is that Spring acts as a sort of portal in the Java technology ecosystem.

Internet portals receive a lot of visits from users and provide consistent UX for them by combining diverse information. CPs (Contents Providers) and portals enter into partnerships for contents sharing because the former can have an opportunity to make its contents widely known while the latter can bring in more visitors by offering them richer contents. User can access a wide variety of contents via portals’ consistent access routes and UX. For example, oil prices available on Naver.com, Korea's #1 search portal operated by NHN, are in fact provided by Opinet operated by Korea Oil Corporation, but users can easily access the information in the same way and style as other types of information.

Likewise, Spring is a technology used by many developers and offers a consistent setting method, annotation rules and API style by coordinating various technologies. Just as the user interface of a portal is what the user sees, the API at framework and library is the interface developers see.  Companies with component technologies who partner with Spring will have a chance to increase their user base by making their technologies more widely known, just as CPs do. SpringSource, on the other hand, can benefit from a richer technology ecosystem related to Source.

Spring provides flexibility by abstracting diverse technologies into the same API, or organizing them into a similar style to reduce the initial learning cost for developers. For example, GemfireTransactionManager, which is adapted to Spring’s DB transaction management interface can be offered in Gemfire, a datagrid. Even for recently added projects like Spring-amqp, the class names, interface names and method names look familiar to anyone who knows Spring well.

VMware’s acquisition of SpringSource, partnership with Salesforce.com and Spring’s link storages like Neo4j and Terracotta and cache technologies work in some respect to pursue technological synergy. These events, however, could happen because Spring was already widely-known enough to attract people. The acquisition of SpringSource seems to have done a lot to raise awareness, since VMware is now far more familiar to Java developers. Now what needs to be done is to go beyond raising awareness to offering a synergy that users can feel.

Non-intrusive design and the Cloud era

Obviously, Spring was not intended for the Cloud era when it was first created. The rapidly changing environment requires the adaptation of Spring as well, and it has received plenty of love letters from Cloud vendors so far.  The popularity of Spring is partly because it is well-known, and thus useful for publicizing. But mostly, it is the technology’s flexibility and portability that is deemed valuable, even in Cloud, which have attracted such plentiful investment and partnerships.

One of the fundamental reasons for Spring’s popularity is the technology’s design philosophy of ensuring that infrastructure related codes and business logic codes are not mutually intrusive.  Spring’s POJO (Plain Old Java Object) approach ensures that Java codes not dependent on any conventions are used for core logic, so that codes less dependent on the running environment are created.

For example, infrastructure-related codes, such as codes for transaction processing, can use conventions like JTA (Java Transaction API) provided by WAS. Putting codes dependent on specific middleware together in a single place will require less modification when the middleware is replaced.  In Spring, JTA-neutral transaction API (PlatformTransactionManager) is created and used via AOP to get codes using JTA together in one place, so that the modification of just a couple of lines can allow a switch to another method. As another example, Spring’s security can be used for logon verification in Google App Engine, probably because the part dependent on the infrastructure for user data storage is well abstracted.

Design must take into account, first of all, how to minimize modifications when there is a need for change in the system, instead of calling it "Cloud Portability." Design with clearly distinguished roles and responsibilities goes beyond time and environments. Eventually, flexibility and portability are the results of good modularization, and do not necessarily require special technologies. Spring’s AOP, Dependency Injection and so on only exist to support this. Before and after Spring, modularization is important, and the power behind Spring’s prosperity in the Cloud era can be attributed to having honored such universal design principles.

How different will application development be in the Cloud era? Infrastructure, like the storages and middleware required, will be a lot different. Adaptability to new components introduced into Cloud will determine productivity, at least in the early stage. Still, I don’t think the way of developing applications will be significantly different. A well-modularized code with clear roles and responsibilities will survive the new wave and exist even beyond the era, causing lower cost of modification and addition. This will be the case for whatever code is used, whether it is a framework code or application code written by an app developer.

How can one demonstrate that he or she has created a code with clear roles and responsibilities? Just write a test code. It will be difficult to write a test code if the code for infrastructure management and the code for business rules are mixed. Such a code will required more and more costs for modification, and eventually will not survive. Use of a specific framework does not necessarily lead to a good design. The best way to determine whether Spring is optimally utilized will be to see whether you write a code that allows the easy writing of a test code.

Conclusion

The increased investment by SpringSource in Cloud-related technologies was part of its plan to identify a scalable model, and has already produced some tangible results through a reinforced portfolio and partnerships. Spring can be an asset for Cloud that provides limited WAS due to its scalability, and is recognized for its role as a portal for a technology ecosystem with a wide user base. However, the fundamental reason for the prosperity and adaptation of Spring in the Cloud era is that it is a technology that allows the separate design of application core logic and infrastructure.

There will be a rush of news about Spring throughout the year, and most of it will be related to Cloud. I hope their strategies will unfold in a way that benefits all participants in the Java ecosystem.

By Jung Sang Hyuk, Manager, Productivity Innovation Team, NHN Business Platform Corp.

These days many websites (in fact, millions) are implemented by the use of popular open source content management systems (CMS). I would say, for everything you would imagine your site doing, there is one or another CMS which can do that. Thus, the reason for choosing a particular one depends on the tasks you want to accomplish. For blogs there is a common sense of using WordPress blog management system, for large scale websites Drupal CMS is perceived to be a better fit, while Joomla CMS is easier to learn for newbies. .NET developers prefer alternative DonNetNuke or umbraco CMS. Besides these, CMS Made Simple and Liferay are amongst the most popular content management systems based on the number of the number of downloads. Elgg and MODx are amongst the rising stars.

I will not tell you what a CMS is. If you want to learn, see the CMS Wikipedia page. In this blog I will give a comprehensive overview of top 14 content management systems based on the number of weekly downloads, installations, and brand familiarity. The full statistical data can be found in the 2010 OPEN SOURCE CMS MARKET SHARE REPORT by water&stone (2010). The report is distributed under the Creative Commons Attribution-Noncommercial License (3.0).

Today we will cover Drupal, Joomla, MODx, WordPress, DotNetNuke, umbraco, Liferay, TYPO3, CMS Made Simple, MOVABLE TYPE, Plone, eZ Publish, concrete5, and Alfresco content management systems. You might already be familiar with some of these, while some can be new to you. I bet you will be surprised to learn that some unknown for you systems suit your requirements much better than those you have got to use so far (my personal experience). So, let's learn about each of them, and do not forget to leave your comments below on what other CMS should have been included in this list and why.

Note: the order of the CMS does not represent its quality, performance, or superiority over the other. They are numbered to give a better perception to readers. The users feedback and opinions are retrieved from Stack Overflow, thus can be subjective as they are based on personal views and preferences.

# 1: Drupal

Official site: http://drupal.org/

1% of all websites on the Internet are based on this platform. An estimated 7.2 million sites were powered by Drupal as of July 2010. Drupal started out in 2001. In one year from May 2007 to April 2008, Drupal was downloaded from their official site more than 1.4 million times, an increase of approximately 125% from the previous year.

Weekly downloads: 33,671 (ranked #3 after Joomla and before DotNetNuke).

Installations: 575 according to the survey (#3 after WordPres and before DotNetNuke), but 1.4% (#3 after Joomla and before Typo3) of Alexa Top 1 million sites.

Brand Familiarity: Drupal is known to be the #3 most familiar content management system.

Major Features:

• You can manage multiple sites with Drupal in multiple languages. You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine.
• You can easily manage your site users, providing standard registration, including OpenID support. You can set various access control rules to limit the activity of your site users.
• Provides multiple-level menu system, template customization, advanced search, RSS feed aggregator.
• Officially Drupal supports several databases including MySQL, PostgreSQL, MariaDB, and SQLite.
• To increase its performance, you can use caching. At the same time it provides high security with notifications about the new update releases.
• Provides Search Engine Friendly descriptive URLs.
• Powered by jQuery JavaScript framework.

Distinguished Clients:

• http://www.whitehouse.gov/
• http://data.gov.uk/
• http://www.ubuntu.com/
• http://www.alrc.gov.au/
• http://www.nysenate.gov/
• http://london.gov.uk/
• http://www.rutgers.edu/
• http://www.economist.com/
• http://wfp.org/

• More difficult to master especially for newbies. It's more for advanced users. Though the new Drupal 7 claims to provide significantly improved usability (maybe more toward WordPress style). To achieve this, they hired web designers to specifically address the UX problems it had in previous versions.
• New major releases are not quite backward compatible. More focus on new features and functionality. External module developers should take care of compatibility themselves, except for data representation, which Drupal is intended to keep same.
• Drupal is often compared with Joomla and is perceived to be a bit slower. Though this depends on the type of website you develop. However, leveraging its caching and gzipping technologies, it is possible to achieve quite impressive results.
• jQuery, the default JavaScript framework in Drupal, allows to use alternative Mootools framework at the same time.

# 2: Joomla

Official site: http://www.joomla.org/

Joomla is the result of a fork of Mambo CMS on August 17, 2005. Within its first year of release, Joomla had been downloaded 2.5 million times.

Weekly downloads: 113,836 (ranked #2 after WordPress and before Drupal).

Installations: 1,297 according to the survey (#1 before WordPress), but 2.5% (#2 after WordPress and before Drupal) of Alexa Top 1 million sites.

Brand Familiarity: Joomla is known to be the #1 most familiar content management system.

Major Features:

• You can manage multiple site with Joomla in multiple languages natively (since Joomla 1.6). You can use it for blogging site, corporate site, personal site, gallery, briefly, whatever you imagine.
• You can easily manage your site users, providing standard registration, including Google OpenID support. Full support for Access Control List.
• Provides Multiple-level menu and content category system, template customization, advanced search, RSS feed aggregator.
• Officially supports only MySQL.
• Page cashing for increased performance.
• Provides moderate descriptive URLs (still not fully customizable as you can do in WordPress).
• Powered by MooTools JavaScript framework.

Distinguished Clients:

• http://www.linux.com/
• http://www.itwire.com/
• http://www.quizilla.com
• http://www.ihop.com
• http://gsas.harvard.edu
• Citibank (Financial institution intranet) - Not publicly accessible
• http://www.greenmaven.com
• http://www.outdoorphotographer.com
• http://www.playshakespeare.com
• http://www.sensointeriors.co.za

• More intuitive and easy to use than Drupal, though still not like WordPress.
• Powerful. Fully-fledged content management system, so you can create whatever site you want.
• Really strong security. If security problems found, immediately fixed.
• The new Joomla 1.6 release is expected to be faster, more convenient, with more features.
• Its strong dependency on Mootools JavaScript framework sometimes bothers users as Joomla does not give easy workaround to disable it and use jQuery instead.
• Support of only one database, limits Joomla a lot in terms of the number of users. However, this is a compromise for high optimizations for MySQL, thus increased overall performance.
• Does not allow to fully customize URLs - a must feature for CMS.

#3: modx

Official site: http://www.modxcms.com/

modx is not just an open source CMS but also a web application framework. Raymond Irving and Ryan Thrash began the MODx CMS project in 2004 as a fork of Etomite. In 2008 MODx users created a new logo and branding for the project. Now MODx allows for full segregation of content (plain HTML), appearance and behavior (standards compliant CSS and JavaScript) and logic (PHP, snippets).

Weekly downloads: 4,500 (ranked #11 after umbraco and before Tiki).

Installations: 58 according to the survey (#12 after eZ Publish and before umbraco), has less than 0.1% among the Alexa Top 1 million sites.

Brand Familiarity: #14 (before Liferay and after eZ Publish).

Major Features:

• As with Joomla, modx officially supports only MySQL database.
• Not just CMS but a PHP framework for Web.
• Freedom to choose jQuery, Mootools, ExtJS, Prototype or any other JavaScript library.
• Supports PHP 4.3.11 and above.
• Complete control of all metadata and URL structure for SEO (Search Engine Optimization).
• Unlimited hierarchical page depth.
• Can create custom fields and widgets for templates.
• Role-based permissions for the Manager.
• Ability to customize the Manager on a per-deployment basis.
• Ecommerce integration via Foxy Cart.

Distinguished Clients:

• http://www.pippatoledo.com/
• http://www.aquevix.com/
• http://www.not1bug.com/
• http://everlight-uva.com/
• http://www.tritopora.ru/
• http://www.strategische-webloesungen.de/

• Good to have a choice for favorite JavaScript framework.
• Light CMS solution (but not necessarily the fastest).
• PHP 4 support for developers mean that a lot of compromises had to be made in terms of OOP (Object Oriented Programming) in order to offer PHP 4 support .
• Nice to have freedom to set custom URL.

#4: WordPress

Official site: http://wordpress.org/

WordPress was first released on May 27, 2003, by Matt Mullenweg as a fork of b2/cafelog. As of August 2010, version 3.0 had been downloaded over 12.5 million times. Nowadays, known as the #1 CMS for blogging.

Weekly downloads: 983,625 (ranked #1 before Joomla).

Installations: 1,012 according to the survey (#2 after Joomla and before Drupal), but 12.9% of the Alexa Top 1 million sites (#1 before Joomla).

Brand Familiarity: #2 (after Joomla and before Drupal).

Major Features:

• Highly optimized for blogging.
• Custom and easy to switch themes.
• Users can re-arrange widgets without editing PHP or HTML code.
• Official support for only MySQL.
• Custom URL, clean permalink structure, excellent for SEO.
• Nested, multiple categories to articles.
• Support for tagging. Advanced search by tags.
• Highly intuitive UI (User Interface).
• jQuery JavaScript framework.
• Supports the Trackback and Pingback standards for displaying links to other sites that have themselves linked to a post or article.
• Rich plugin architecture which allows users and developers to extend its functionality beyond the features that come as part of the base install.

Extensions: 12,780 plugins and 1,315 themes.

Distinguished Clients:

• http://www.nytimes.com/interactive/blogs/directory.html
• http://tmagazine.blogs.nytimes.com/
• http://crowdfavorite.com/
• http://blog.broadband.gov/
• http://blogs.america.gov/
• http://www.number10.gov.uk/
• http://www.speaker.gov/
• http://allthingsd.com/
• http://politicalticker.blogs.cnn.com/
• http://stylenews.peoplestylewatch.com/

• Perhaps, the most convenient, easy to use and intuitive CMS (or BMS) in the world. Perfect for blog sites. But if you need to develop a dynamic site with various components, perhaps, other CMS would fit better, though, WordPress provides enough plugins to accomplish almost all tasks.
• I would avoid Wordpress as a CMS in a professional environment. As stated earlier, it's a great blogging platform, but doesn't generally offer the robustness that most professional environments require.
• Availability of jQuery makes the plugin development a lot easier for external developers and site owners.
• Endless themes - no need to worry about the new design for your site, unless you really need something special.

#5: DotNetNuke

Official site: http://www.dotnetnuke.com/

DotNetNuke is an open source platform for building web sites based on Microsoft .NET technology. It is written in VB.NET and distributed under both a Community Edition BSD-style license and a commercial proprietary license. The Community Edition is a popular web content management (WCM) system and application development framework for ASP.NET, with over 6 million downloads and 600,000 production web sites as of October 2010. More than 8,000 DotNetNuke apps are available for purchase on Snowcovered.com. DotNetNuke.com has over 800,000 registered members as of October 2010.

Weekly downloads: 13,000 (ranked #4 after Drupal and before CMS Made Simple).

Installations: 402 according to the survey (#4 after Drupal and before Liferay), but 0.2% of the Alexa Top 1 million sites (#4 after Typo3 and before MOVABLE TYPE).

Brand Familiarity: #4 (after Drupal and before Typo3).

Major Features:

• Distinguishes between community (common features) and enterprise (full set of features) editions.
• Various modules, and data providers.
• Provides language packs for about 60 languages.
• Customizable through skins and templates.

• http://www.chamberlain.edu/
• http://magenic.com/
• http://www.graphiksolutions.ca/
• http://www.marly.com/
• http://www.dotcomsoftwaresolutions.com
• http://www.cityplaceascent.com/
• http://sites.kiwanis.org/kiwanis/en/home.aspx
• http://www.zonediet.com/

• A little bit difficult to create modules.
• It is perceived as a little bit bulky CMS.
• Does not provide extensive documentation and user guides.
• Similar to most CMS, does not provide full backward compatibility in its new major releases.
• Unlike its enterprise edition, the community edition is not tested and certified by the DotNetNuke Corporation.
• Even if it provides the language packs, the sites cannot be created to support multiple languages. Third party module should be used to enable this feature.
• Auto-upgrade, Advanced Site Search, Page Cashing, and many must-have features are not included in the Community edition, only in Professional or Enterprise editions.

#6: Umbraco

Official site: http://umbraco.org/

Umbraco is also an open source content management system. It was developed by Niels Hartvig in 2000 and released as open source software in 2004. It is written in C# and can be deployed on Microsoft based infrastructure. In 2010, with about 1,000 downloads a day, Umbraco was in the Top 5 most popular downloads via the Microsoft Web Platform Installer, two places below its main rival DotNetNuke.

Weekly downloads: 5,420 (ranked #10 after Alfresco and before MODx).

Installations: 57 according to the survey (#13 after MODx and before e107), but less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #16 (after Liferay and before e107).

Major Features:

• Can be deployed with several databases, including MySQL, SQL Server, and VistaDB.
• SEO-friendly URLs.

What users say:

• Limited number of extensions.
• Official support for Windows OS only.
• Umbraco is oriented to small low-cost sites.
• Video trainings have to be purchased.
• Many must-have features have to purchased.

#7: Liferay

Official site: http://www.liferay.com/

Liferay Portal is a free and open source enterprise portal written in Java and distributed under the GNU Lesser General Public License. It allows users to set up features common to websites. It is fundamentally constructed of functional units called portlets. Liferay is sometimes described as a content management framework or a web application framework. It comes with certain portlets preinstalled. These comprise the core functionality of the portal system.

Weekly downloads: 9,435 (ranked #6 after CMS Made Simple and before TYPO3).

Installations: 154 according to the survey (#5 after DotNetNuke and before TYPO3), but less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #15 (after MODx and before Umbraco).

Major Features/ Portlets:

• Can tag and categorize contents.
• Document Library Manager, Recent Documents.
• Alfresco, Documentum, and other document library integration.
• User management based on various roles and groups (ACL).
• WebDAV Integration (Web-based Distributed Authoring and Versioning which allows users to collaboratively edit and manage files on remote web servers).
• Nested Portlets
• User Directory
• LDAP Integration
• Microsoft Office Integration
• Calendar/Chat/Mail/Message Boards/Polls
• Wiki (supports Creole as well as MediaWiki syntax)
• Alerts and Announcements
• Knowledge Base
• Social Equity
• Can create multi-language sites.
• Asset Publisher to publish many contents, tagged by a specific term, at once.

Distinguished Clients: It is primarily used to power corporate business sites.

• http://developer.cisco.com
• http://www.t-mobile.cz
• http://www.betavine.net
• http://www.foxchannel.de/
• http://www.ixarm.com

• It is mostly used by enterprise companies rather than for powering personal or community sites, though it has social and collaboration features.
• More professional developers driven project (backed by Liferay Inc.) rather than the community driven.
• Provide paid Enterprise edition.
• All features are available in Community edition (except for support and customization related).

#8: Typo3

Official site: http://typo3.org/

TYPO3 is a free and open source CMS released under the GNU General Public License oriented to small to mid size enterprise-class users. TemplaVoila is an alternative template engine extension for TYPO3. A graphical mapping tool for creating templates is included, an alternative page module, the ability to create flexible content elements and an API for developers. New content element types can be created without programming. TemplaVoila facilitates more flexibility for maintaining web pages than TYPO3's standard templating, while making it possible to enforce a strict corporate design and allowing editors to work with content more intuitively.

Weekly downloads: 7,461 (ranked #7 after Liferay and before eZ Publish).

Installations: 122 according to the survey (#6 after Liferay and before Tiki), 0.6% of the Alexa Top 1 million sites (#4 after Drupal and before DotNetNuke).

Brand Familiarity: #5 (after DotNetNuke and before OpenCMS).

Major Features:

• Supports MySQL, Oracle, MS-SQL, PostgreSQL, ODBC, LDAP - virtually any external data source.
• You can undo any change you make on the site.
• Can create multiple sites with multiple domains for each.
• Can have multiple template per site.
• User management.
• Able to switch from administrator user to general user to check permissions.
• Sandbox: administrators can set up a section within the system to test new features without disturbing the main site.
• Versioning of content pages.
• Advanced caching: template, navigation or page level.
• Link management.
• Multi-language content.
• Search Engine friendly URLs.

Distinguished clients:

• http://www.volkswagen-ir.de
• http://www.eadssecureuk.com
• http://www.geveriwise-eu.com
• http://www.rewe-xxl.de

• The administrator's UI (User Interface) is not user-friendly, though intuitive and functional.
• It is extremely powerful which provide more enterprise level features, but the learning curve is incredibly steep.
• Great ease of multi-lingual site management.
• Difficult to customize templates. Need to learn TypoScript and TemplaVoila, two TYPO3-specific systems.

#9: CMS Made Simple

Official site: http://www.cmsmadesimple.org/

CMS Made Simple is an open source cms built using PHP with support for MySQL and PostgreSQL. The template system is driven using the Smarty Template Engine.

Weekly downloads: 9,948 (ranked #5 after DotNetNuke and before Liferay).

Installations: 72 according to the survey (#7 after Tiki and before Alfresco), 0.1% of the Alexa Top 1 million sites (#8 after Xoops and before eZ Publish).

Brand Familiarity: #12 (after Xoop and before Ez Publish).

Major Features:

• Officially supports MySQL and PostgreSQL.
• Search Engine Friendly URLs.
• Users management and group based permissions system.
• Content tagging.
• Site localization is available in 20 languages.

What users say:

• It is oriented more toward professional users.
• It does not provide many templates, so coding knowledge is a must.

#10: Movable Type

Official site: http://movabletype.org/

Movable Type is a weblog publishing system, similar to WordPress, developed by the Six Apart company in Perl programming language. At various times, this company maintained three other publishing systems - TypePad, Vox, and LiveJournal. Movable Type was publicly announced on September 3, 2001. Version 1.0 was publicly released on October 8, 2001, thus it is a blogging system older than WordPress. On 12 December 2007, Movable Type was relicensed as free software under the GNU General Public License. Based on the list of its customers, Movable Type is quite credible CMS.

Weekly downloads: unavailable.

Installations: 30 according to the survey (#17 after Silverstripe and before OpenCMS), 0.1% of the Alexa Top 1 million sites (#6 after DotNetNuke and before Xoops).

Brand Familiarity: #8 (after Plone and before Alfresco).

Major Features:

• Convenient blogging system with social community features.
• Since version 5 officially supports only MySQL. PostgreSQL and SQLite can be used via plugins. Databases such as Oracle can be integrated with Movable Type Enterprise edition.
• Manage user roles. OpenID support.
• Multiple site hosting.
• Easily customizable templates.
• Revision history.
• Can add custom fields.
• Content tags and categories.
• Feeds and trackback links.
• Available localization and internationalization.
• Can generate static pages (updated whenever the content of the site is changed).

Distinguished clients:

• http://www.britneyspears.com/
• http://www.barackobama.com/
• http://boeingblogs.com/
• http://blogs.oracle.com/
• http://gehealthcare.typepad.com/
• http://www.spd.org/
• http://blog.ted.com/

• Users often deploy Movable Type for their blog sites, newspaper or other type of publishing sites.
• A powerful alternative for WordPress.

#11: Plone

Official site: http://plone.org/

Plone, a free and open source CMS, started in 1999 by Alexander Limi, Alan Runyan, and Vidar Andersen. It was made as a usability layer on top of the Zope content management framework, thus Plone is written in Python. The first version was released in 2001. In 2004, Plone 2.0 was released. This release brought more customizable features to Plone, and enhanced the add-on functions. In 2007, Plone 3 was released. This new release brought inline editing, an upgraded visual editor, and strengthened security, among many other enhancements. Recently in 2010, Plone 4 was released with major improvements in performance.

Weekly downloads: unavailabe.

Installations: 34 according to the survey (#15 after e107 and before Silverstripe), 0.1% of the Alexa Top 1 million sites (#10 after eZ Publish).

Brand Familiarity: #7 (after OpenCMS and before Movable Type).

Major Features:

• Inline editing - no need to reload the page for editing.
• Localized into 40 languages.
• Plone can be integrated with Active Directory, Salesforce, LDAP, SQL, Web Services, and Oracle.
• Working Copy support - keep the old version of a content published until you publish a new version.
• Cut/copy/paste operations on content.
• Link and reference integrity checking - no more broken links within your site.
• Can create workflows - useful for organizations.
• LiveSearch - instant site search powered by AJAX.
• Full-text indexing of Word and PDF documents.
• Wiki support
• Collaboration and sharing
• Automatic locking and unlocking
• Versioning, history and reverting content
• Authentication back-end
• Collections
• Multilingual content management
• Automatic previous/next navigation
• Human-readable URLs
• Caching proxy integration
• Drag and drop reordering of content
• Adjustable templates on content
• RSS feed support
• Automatic image scaling and thumbnail generation
• Comment capabilities on any content
• WebDAV and FTP support
• Hot -backup support

Distinguished clients:

• http://www.amnesty.ch/en
• http://www.brasil.gov.br/
• http://www.chicagohistory.org/
• http://ccnmtl.columbia.edu/
• http://cnx.org/
• http://discovermagazine.com/
• http://www.ece.rice.edu/
• http://www.engagemedia.org/
• http://science.nasa.gov/

• It has a lot of features (several times more than listed above) built-in.
• To become a Plone expert is long and expensive. It has a steep learning curve.
• Plone is backed by Zope framework which is very powerful with support for caching, rollback, etc. - everything what your organization might need.
• Plone is really complex to deeply tweak.
• Plone is known for its high security.
• Plone runs slow if you don't know how to optimize it.

#12: eZ Publish

Official site: http://ez.no/

eZ Publish is an open source enterprise CMS developed by the Norwegian company eZ Systems in 1999 using PHP programming language. eZ Publish is freely available under the GPL licence, as well as under proprietary licenses that include commercial support. eZ Publish supports the development of customized web applications. Typical applications range from a personal homepage to a multilingual corporate website, which include role-based multi-user access, e-commerce functions and online communities.

Weekly downloads: 7,031 (ranked #8 after TYPO3 and before Alfresco).

Installations: 60 according to the survey (#11 after Concrete5 and before MODx), 0.1% of the Alexa Top 1 million sites (#9 after CMS Made Simple and before Plone).

Brand Familiarity: #13 (after CMS Made Simple and before MODx).

Major Features:

• Advanced Search feature is available in only Enterprise Edition.
• Online image editor.
• Can create building blocks for the site and later reuse them in several pages.

• http://www.schemexpert.com/
• http://www.ticotimes.net/
• http://jp.wsj.com/
• http://www.laborange.fr/
• http://canalstreet.canalplus.fr/
• http://www.hks.harvard.edu/
• http://www.vogue.com.au/

• The documentation is a bit limited especially there is almost no documentation for module development.

#13: Concrete 5

Official site: http://www.concrete5.org/

Concrete5 is an open source CMS started in 2003 as a rapid-design approach to building the now-defunct LewisAndClark200.org, the official site for the Ad Council's National Council for the Lewis & Clark Bicentennial. Concrete5 is developed in PHP and is distributed under MIT software license.

Weekly downloads: unavailable.

Installations: 62 according to the survey (#10 after Alfresco and before eZ Publish), has less than 0.1% of the Alexa Top 1 million sites.

Brand Familiarity: #20 (after TextPattern).

Major Features:

• Integrated server side caching.
• Support for only MySQL.
• Inline content editing.
• Image editing tool.
• Editable areas are defined in concrete5 templates which allow editors to insert 'blocks' of content. Additional blocks are available as add-ons.
• Automatic upgrade is available.
• Advanced Permissions to track content versions.

• http://www.genco.com
• http://www.cottonfrombluetogreen.org/
• http://www.cs.uh.edu/
• http://www.signals.ca/
• http://www.paulraymondgregory.com

• It's PHP based and quite new but it's quite a nice layout and it's really natural for new CMS users. You can go from a paper-based sitemap and PSD to a full site structure, ready for data entry, within a day, two at a push.
• Concrete5 is simple, suitable for creating sites quickly.
• Creating template is very easy with Concrete5.

#14: Alfresco

Official site: http://www.alfresco.com/

Alfresco is an open source enterprise content management system for Microsoft Windows and Unix-like operating systems. Alfresco includes a content repository, an out-of-the-box web portal framework for managing and using standard portal content, a CIFS interface that provides file system compatibility on Microsoft Windows and Unix-like operating systems, a web content management system capable of virtualizing web apps and static sites via Apache Tomcat, Lucene indexing, and jBPM workflow. The Alfresco system is developed using Java technology. John Newton (co-founder of Documentum) and John Powell (a former COO of Business Objects) founded Alfresco Software, Inc. in 2005.

Weekly downloads: 7,000 (ranked #9 after Ez Publish and before Umbraco).

Installations: 70 according to the survey (#9 after CMS Made Simple and before Concrete5) Alexa ranking is not available.

Brand Familiarity: #9 (after Movable Type and before Tiki).

Major Features:

• Document Management.
• Web Content Management (including full webapp & session virtualization).
• Repository-level versioning (similar to Subversion).
• Records Management, including 5015.2 certification.
• Repository access via CIFS /SMB, FTP, Web DAV, NFS and CMIS.
• j BPM workflow.
• Advanced search with Lucene.
• Multi-language support.
• Officially runs on Windows, Linux and Solaris.
• User Interface official supports Internet Explorer and Firefox.
• Desktop integration with Microsoft Office and OpenOffice.org.
• Clustering support.
• Pluggable authentication: NTLM, LDAP, Kerberos, CAS.

• France Air Force
• Harvard Business School Publishing
• Toyota
• Sony Pictures
• Fox
• National Academy of Sciences
• Cisco

• Alfresco is mostly for enterprises rather than for personal sites.
• Simple to install and use, flexible and open-ended.
• Alfresco is a solution with the broadest range of technical capabilities and the best feedback from users. In addition to demonstrating a promising roadmap for collaboration tools, Alfresco was highly attractive from a cost perspective, compared to the proprietary products offered by other ECM vendors.
• All in one solution for enterprises.

See also

Besides, the official home pages, you can refer to Wikipedia for general information on these content management systems. Also Stack Overflow provides very informative users feedbacks on each of these CMS.

1. http://en.wikipedia.org/wiki/Drupal
2. http://en.wikipedia.org/wiki/Joomla
3. http://en.wikipedia.org/wiki/Modx
4. http://en.wikipedia.org/wiki/Wordpress
5. http://en.wikipedia.org/wiki/DotNetNuke
6. http://en.wikipedia.org/wiki/Umbraco
7. http://en.wikipedia.org/wiki/Liferay
8. http://en.wikipedia.org/wiki/Typo3
9. http://en.wikipedia.org/wiki/CMS_Made_Simple
10. http://en.wikipedia.org/wiki/Movable_Type
11. http://en.wikipedia.org/wiki/Plone_(software)
12. http://en.wikipedia.org/wiki/EZ_Publish
13. http://en.wikipedia.org/wiki/Concrete5
14. http://en.wikipedia.org/wiki/Alfresco_(software)
15. http://www.waterandstone.com/sites/default/files/2010%20OSCMS%20Report.pdf
16. http://php.opensourcecms.com/
17. http://stackoverflow.com/