Open Source RDBMS - Seamless, Scalable, Stable and Free

한국어 | Login |Register

Current Events
Join our developers event to win one of the valuable prizes!
posted 2 years ago
viewed 8126 times
Share this article

Optimization and Distribution of Static Files on the Web Using Maven

maven-css-js.pngThe distribution of a Web application is as important as its development. Distribution methods can enhance the user experience or cause a failure. In this article, I will explain how we distribute static files at Knowledge Shopping Service department here at NHN

Difficulty of Distributing Static Files

When static files such as the CSS and JavaScript are allowed to be cached on the client side by a browser, users can experience an increased speed and the server can be relieved from the burden of serving those resources. However, once the content of these static files changes, instead of using files stored in the cache, the browser must download the latest versions from the Web server. The most common method used to achieve this is by adding a query string-type timestamp to the name of a static file like common.css?t=20110405. The timestamp value used here can represent either the build time of the file, or the start time of the Web server, or the revision number of a repository. However, these solutions may cause some issues. I will explain them below.

server-before-after-distribution.png

Figure 1: Server Before and After Distribution.

Inconsistent distribution

For example, if a user accesses a Web page and obtains an HTML file from Server A (post-distribution) but CSS files used in this HTML files are served from Server B (pre-distribution), the user will end up using the new HTML file and the old CSS file, which may result in broken layout of the page or site malfunction.

  1. To prevent this kind of problem, it is necessary to change the name of a newly distributed file making sure that the old HTML file refers to the old CSS file, and the new HTML file refers to the new CSS file.
  2. If several Web servers are used for file distribution, the static files may be distributed in advance to prevent CSS or JavaScript files from being distributed to only some of the servers.

By using the method above, most problems that can occur while distributing static files can be prevented.

Decreased Performance

However, if the number of static files referred by a Web page is large, the performance may suffer.

  1. For instance, in case a file has not been cached on the client side, it must be downloaded whenever it is requested.
  2. On the other hand, if a file has been cached, a "304 Not Modified" will be received whenever a user accesses the server, incurring a DNS resolve cost and a socket connection cost.
  3. Even when you merge the static files to reduce their number, unnecessary blanks and long variable names tend to increase the size of the transferable data.
  4. More than that, the size of the transferrable data will grow when static files are distributed to the Web server, as cookies included in an HTTP request message will be transferred as well.

Proposed Solution

Though it would seem tempting to have a single person manage all of the static files written during the development process, it often proves inefficient due to an increased management cost. Therefore, the distribution process must be automated to the greatest extent possible, and the following rules must be kept for the automatic and efficient distribution of static files.

  1. Add the timestamp value to the query string.
  2. Manage the versions of CSS JavaScript files.
  3. Reduce the number of files.
  4. Minify file content.
  5. Distribute CSS and JavaScript files from a cookie free domain.

Optimization by Using Apache Web Server Configuration

  1. Compress the static files using the deflate module of the Apache Web server and transfer the compressed file. Add the deflate module configuration to the Apache configuration file, as shown below:
  2. # Add the deflate module
    LoadModule deflate_module modules/mod_deflate.so
    
    # Compression level
    DeflateCompressionLevel 1
    
    # Set the compression level based on MIME type
    AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml text/javascript
    
    # Add the MIME type 
    AddType text/javascript .js
    AddType text/css .css

  3. Then, add the expire module to allow the browser to cache the static file.
  4. # Add the expires module
    LoadModule expires_module modules/mod_expires.so
    
    ExpiresActive On
    
    # Set the default expires to one year
    ExpiresDefault "access plus 1 years"

HOW TO manage static files

Assume that JavaScript files are located in /web/js directory and main.js and detail.js files referred in a Web page are located in that /web/js directory.

directory-structure.png

Figure 2: Directory Structure.

For an easy version control, mark each JavaScript file with its version (ex: main_v1.js, detail_v1.js).

How can I add a JavaScript file without describing the version and timestamp on each Web page?

When a Web application is loaded...
  1. ... scan the subdirectory files including /web/js;
  2. and create a list of files with the highest version;
  3. then mark the file path by using the custom tag as shown below:
    <link:js prefix="/js/main.js"/>

The following code will be displayed in the HTML:

<script type="text/javascript" src="/js/main_v1.js?t=2011040512"></script>

To add new content or modify the existing one that does not support backward compatibility, get a higher version (ex: main_v1.js  main_v2.js). The script will load the latest version.

What are the potential issues which can arise when using this solution?

If static files have been distributed in advance, all Web servers will have both main_v1.js file and main_v2.js file. Therefore, no problem would occur during the distribution.

On the Web server after distribution, a new Web page will be linked to a new JavaScript file (main_v2.js), and on the Web server before distribution, the existing Web page will be linked to the existing JavaScript file (main_v1.js).

During the Web page development process, the development versions of static files are used (external URL). What should I do in this case?

Modify the custom tag to describe the development URL like:

<link:js prefix="/js/main.js" devOnlyUrl="http://svn.nhndesign.com/service/main_20110404.js" />

While developing the Web application, use devOnlyUrl. After completing the development, use the highest version of the main.js file. Of course, there should be a way to set the development status of the file.

If a CSS or a JavaScript file is provided from a domain that does not use cookies, will there be any problem?

If a user-defined cookie is used in the JavaScript file, that file must exist in the path within the Web server. To achieve this, an additional setting such as needCookie is required in the custom page.

<link:js prefix="/js/cookie.js" needCookie="true" />

HOW TO merge static files

So far, I have described how to manage static file versions and how to describe files being developed. Now, I will explain how to bind the main.js file and the detail.js file to one core.js file to reuse their number. Write the configuration file as shown below:

<?xml version="1.0" encoding="UTF-8"?>
<compression>
    <file target="/js/core.js">
        <include>/js/main</include>
        <include>/js/detail</include>
    </file>
</compression>

How do I manage the versions of the created files?

  1. If you have bound main_v1.js and detail_v1.js files to create one core_v1.js file, and there is no change in those files, keep the file name as 'core_v1.js'.
  2. When the content or the version of the main_v1.js or the detail_v1.js file is changed, the core_v1.js file must be changed to the core_v2.js file. To do this, you must manage the name, size, and checksum of each file that is part of the file to be created.

version=1
encoding=UTF-8
file./js/main_v1.js=945,UTF-8,ac3fecacd0a6ecf74a0c711180582a8c
file./js/detail_v1.js=22572,UTF-8,ee2565f0938bfd1baa21c73afa633c81

Now, you should minify the bound files. There are a lot of libraries related to optimization, but I will use the most popular one, YUI Compressor.

When should I create or optimize the files? At runtime or at build time?

To provide static files from CDN (Content Delivery Network) or a separate server, I recommend creating or minifying the file at build time. I have implemented a file creation and minification function as a Maven plug-in for that specific purpose. The following code is an example of the pom.xml file:

<build>
  <plugins>
    <plugin>
      <groupId>com.naver.shopping.maven</groupId>
      <artifactId>maven-compress-plugin</artifactId>
      <version>1.0.12</version>
      <configuration>
        <webDirectory>web</webDirectory>
        <fileEncoding>UTF-8</fileEncoding>
        <configLocation>web/WEB-INF/compression.xml</configLocation>
        <repository>http://xxx.shopping.naver.net/static/</repository>
      </configuration>
    </plugin>
  </plugins>
</build>

Distribution with BDS

NHN distributes Web applications using its in-house developed Build Distribution System (BDS). The following figure illustrates the existing deployment process in a simple manner.

deployment-with-bds.png 

Figure 3: Deployment with BDS.

The following figure shows the process of CSS and JavaScript file compression and the existing deployment process.

deployment-minifying-css-javaScript-file.png 

Figure 4: Deployment for Minifying CSS and JavaScript File.

  1. Compressed file creation: When you build a Web application using Maven, create a compressed file by reading the compression.xml file.
  2. Version file creation: By using the information described in the compression.xml file, create a version file and save it in the version file repository (the repository described in the pom.xml file).
  3. CSS and JavaScript file copy: By using the PostScript function of BDS, copy the CSS file and the JavaScript file to the CDN or the cookie free domain server.

How to Configure and Use

  1. Create a compression.xml file and save it in the web/WEB-INF directory.
  2. For Web projects that use Lucy, set the Lucy plug-in as shown below:
  3. <lucy:plug-in name="compress" class="com.naver.shopping.common.compress.web.CompressPlugIn">
            <lucy:param name="configLocation" value="/WEB-INF/compression.xml"/>
            <lucy:param name="devOnly" value="false"/>
            <lucy:param name="baseUrl" value="http://static.shopping.naver.net/f"/>
    </lucy:plug-in>

    For Spring MVC-based Web projects, register Bean as shown below:

    <bean class="com.naver.shopping.common.compress.web.CompressBean">
             <property name="configLocation" value="/WEB-INF/compression.xml"/>
             <property name="devOnly" value="false"/>
             <property name="baseUrl" value="http://static.shopping.naver.net/f"/>
    </bean>

  4. Add the Maven plug-in settings to the pom.xml file.
  5. Describe the following in the jsp file:
  6. <link:js prefix="/js/lib.js"/>
    <link:js prefix="/js/ac.js" needCookie="true"/>
    <link:js prefix="/js/notexists.js"/>

What I have described in the above steps will result in the following. Among the static files, the JavaScript file that requires cookies is served from the Web server itself, and the CSS file and the JavaScript file that does not require cookies are served from the cookie free domain or the CDN.

<script type="text/javascript" src="http://static.shopping.naver.net/f/js/ lib_v1.js"></script>
<script type="text/javascript" src="/js/ac_v1.js"></script>
<script type="text/javascript" src="http://static.shopping.naver.net/f/js/ notexists_v2.js?t=2011040512"></script>

The /js/notexists.js file is not defined in the compression.xml file. In this case, the file is returned with the file name and timestamp of the highest version that starts with notexists under the /js directory.

Conclusion

In this article, I have briefly explained how we distribute and deploy static files at Knowledge Shopping service department here at NHN. The method I have described is not new. But I am making, what used to be an insider-only tip, public so that more people are aware of a better way to distribute static files and share their thoughts.

I would be really glad if you shared in the comments below how you distribute static files. Perhaps there are better ways to do this.

By Young-eun Oh, Shopping Service Development Team, NHN Corporation.



comments powered by Disqus