posted last year in Dev Platform category by Young Eun Oh
The distribution of a Web application is as important as its development. Distribution methods can enhance the user experience or cause a failure. In this article, I will explain how we distribute static files at Knowledge Shopping Service department here at NHN.
Difficulty of Distributing Static Files
Figure 1: Server Before and After Distribution.
For example, if a user accesses a Web page and obtains an HTML file from Server A (post-distribution) but CSS files used in this HTML files are served from Server B (pre-distribution), the user will end up using the new HTML file and the old CSS file, which may result in broken layout of the page or site malfunction.
- To prevent this kind of problem, it is necessary to change the name of a newly distributed file making sure that the old HTML file refers to the old CSS file, and the new HTML file refers to the new CSS file.
By using the method above, most problems that can occur while distributing static files can be prevented.
However, if the number of static files referred by a Web page is large, the performance may suffer.
- For instance, in case a file has not been cached on the client side, it must be downloaded whenever it is requested.
- On the other hand, if a file has been cached, a "304 Not Modified" will be received whenever a user accesses the server, incurring a DNS resolve cost and a socket connection cost.
- Even when you merge the static files to reduce their number, unnecessary blanks and long variable names tend to increase the size of the transferable data.
- More than that, the size of the transferrable data will grow when static files are distributed to the Web server, as cookies included in an HTTP request message will be transferred as well.
Though it would seem tempting to have a single person manage all of the static files written during the development process, it often proves inefficient due to an increased management cost. Therefore, the distribution process must be automated to the greatest extent possible, and the following rules must be kept for the automatic and efficient distribution of static files.
- Add the timestamp value to the query string.
- Reduce the number of files.
- Minify file content.
Optimization by Using Apache Web Server Configuration
- Compress the static files using the deflate module of the Apache Web server and transfer the compressed file. Add the deflate module configuration to the Apache configuration file, as shown below:
- Then, add the expire module to allow the browser to cache the static file.
# Add the expires module LoadModule expires_module modules/mod_expires.so ExpiresActive On # Set the default expires to one year ExpiresDefault "access plus 1 years"
HOW TO manage static files
Figure 2: Directory Structure.
- ... scan the subdirectory files including /web/js;
- and create a list of files with the highest version;
- then mark the file path by using the custom tag as shown below:
The following code will be displayed in the HTML:
To add new content or modify the existing one that does not support backward compatibility, get a higher version (ex: main_v1.js main_v2.js). The script will load the latest version.
What are the potential issues which can arise when using this solution?
If static files have been distributed in advance, all Web servers will have both main_v1.js file and main_v2.js file. Therefore, no problem would occur during the distribution.
During the Web page development process, the development versions of static files are used (external URL). What should I do in this case?
Modify the custom tag to describe the development URL like:
<link:js prefix="/js/main.js" devOnlyUrl="http://svn.nhndesign.com/service/main_20110404.js" />
While developing the Web application, use devOnlyUrl. After completing the development, use the highest version of the main.js file. Of course, there should be a way to set the development status of the file.
<link:js prefix="/js/cookie.js" needCookie="true" />
HOW TO merge static files
So far, I have described how to manage static file versions and how to describe files being developed. Now, I will explain how to bind the main.js file and the detail.js file to one core.js file to reuse their number. Write the configuration file as shown below:
<?xml version="1.0" encoding="UTF-8"?> <compression> <file target="/js/core.js"> <include>/js/main</include> <include>/js/detail</include> </file> </compression>
How do I manage the versions of the created files?
- If you have bound main_v1.js and detail_v1.js files to create one core_v1.js file, and there is no change in those files, keep the file name as 'core_v1.js'.
- When the content or the version of the main_v1.js or the detail_v1.js file is changed, the core_v1.js file must be changed to the core_v2.js file. To do this, you must manage the name, size, and checksum of each file that is part of the file to be created.
version=1 encoding=UTF-8 file./js/main_v1.js=945,UTF-8,ac3fecacd0a6ecf74a0c711180582a8c file./js/detail_v1.js=22572,UTF-8,ee2565f0938bfd1baa21c73afa633c81
Now, you should minify the bound files. There are a lot of libraries related to optimization, but I will use the most popular one, YUI Compressor.
When should I create or optimize the files? At runtime or at build time?
To provide static files from CDN (Content Delivery Network) or a separate server, I recommend creating or minifying the file at build time. I have implemented a file creation and minification function as a Maven plug-in for that specific purpose. The following code is an example of the pom.xml file:
<build> <plugins> <plugin> <groupId>com.naver.shopping.maven</groupId> <artifactId>maven-compress-plugin</artifactId> <version>1.0.12</version> <configuration> <webDirectory>web</webDirectory> <fileEncoding>UTF-8</fileEncoding> <configLocation>web/WEB-INF/compression.xml</configLocation> <repository>http://xxx.shopping.naver.net/static/</repository> </configuration> </plugin> </plugins> </build>
Distribution with BDS
NHN distributes Web applications using its in-house developed Build Distribution System (BDS). The following figure illustrates the existing deployment process in a simple manner.
Figure 3: Deployment with BDS.
- Compressed file creation: When you build a Web application using Maven, create a compressed file by reading the compression.xml file.
- Version file creation: By using the information described in the compression.xml file, create a version file and save it in the version file repository (the repository described in the pom.xml file).
How to Configure and Use
- Create a compression.xml file and save it in the web/WEB-INF directory.
- For Web projects that use Lucy, set the Lucy plug-in as shown below:
- Add the Maven plug-in settings to the pom.xml file.
- Describe the following in the jsp file:
<lucy:plug-in name="compress" class="com.naver.shopping.common.compress.web.CompressPlugIn"> <lucy:param name="configLocation" value="/WEB-INF/compression.xml"/> <lucy:param name="devOnly" value="false"/> <lucy:param name="baseUrl" value="http://static.shopping.naver.net/f"/> </lucy:plug-in>
For Spring MVC-based Web projects, register Bean as shown below:
<bean class="com.naver.shopping.common.compress.web.CompressBean"> <property name="configLocation" value="/WEB-INF/compression.xml"/> <property name="devOnly" value="false"/> <property name="baseUrl" value="http://static.shopping.naver.net/f"/> </bean>
<link:js prefix="/js/lib.js"/> <link:js prefix="/js/ac.js" needCookie="true"/> <link:js prefix="/js/notexists.js"/>
The /js/notexists.js file is not defined in the compression.xml file. In this case, the file is returned with the file name and timestamp of the highest version that starts with notexists under the /js directory.
In this article, I have briefly explained how we distribute and deploy static files at Knowledge Shopping service department here at NHN. The method I have described is not new. But I am making, what used to be an insider-only tip, public so that more people are aware of a better way to distribute static files and share their thoughts.
I would be really glad if you shared in the comments below how you distribute static files. Perhaps there are better ways to do this.
By Young-eun Oh, Shopping Service Development Team, NHN Corporation.