Home
T0xicCode identity
Thoughts on software, computers and technology.

Get to know me.

Web server configuration with Nginx and Applications Servers

This is part 1 of the series

There are hundreds of ways to configure any web server, but it is my opinion that some are better than others.

I recommend, and implement, where I can, a separation in the roles of web server and application server. To me, the web server should not interpret or render any content. Its sole job should be to serve content from the disk (I include content, i.e. directives, specified in configuration files) and proxy requests to and from a dedicated application server. The application server, on the other hand, should not be wasting CPU cycles loading and parsing a static asset. Its sole job should be to run the code that makes the website dynamic. Depending on the language, it might even never access the disk once initialized.
Separating the application layer from the web layer also allows us to do nice things such as spin up more application servers to deal with more incoming traffic, move the applications to separate machines not publicly accessible, or even replace the web or applications servers without impacting the availability of the website.
Today, I see of no reason to keep your web server tightly coupled with your application server.

My web server of choice is nginx. It is a small, fast and efficient web server, with a simple configuration syntax and, very importantly, does not easily permit the embedding of application servers within its core. It forces you to separate the application servers, and, as a result, you wont need to warm up the application to serve a static image.

After installing nginx (my preferred method involves running sudo apt-get install nginx), the first thing I did was disable the default website: sudo rm /etc/nginx/sites-enabled/default. Next, I setup and install a "website" that loads the configuration files for all hosted websites. I saved the following content in the file webapps (use nano or vim to edit it):

include /var/webapps/*/conf/nginx.conf;

The next thing I did was enable that website and restart nginx:

sudo ln -s ../sites-available/webapps /etc/nginx/sites-enabled/webapps
sudo service nginx restart

Now that nginx was configured, it was time to create the first website. I created a convention for all my websites:

/var/webapps
└── unique_app_name
    ├── conf (configuration files)
       ├── nginx.conf (nginx configuration file)
       ├── php5.conf (php5-fpm configuration file, if needed)
       └── uwsgi.ini (uwsgi configuration file, if needed)
    ├── logs (access, debug, application, error, etc.)
    ├── run (pid files and unix sockets)
    └── www (document root)

This convention allows me to create and publish a website in record time, as the servers know where to find their configuration files, and I don't have to look around to find where some setting is hidden.

Here, I'll create an initial website to server only static content. In the following snippets, comments will start with the pound character (#). I make several assumptions here: your web server runs as www-data:www-data, you are in the group www-data.

# Create the webapps folder
sudo mkdir /var/webapps
sudo chown www-data:www-data /var/webapps
sudo chmod g+s /var/webapps

# Next, we'll create the default website
cd /var/webapps
mkdir -p default/{conf,logs,www}
touch default/conf/nginx.conf

Next, we'll edit the nginx configuration file (use your favourite editor for that) to something similar to this:

server {
    listen 80;

    access_log /var/webapps/default/logs/access.log;
    error_log /var/webapps/default/logs/error.log;

    root /var/webapps/default/www;

    index index.html index.htm;

    server_name _;

    # hide non-existent favicon
    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }

    # hide dotfiles
    location ~ /\. {
        deny all;
    }

    location / {
        try_files $uri $uri/ =404;
    }
}

Once we are done, we can create an index.html file in www, a simple page looks like so:

<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>

The last step is to reload nginx, and navigate to the website:

sudo service nginx reload

In the next part of the series, I'll be adding php to our website. Stay tuned!

Finishing the Website

Quick update! Over this past weekend, I finally got around to completing the templates and the website.

When I finished the previous blog post, I had made limited modifications to the theme's template, the bare minimum really. This time around, I gave it my all (or close to it). First off, I re-organized the raw template files. I introduced prefixes and arranged the files in a modular system. Next up, was the template code. This required a bit more work, as I had to make many changes (hurray for development servers with auto-reload), tweaking the layout and the style. Finally, I went over the stylesheet, removing as much cruft as I could.

With this complete, the only thing left to finish my changes to the system are the changes I wanted to introduce in the development script and the Makefile. I don't anticipate them to be too demanding, but I'll have to find some free time first.

A New Look

I recently decided to try out a nice static blog generator, and given that I had experience with python (and love the language), I decided to give pelican a try. It's a static blog generator inspired by the very successful jekyll static blog generator, but implemented in python rather than ruby.

There is definitely something good to be said for a functioning website that is entirely managed by committing regular text files to a repository, and quite frankly, I like it!

The install it self was as quick and painless as could be, only having to issue two pip commands (after having setup the virtual environment) to have the packages installed (I had to install markdown separately). However, after taking a look at the facilities provided by the package, I found them severely lacking. Important paths, such as the input and output paths, where both hardcoded and defined in multiple files. There were a slew of similar issues throughout the 5 files that managed and facilitated the development.

Fixing it

The first thing I tackled was removing the files that were unnecessary for my workflow, namely the fabfile and half of the make file. Once that was done, I started moving the configuration files (default + production) from the root of the webapp (my folder structure merits its own post, for the future) to the configuration folder. That went well as well, but I had to edit the rest of the files to look for configuration in the right place.

Next, I edited the configuration files to have a sane folder structure, i.e. the raw content in src and the output in www. That required a few edits, but pelican itself accepted the change with no problem.

Following that, I modified the makefile to take into account my workflow, and build the website directly on the server. This is because I revision the source files in the git repository, but not the generated files since production and development generate different files, and it's good practice to not revision your generated output with the source (although I recommend on revisioning your generated output in a separate repository or storing it with unique names in a backed up folder).

Finally, I installed a good looking minimalistic theme (SoMA) and made a few modifications to it. I still have to go through it and remove all the superfluous code, but it works right now.

With these changes, the system matched my workflow so I called it a day and wrote this post.

What's next?

There is a develop_server.sh script that helps with development by watching your input directory and automatically regenerating the html on each change and provides a basic (very basic) http server. I'd like to either heavily refactor the shell script to a dump dispatcher or, if possible, move all of the script's function to the makefile.

Hopefully I'll have time to finish that by the time I post my next article.