Varnish + HTTP Cache: An intro guide for web developers – Part 1

Concept:
Varnish accelerates webapps by sitting in between the application server and your visitors. It intelligently keeps responses from your application server in memory to give blazing-fast responses to the clients that request the same information; giving the application server less workload.

Varnish as a developer:
I believe that all professional web developers should have an understanding of HTTP caching mechanisms that aid in creating high performance webapps, and have the skillset needed to work closely with sysops.

This blogpost will initiate a series of posts that will help you as a developer to get up to speed with relevant aspects of HTTP and server-side accelerators like Varnish.

Assumptions:
You are familiar with the common *nix shells, server side web development and have a basic understanding of the HTTP protocol concepts.

Goal of part #1:

• Initial Varnish configuration and starting it

• Start a PHP 5.4+ development webserver and serve it’s content through Varnish. This is just a simple alternative, Apache is preferred if you are comfortable with setting that up.

Installation of Varnish, to get started:
We are assuming a Linux based environment for these posts.
A lot of things will be distribution-agnostic, but for the sake of convenience I will cover the Debian based distros like Ubuntu and Arch Linux for the things that are not. To get started learning I recommend simply installing Varnish on your workstation (or virtual machine image) and play around with it as you follow these posts. When installing Varnish on a dedicated server the process will be similar, but usually automated and managed by your ops team using something like Chef.

Most modern distributions have Varnish in their official package repositories. These packages will include Varnish itself (3.x) as well as the dependencies needed for it to run. That makes installation super-easy.

Here we see som information on the varnish package from Arch Linux package manager:

As you can see Varnish has a very small number of dependencies.

Go ahead and install it (Arch Linux):

Ubuntu:

Varnish is now installed. If we try to start it now it will fail as it is currently unconfigured (Arch Linux example output):

Configuration:
We start with the basics, we simply just want to get Varnish up and running with a default configuration that is easy to modify onwards as we learn.

1. Open /etc/varnish/default.vcl in your favourite editor. This is the default configuration loaded by Varnish and is currently incomplete.
2. Uncomment (remove # in front of the respective lines) so that the default configuration template is exposed. (vim: ctrl+v, s-G, d)
3. We now have a valid Varnish Configuration (VCL) that contains some informative boilerplate! Take a few minutes to read through the syntax to get a feel for how this works.

On Ubuntu you will have to specify the startup parameters for Varnish in /etc/default/varnish, for example:

Starting Varnish (Arch Linux):

Starting Varnish (Ubuntu):

Now we are up and running! Varnish is now listening to port 80.
To confirm you can visit “localhost” in your browser to see the content, or simply perform a HEAD request using curl to see the response headers:

The 503 is expected and correct at this time. This is simply Varnish telling us that it cannot find our backend. The important thing here is that we verify that we got an actual HTTP response from Varnish.
Note: Varnish does not care about what you use to serve your content because it operates on the HTTP level. Apache, NodeJS, Tomcat are all fine examples of servers!

Caching your web application:
We have Varnish running now, and the default VCL has a “backend” defined at 127.0.0.1:8080.
Feel free to revisit /etc/varnish/default.vcl to confirm.

Let’s start a webserver serving a super-simple webapp on 127.0.0.1:8080. The rest of this article assumes that PHP 5.4+ is installed, and we will use PHP without any framework and the built-in webserver onwards.

Install php (if not already installed) (Arch Linux):

Ubuntu:

Create a directory for our little project:

Create a file in your favourite editor, index.php:

Start the PHP development webserver on the backend that Varnish is currently connected to. Remember to change the port if you did something different in varnish.vcl:

NOTE: If you use Apache instead, just ignore the information regarding php -S

Check that the webserver serves content:

Awesome. Now let’s see if Varnish manages to serve us some content via that backend (Varnish runs on port 80):

All good. Varnish is now serving the content from our backend (PHP 5.4 dev http server). As you can see Varnish adds some header information. This is also configurable to developers and sysops through the Varnish configuration language that you have already seen briefly in /etc/varnish/default.vcl.

Let’s request the same content again:

Notice the “Age” header. Varnish now served the response to us straight from memory without bothering php! You may also notice that the accesslog for PHP webserver running in terminal or Apache accesslog did not generate any output for this request. Varnish handled it all without asking PHP for anything.

Terminology you should be comfortable with now since used in the later parts:

• Update Varnish configuration: Refers to changing the VCL configuration file and reloading it by restarting varnish
• Change webapplication: Refers to updating the PHP code in our example project

Summary part #1:
We now have a basic environment set up for testing and learning Varnish and HTTP cache concepts.

In part #2 we will get started with HTTP headers and how to use these to get Varnish to behave exactly how we want.