Installing Varnish 4.x on Apache Debian 8 server (Jessie) with Drupal 8

Varnish is a very fast caching mechanism for web applications but it is not always easy to install.  In the latest version there are a number of changes pertinent to Debian servers.

The following assumes you are working as root.  If not, then you need to add sudo to the front of all commands.

Installing Varnish on the server

At the command line:

apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.1"\
    >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

In Ansible:

  - name: Get Varnish key
    apt_key: url=https://repo.varnish-cache.org/ubuntu/GPG-key.txt state=present   - name: Apt Update
    apt: update_cache=yes   - name: Install Varnish
    apt: name=varnish state=present
 

The first step is to make Varnish the first call for all http requests.  Http normally uses port 80 and so this is the port we will use.  Before Debian 8 and Varnish 4.x custom configuration could be made in  the file /etc/default/varnish by changing the value for -a :6081 to -a :80

DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"
 

In Debian 8, however, configuration is done by systemd.  To enable this create a new file in the directory at /etc/systemd/system/ called varnish.service, whose contents can vary depending on your setup, but the following will work on most installs:

[Unit]
Description=Varnish HTTP accelerator [Service]
Type=forking
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/default.vcl
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,512m
ExecReload=/usr/share/varnish/reload-vcl [Install]
WantedBy=multi-user.target

In ansible we can use:

  - name: Override Systemd service
    template: src=varnish.service dest=/etc/systemd/system/varnish.service
 

The value -a :80 determines the port at which Varnish listens.  This is the key item for this configuration.

The value -T localhost:6082 determines the port for varnish administration.  We will revisit this when we configure Varnish for Drupal.

Varnish requires one other configuration file.  This too has changed a lot since Varnish 3.x.  There are a number of examples available on the net written expressly for Drupal.  I got mine from https://github.com/NITEMAN/varnish-bites.  Download a suitable Drupal default.vcl file to use on your server.  There is one setting that needs to be changed in the Backend Definitions.

backend default {
  .host = "127.0.0.1";
  .port = "8000";
  .max_connections = 100;
  .connect_timeout = 60s;
  .first_byte_timeout = 60s;
  .between_bytes_timeout = 60s;
  .probe = basic;
}

For this setup we should change the port to 8000.  This means that Varnish will listen on port 80 and pass requests onto port 8000 which is the port we are going to choose from which to load the web pages.  

The default.vcl file belongs in /etc/varnish.  In Ansible we can use:

  - name: Upload default.vcf
    template: src=default.conf dest=/etc/varnish/default.vcl

Until you know a bit more about the configuration settings in default.vcl this should do the trick.

Now that we have Varnish ready to go we should restart it.  At the command line:

systemctl daemon-reload

systemctl restart varnish.service

Or with Ansible:

  - name: reload systemd
    command: systemctl daemon-reload   - name: restart varnish
    command: systemctl restart varnish.service
 

On the command line you can check that Varnish is running correctly with:

systemctl status varnish.service -l

or

netstat -lp | grep varnish

You will get an error such as:

varnish.service - Varnish HTTP accelerator
   Loaded: loaded (/etc/systemd/system/varnish.service; enabled)
   Active: failed (Result: exit-code) since Sun 2016-07-03 17:33:29 NZST; 9min ago
 Main PID: 10121 (code=exited, status=0/SUCCESS)
Jul 03 17:33:28 server2016 varnishd[13512]: .backend_error_func = VGC_function_vcl_backend_error,
Jul 03 17:33:28 server2016 varnishd[13512]: .init_func = VGC_function_vcl_init,
Jul 03 17:33:28 server2016 varnishd[13512]: .fini_func = VGC_function_vcl_fini,
Jul 03 17:33:28 server2016 varnishd[13512]: };
Jul 03 17:33:29 server2016 varnishd[13522]: bind(): Address already in use
Jul 03 17:33:29 server2016 varnishd[13522]: bind(): Address already in use
Jul 03 17:33:29 server2016 varnishd[13522]: Error: Failed to open (any) accept sockets.
Jul 03 17:33:29 server2016 systemd[1]: varnish.service: control process exited, code=exited status=2
Jul 03 17:33:29 server2016 systemd[1]: Failed to start Varnish HTTP accelerator.
Jul 03 17:33:29 server2016 systemd[1]: Unit varnish.service entered failed state.

This is because Apache is still bound to port 80.  Now to change that.

Configuring Apache

When Apache is bound to port 80 it takes precedence over Varnish.  In order to allow Varnish to serve incoming http requests, we will assign Apache to port 8000. in all apache.conf files:

• Change Listen 80 to 8000.  This might be in your apache2.conf or in ports.conf or elsewhere.
• Note that in Debian 8 NameVirtualHost is deprecated so you can ignore it.
• Go through all vhost configs and change *:80 to *:8000 in each of them.  If you are applying Varnish to a server then all vhosts need to lie behind it.  You cannot leave any of them assigned to port 80 or Varnish won't work and will give the error above.

You should now have:

Listen 8000

And directives such as 

<VirtualHost *:8000>
  
  ServerName  example.com
  ServerAlias www.example.com   # Index file and Document Root (where the public files are located)
  DirectoryIndex index.php
  DocumentRoot /var/www/html

</VirtualHost>

Upload those changes to the server and restart Apache.  Now run the check:

netstat -lp | grep varnish

And you should get something like:

tcp        0      0 *:80                       *:*                     LISTEN      17239/varnishd  

tcp        0      0 server.vps.s:6082 *:*                     LISTEN      17238/varnishd  

tcp6       0      0 [::]:80                    [::]:*                  LISTEN      17239/varnishd  

This tells us that Varnish is listening on port 80 and using port 6082 as desired.  If Varnish does not work for you please check the Troubleshooting list at the end of this page.

Now that we have Varnish running on the server we should install the Drupal Varnish module.  Go to the admin section at /admin/config/development/varnish.  The Varnish Control Terminal should be 127.0.0.1:6082 for most servers.  Since we have installed Varnish with a secret key, then you'll need to obtain this from /etc/vanish/secret.  Make sure that Varnish 4.x is selected and when you submit the form it should show that Varnish is running.

Now check that Varnish is working on your website by using this handy widget.

 

Troubleshooting

If you get the error 

varnish.service has more than one ExecStart setting, which is only allowed for Type=oneshot services. Refusing.

I found it difficult to get Varnish to read my varnish.service file and this problem was a time hog.

•  It is likely that your script editor / IDE is adding extra characters or newlines to you custom varnish.service file and you need to use a simple editor like notepad with no extra characters.  Try copying from the source file at /lib/systemd/system/varnish.service and making the customizations you require in TextWrangler (Mac) or Notepad (Windows).
• As shown in the Varnish documentation, it is sometimes wise to put an empty ExecStart declaration in place especially when you are repeatedly restarting Varnish.  This has the effect of clearing the variable out before we add a new value.

 

Useful commands

Purge a domain

curl -X PURGE http://domain.org

 

Useful resources

A Concise summary

Varnish Docs

Varnish Bites