1 of 125

DIY Drupal 7 Performance

Mike Carper @mcarper (mikeytown2)

​

Pacific Northwest Drupal Summit�October 2015

https://goo.gl/L4P3MA

2 of 125

Follow along

https://goo.gl/L4P3MA

https://goo.gl/L4P3MA

3 of 125

https://goo.gl/L4P3MA

4 of 125

About Mike

  • November 2008 - Started using Drupal

​

  • January 2009 - Discovered Boost module

​

  • September 2009 - Boost 6.x-1.0 released

​

  • November 2009 - Started @ Datasphere

​

https://goo.gl/L4P3MA

5 of 125

About Mike (@ Datasphere)

  • Our team scaled a single Drupal 6 domain access multisite up to 1,300 different domains�
  • Our Drupal 7 site is 99% authenticated with a nothing shared between users, making caching very hard to do�
  • Created a lot of performance related modules that helped us out a lot

​

https://goo.gl/L4P3MA

6 of 125

https://goo.gl/L4P3MA

7 of 125

https://goo.gl/L4P3MA

8 of 125

What can we do to make sure users enjoy using your site?

https://goo.gl/L4P3MA

9 of 125

Make sure it's FAST

For logged in & logged out users

https://goo.gl/L4P3MA

10 of 125

How?

https://goo.gl/L4P3MA

11 of 125

Throw Money At

https://goo.gl/L4P3MA

12 of 125

Or you can DIY!

https://goo.gl/L4P3MA

13 of 125

Some rules to follow

https://goo.gl/L4P3MA

14 of 125

https://goo.gl/L4P3MA

15 of 125

General Guidelines

Anonymous visitor caching is easy.

​

Big performance gains come asynchronous/parallel and deferred code execution. Why do it now when you can do it later.

​

Less is usually better.

https://goo.gl/L4P3MA

16 of 125

General Guidelines (cont'd)

Big improvements can be found on the frontend.�

If you have to pick scalability over performance 9 times out of 10 pick scalability; you’ll thank yourself later on when you have more than 20 users logged in at once.

​

Measure changes - Was that good or bad?�

https://goo.gl/L4P3MA

17 of 125

Page Caching

mainly useful for anonymous users

https://goo.gl/L4P3MA

18 of 125

Caching to the rescue!

https://goo.gl/L4P3MA

19 of 125

Popular Page Cache Options

  • Drupal Core

​

  • Boost�
  • Memcache/Redis & Varnish

​

https://goo.gl/L4P3MA

20 of 125

Drupal Core

Advantages:

​

  • Comes with core

​

  • Better than nothing

Disadvantages:

​

  • Relies on the database

​

  • Slow as PHP is needed

https://goo.gl/L4P3MA

21 of 125

https://goo.gl/L4P3MA

22 of 125

https://goo.gl/L4P3MA

23 of 125

Boost

Advantages:

​

  • Works on almost all shared hosting

​

  • Fast as no PHP is executed

Disadvantages:

​

  • Need to copy in htaccess rules

​

  • Cache is not easily shared with multiple servers

​

https://goo.gl/L4P3MA

24 of 125

https://goo.gl/L4P3MA

25 of 125

Memcache/Redis & Varnish

Advantages:

​

  • Fast

​

  • Can help logged in users as well

​

Disadvantages:

​

  • Need shell access

​

  • Not simple to setup�
  • High availability is hard

https://goo.gl/L4P3MA

26 of 125

https://goo.gl/L4P3MA

27 of 125

Optimization and Tuning of Drupal 7

https://goo.gl/L4P3MA

28 of 125

Optimization Overview

Core page and block caching

Core CSS & JS aggregation

Panels & views caching

Disable unnecessary modules

Modules that usually help

Poormanscron in core

Enable fast 404

Core patches

​

https://goo.gl/L4P3MA

29 of 125

Optimization Overview (cont'd)

Database Connection

Op Code

php.ini

PHP Notices/Warnings

Basic database tuning

Deadlock Detection

Diagnosing Stuck Queries

Slow Query Analysis

Cachegrind

​

​

​

​

​

​

​

https://goo.gl/L4P3MA

30 of 125

TLDR

https://goo.gl/L4P3MA

31 of 125

Core and Block Caching

admin/config/development/performance

https://goo.gl/L4P3MA

32 of 125

admin/config/development/performance

https://goo.gl/L4P3MA

33 of 125

admin/config/development/performance

https://goo.gl/L4P3MA

34 of 125

Enable CSS & JS Aggregation

admin/config/development/performance

https://goo.gl/L4P3MA

35 of 125

admin/config/development/performance

https://goo.gl/L4P3MA

36 of 125

Views Caching

https://goo.gl/L4P3MA

37 of 125

https://goo.gl/L4P3MA

38 of 125

https://goo.gl/L4P3MA

39 of 125

https://goo.gl/L4P3MA

40 of 125

Panels Caching

https://goo.gl/L4P3MA

41 of 125

https://goo.gl/L4P3MA

42 of 125

https://goo.gl/L4P3MA

43 of 125

https://goo.gl/L4P3MA

44 of 125

https://goo.gl/L4P3MA

45 of 125

Disable unnecessary modules

https://goo.gl/L4P3MA

46 of 125

https://goo.gl/L4P3MA

47 of 125

https://goo.gl/L4P3MA

48 of 125

https://goo.gl/L4P3MA

49 of 125

https://goo.gl/L4P3MA

50 of 125

Most modules behave well; some do not. The poor performing modules are usually the ones without a lot of users.

​

The next release of D7 (7.40) contains a big fix that should speed up most drupal installs by 5-10%; the module_implements() function has been improved in core.

​

https://goo.gl/L4P3MA

51 of 125

Modules that help with optimizing Drupal

https://goo.gl/L4P3MA

52 of 125

Backend

https://goo.gl/L4P3MA

53 of 125

Entity Cache Module

A lot more useful if using memcache

https://drupal.org/project/entitycache

https://goo.gl/L4P3MA

54 of 125

Render Cache

Caches rendered entities currently�https://www.drupal.org/project/render_cache

https://goo.gl/L4P3MA

55 of 125

Cache Expiration

https://goo.gl/L4P3MA

56 of 125

Imageinfo Cache

Generate image styles on file upload

https://www.drupal.org/project/imageinfo_cache

https://goo.gl/L4P3MA

57 of 125

Asynchronous Prefetch Database Query Cache

Fixes everything that is wrong with the core database cache; MySQL only

https://www.drupal.org/project/apdqc

​

https://goo.gl/L4P3MA

58 of 125

HTTP Parallel Request & Threading Library

Doesn’t do much out of the box but has some awesome tools when creating custom code. Yes this works on shared hosting and on windows.

https://www.drupal.org/project/httprl

https://goo.gl/L4P3MA

59 of 125

Frontend

https://goo.gl/L4P3MA

60 of 125

Frontend

I always test changes made with Google’s PageSpeed Insights and webpagetest.org

​

I set the connection speed to 3G or slower when testing on webpagetest.org

​

https://goo.gl/L4P3MA

61 of 125

ImageAPI Optimize

Optimize the image when it is saved

https://www.drupal.org/project/imageapi_optimize

https://goo.gl/L4P3MA

62 of 125

Defer Image

Prevent images from slowing down the initial load and render of a page

https://www.drupal.org/project/defer_image

https://goo.gl/L4P3MA

63 of 125

Font Awesome SVG

Use font awesome? Use this to reduce the bytes downloaded.

https://www.drupal.org/project/fontawesome_svg

https://goo.gl/L4P3MA

64 of 125

AdvAgg

Faster CSS/JS aggregation

https://www.drupal.org/project/advagg/

https://goo.gl/L4P3MA

65 of 125

More info on AdvAgg

Drupal.org started using it this year. According to Google Analytics the average UC Browser page load time went from 13.3 seconds down to 5.4 seconds.

​

UC Browser is a mobile browser with a huge user base in China, strong adoption in India and continued growth in emerging regional markets.

https://goo.gl/L4P3MA

66 of 125

Stats from https://www.drupal.org/project/drupal�The slower the connection the bigger an effect AdvAgg has on improving the front end responsiveness.

https://goo.gl/L4P3MA

67 of 125

AdvAgg D.O Settings.

  • Use Aggressive Render Cache
  • Move all inline scripts to the bottom of the execution order & Move all external scripts to the top of the execution order
  • Use DNS Prefetch for external CSS/JS
  • Combine CSS files by using media queries
  • Move JS to the footer - All but what is in the $all_in_footer_list
  • Deferred JavaScript Execution: Add The defer Tag To All Script Tags - All but external scripts

https://goo.gl/L4P3MA

68 of 125

AdvAgg D.O Settings. (cont'd)

  • Deferred inline JavaScript Execution: Put a wrapper around inline JS so it runs from a setTimeout call
  • Remove ajaxPageState CSS and JS data if ajax.js is not used on this page
  • Change CSS bundles to 2
  • Change JS bundles to 5
  • Move Google Analytics analytics.js code from inline to be a file
  • Prefetch stats.g.doubleclick.net/robots.txt

https://goo.gl/L4P3MA

69 of 125

Other AdvAgg settings

Future AdvAgg settings

  • Use JSMin
  • Use Async Font loading

​

​

  • Inline critical css�

� https://www.drupal.org/node/2493801

​

​

https://goo.gl/L4P3MA

70 of 125

Poormans cron in core

https://goo.gl/L4P3MA

71 of 125

admin/config/system/cron

https://goo.gl/L4P3MA

72 of 125

Follow a guide on how to setup cron on your server. Almost all shared hosting allows for cron jobs to be scheduled.

https://goo.gl/L4P3MA

73 of 125

Even bad hosts have instructions

https://goo.gl/L4P3MA

74 of 125

https://goo.gl/L4P3MA

75 of 125

admin/config/system/cron

https://goo.gl/L4P3MA

76 of 125

Once you have your host running the drupal cron job you can disable the one that comes with core by default

https://goo.gl/L4P3MA

77 of 125

admin/config/system/cron

https://goo.gl/L4P3MA

78 of 125

Enable fast 404

Unless you’re using the private file system

https://goo.gl/L4P3MA

79 of 125

Go to the sites/default directory

Open up to the setting.php file

Around line 500 uncomment this

drupal_fast_404();�by removing #

​

https://goo.gl/L4P3MA

80 of 125

Also use fast 404 to block urls commonly used by bots

Requests like /postnuke/article.php should be 404-ed ASAP. Do not waste server resources on serving a nice 404 to bots.��

​

https://goo.gl/L4P3MA

81 of 125

While inside the settings.php file above where we enabled fast 404 are the settings that control it. add

^(postnuke|wp-content|mailman|phpBB)|

to the start of the $conf['404_fast_paths']

variable, right after the first /

�This means that if any path starts with any of these strings we'll skip booting drupal and 404 right here taking very little server resources.

https://goo.gl/L4P3MA

82 of 125

In your settings.php Replace this:

�$conf['404_fast_paths'] = '/\.(?:txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

�With this�

$conf['404_fast_paths'] = '/^(postnuke|wp-content|mailman|phpBB)|\.(?:txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

​

https://goo.gl/L4P3MA

83 of 125

You can of course add more paths to this as you discover the other paths used by bots. ��Lets say you also want to fast 404 any path that starts with browserconfig.xml. This �(postnuke|wp-content|mailman|phpBB)�Now should be this�(postnuke|wp-content|mailman|phpBB|browserconfig\.xml)��Be careful when doing this; you can 404 a lot of paths.

https://goo.gl/L4P3MA

84 of 125

Core Patches

Hacking core for fun and profit

https://groups.drupal.org/node/210683

List of patches assumes Drupal 7.40 or higher

https://goo.gl/L4P3MA

85 of 125

Add static cache to module_load_include()

​

https://www.drupal.org/node/1443308#comment-10329229

​

Usually shaves 90ms off of a big site

https://goo.gl/L4P3MA

86 of 125

If item is hidden in _menu_tree_check_access() skip it right away

​

https://www.drupal.org/node/1710656#comment-6304412

​

Usually shaves 50ms off of a big site

​

https://goo.gl/L4P3MA

87 of 125

Avoid re-scanning module directory when a filename or a module is missing

​

https://www.drupal.org/node/1081266?page=1#comment-10434917

​

If you have one missing module then this can shave up to a second off of every page load

https://goo.gl/L4P3MA

88 of 125

Improve theme registry build performance

​

https://www.drupal.org/node/2339447#comment-10354975

​

Usually shaves 5-10 seconds off of a cache clear

https://goo.gl/L4P3MA

89 of 125

inline file_uri_scheme() in file_stream_wrapper_uri_normalize()

​

https://www.drupal.org/node/1443342#comment-5613306

​

Shaves 150ms on a cache clear

https://goo.gl/L4P3MA

90 of 125

Fix Notice: Trying to get property of non-object in image_style_deliver()

​

https://www.drupal.org/node/1762772#comment-7387274

​

Needed if the files directory is mounted on a NFS drive.

https://goo.gl/L4P3MA

91 of 125

Database Connection

Use unix sockets if possible

https://goo.gl/L4P3MA

92 of 125

Use the APDQC Module

The status report has a section on setting up unix sockets or an IP instead of hostname.��

https://goo.gl/L4P3MA

93 of 125

Op Code Cache

Use the built in OPcache in php 5.5+

https://goo.gl/L4P3MA

94 of 125

OPcache settings

zend_extension=opcache.so

opcache.enable=1

opcache.memory_consumption=256

opcache.validate_timestamps=0

opcache.max_accelerated_files=100000

opcache.fast_shutdown=1

opcache.interned_strings_buffer=16

​

https://goo.gl/L4P3MA

95 of 125

php.ini

https://goo.gl/L4P3MA

96 of 125

For all

realpath_cache_size = 1M

realpath_cache_ttl = 3600

​

Only if memcache is being used

memcache.hash_strategy="consistent"

​

​

https://goo.gl/L4P3MA

97 of 125

PHP Notices/Warnings

Fix them as these slow down your site

https://goo.gl/L4P3MA

98 of 125

Put this in your settings.php file

<?php

if (!empty($_GET[‘errors’])) {� // Show Errors in output� ini_set('display_errors', '1');� // Report all php errors.� error_reporting(-1);� // Display errors using dsm().� $conf['error_level'] = 2;

}�?>

​

https://goo.gl/L4P3MA

99 of 125

Basic database tuning

https://goo.gl/L4P3MA

100 of 125

Use the APDQC module

Status report is your friend

https://goo.gl/L4P3MA

101 of 125

What it will check for

  • That you’re using the mysqlnd db driver
  • using a unix socket if at all possible
  • Verify tx_isolation is READ-COMMITTED
  • max_allowed_packet is at least 32MB
  • innodb_buffer_pool_size is big enough for your database to fit into memory
  • Verify innodb_flush_log_at_trx_commit is 2
  • Query cache is disabled
  • innodb_lock_wait_timeout and wait_timeout are both sane values

​

https://goo.gl/L4P3MA

102 of 125

Note about the MySQL query cache

This should be disabled.

​

The built in MySQL query cache gets invalidated at the table level, any writes to that table will wipe out the query cache; it also has some nasty global locks on reads and writes.

https://goo.gl/L4P3MA

103 of 125

More info about MySQL & InnoDB

http://www.mysqlperformanceblog.com/2013/09/20/innodb-performance-optimization-basics-updated/

https://goo.gl/L4P3MA

104 of 125

Deadlock Detection

“deadlock found when trying to get lock try restarting transaction”

https://goo.gl/L4P3MA

105 of 125

SHOW ENGINE innodb STATUS

This command will output a bunch of information; the more interesting bit is under the “LATEST DETECTED DEADLOCK” section.

https://goo.gl/L4P3MA

106 of 125

Diagnosing Stuck Queries

What if SHOW PROCESSLIST shows a lot of simple queries that are taking too long

https://goo.gl/L4P3MA

107 of 125

https://goo.gl/L4P3MA

108 of 125

95% of the time database transactions are to blame

https://goo.gl/L4P3MA

109 of 125

What to do?

https://www.drupal.org/project/apdqc Is the answer. Module if used correctly will fix the deadlock and metadata locking issues with core’s db cache.

​

​

​

​

https://goo.gl/L4P3MA

110 of 125

I want to monitor this

You can run this next query and you can then kill that mysql process id if you desire to. This only works in real time; every time you get a locked database, run the query and manually kill that process id.

​

​

https://goo.gl/L4P3MA

111 of 125

ID stuck thread that holds the lock

SELECT trx.trx_id AS trx_id, trx.trx_mysql_thread_id AS thread_id, trx.trx_query AS query, trx.trx_tables_locked AS tables_locked, trx.trx_rows_locked AS rows_locked, trx.trx_state AS trx_state, p.DB AS db, p.STATE AS process_state, trx.trx_operation_state AS trx_op_state, p.COMMAND AS command, p.TIME AS query_age, DATEDIFF( trx.trx_started, NOW( ) ) AS trx_age, DATEDIFF( trx.trx_wait_started, NOW( ) ) AS wait_age, wr.requested_lock_id AS requested_lock_id, wb.requested_lock_id AS blocking_lock_id, trx.trx_isolation_level AS iso_level

FROM information_schema.innodb_trx AS trx

LEFT JOIN information_schema.processlist p ON trx.trx_mysql_thread_id = p.ID

LEFT JOIN information_schema.innodb_lock_waits w ON trx.trx_mysql_thread_id = p.ID

LEFT JOIN information_schema.innodb_lock_waits wb ON trx.trx_id = wb.blocking_trx_id

LEFT JOIN information_schema.innodb_lock_waits wr ON trx.trx_id = wr.requesting_trx_id

WHERE trx_tables_locked >0 OR trx_rows_locked >0

GROUP BY trx.trx_mysql_thread_id

https://goo.gl/L4P3MA

112 of 125

https://goo.gl/L4P3MA

113 of 125

https://goo.gl/L4P3MA

114 of 125

Slow Query Analysis

Percona’s toolkit has a nice perl script called pt-query-digest. Recommend using that.

https://goo.gl/L4P3MA

115 of 125

Basic query performance measurements with the devel module

https://goo.gl/L4P3MA

116 of 125

https://drupal.org/project/devel

This module can tell you what queries are slow and how many ms is spent in the database and in php.

https://goo.gl/L4P3MA

117 of 125

admin/config/development/devel

https://goo.gl/L4P3MA

118 of 125

admin/config/development/devel

https://goo.gl/L4P3MA

119 of 125

https://goo.gl/L4P3MA

120 of 125

https://goo.gl/L4P3MA

121 of 125

https://goo.gl/L4P3MA

122 of 125

Xdebug/Xhprof

https://goo.gl/L4P3MA

123 of 125

Generating a cachegrind file

This allows you to pinpoint slow points. Once you know the bottlenecks, you can try to figure out what options are available to fix them.

​

xhprof is a good alternative to xdebug.

​

Another alternative to this is to use New Relic. It provides a lot of good information and it comes with a 14 day free trial.��

https://goo.gl/L4P3MA

124 of 125

Questions?

Link to this presentation

Twitter: @mcarper

​

D7 performance wiki: https://groups.drupal.org/node/210683

​

https://goo.gl/L4P3MA

125 of 125

2014 presentation

Link to this presentation (2015)

http://goo.gl/30yi39

​

​

https://goo.gl/L4P3MA