socionetInstall

Server msk.socionet.ru ( /home/astronet/app/www )

Testing postgreSQL 8.2.3: about 530tps with fsync=0, 512Mb shared memory and statistics switched off (With statistics on we got only 350 tps, weird). Testing cmdline:

dropdb pgbench&&createdb pgbench&&pgbench -i pgbench&& \
pg_ctl -D /home/astronet/tmp -o "--fsync=on" restart && \
sleep 3 && pgbench -c 10 -t 3000 pgbench
  • perl modules via cpan
    • G/GA/GAAS/HTML-Parser-3.56.tar.gz
    • J/JH/JHI/BSD-Resource-1.28.tar.gz
    • P/PM/PMQS/Compress-Zlib-2.003.tar.gz
    • G/GA/GAAS/libwww-perl-5.805.tar.gz
    • H/HA/HAYASHI/Term-ReadLine-Gnu-1.16.tar.gz
    • T/TI/TIMB/DBI-1.54.tar.gz
    • D/DB/DBDPG/DBD-Pg-1.49.tar.gz

Setting up Backend

Backend is apache2+modperl (dso)

  • apache2
    • ./configure --prefix=/home/astronet/app/apache --with-mpm=prefork
    • make && make install
  • modperl 2.0.3 (http://perl.apache.org/)
    • perl Makefile.PL MP_AP_PREFIX=/home/astronet/app/apache
    • make && make test
    • su2 -c make install
    • LoadModule perl_module modules/mod_perl.so
    • Enable Registry scripts and standard cgi scripts
AddHandler cgi-script .cgi

<Files *.cgp>
     SetHandler perl-script
      PerlResponseHandler ModPerl::Registry
      PerlOptions +ParseHeaders
      #PerlOptions -GlobalRequest
      Options +ExecCGI
      Order allow,deny
      Allow from all
</Files>

Also, since we won't to bloat server with a lot of heavy modperl backends, we should restrict MaxClients (that means no more than MaxClients database connections !). In conf/extra/httpd-mpm.conf:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients           40
    MaxRequestsPerChild   0
</IfModule>

Backend should be accessed only from localhost, so in httpd.conf:

Listen 127.0.01:8100

We want to get original ip address from X-Forwarded-For header, so we need rpaf ( http://stderr.net/apache/rpaf/ ).

apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

Load it:

LoadModule rpaf_module modules/mod_rpaf-2.0.so
RPAFEnable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 193.232.194.24

Testing Backend

Very simple script:

cat t.cgp
#!/usr/bin/perl

use strict;

print "Content-type: text/html\n\n";
my $a = localtime();
print "Timestamp: $a\n";

cgi-version is the same script (actually, it's a symbolic link)

lrwxrwxr-x  1 astronet  astronet    5 Mar  3 21:43 t.cgi@ -> t.cgp
-rwxrwxr-x  1 astronet  astronet  113 Mar  3 21:19 t.cgp*

For testing purposes we configure test location as follow:

# Testing directory
<Location /mp>
    AllowOverride All
    Options MultiViews Indexes SymLinksIfOwnerMatch Includes ExecCGI
</Location>
  • Registry script
./ab -n 1000 -c 10 http://msk.socionet.ru:8100/mp/t.cgp
Requests per second:    1064.99
  • cgi script
./ab -n 1000 -c 10  http://msk.socionet.ru:8100/mp/t.cgi
Requests per second:    141.64

Resume: We got about 700% performance gain.

Performance gain is much more when working with database:

our $dbh;

if ( !$dbh ) {
  $dbh = DBI->connect("dbi:Pg:dbname=pgbench")  or die $DBI::errstr;
  print "Connect to db...\n";
} else {
  print "Reuse connection...\n";
}

Registry script reuses database handler in addition to the perl interpetator preloaded. That gave us 1131.60 against 18.52 for old cgi script.

Add generic query ( \dt ):

print "<P><table border=1><th><td>Schema</td><td>Name</td><td>ObjType</td><td>Ow
ner</td></th>\n";



my $ary_ref = $dbh->selectall_arrayref($query);

foreach my $r  ( @$ary_ref ) {
  print "<tr><td>",join('</td><td>',@$r),"</td></tr>\n";
}

print "</table>\n";

Again, we use ab to test registry and cgi scripts, which are again, the same.

ab -n 1000 -c 10  http://msk.socionet.ru:8100/mp/tdb.cgp
ab -n 1000 -c 10  http://msk.socionet.ru:8100/mp/tdb.cgi

Now, we have 556.70/17.31. We see, that performance of cgi script mostly defined by connection time. Still, we have 3000% performance gain.

Setting up frontend

Frontend is a fast and light nginx (http://sysoev.ru/nginx)

./configure --prefix=/home/astronet/app/apache \
--with-http_realip_module --with-pcre=../pcre-7.0 \
--http-log-path=logs/fe_access.log

We may serve many simultaneous connections with FE:

events {
    worker_connections  1024;
}

We proxy all requests to the backend (BE) and pass original ip address using special header, so BE could get it.

        location / {

        #charset koi8-r;

        proxy_pass http://127.0.0.1:8100/;
        proxy_redirect     off;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

        client_max_body_size       10m;
        client_body_buffer_size    128k;

        client_body_temp_path      /var/nginx/client_body_temp;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;
        proxy_send_lowat           12000;

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;

        proxy_temp_path            /var/nginx/proxy_temp;

        #access_log  logs/host.access.log  main;

            root   html;
            index  index.html index.htm;
        }

We wont serve binary content by heavy BE, so we add:

        location ~* ^.+\.(jpg|jpeg|gif)$ {
            root         htdocs;
            access_log   off;
            expires      30d;
        }

Images will be not logged , expired after 30 days and not served by BE.

Another FE (apache2) configuration

./configure --prefix=/home/astronet/fe --enable-so \
--enable-proxy --enable-proxy_http  --enable-expires  --enable-headers \
 --enable-info --enable-rewrite

in httpd.conf

ProxyRequests Off

<Proxy *>
        Order deny,allow
        Allow from all
</Proxy>
ProxyPass / http://127.0.0.1:8100/
ProxyPassReverse / http://127.0.0.1:8100/

Experimenting with Mason

To install Mason with mp2/apache2 we need libapreq2 (2.0.8 from CPAN). This page is very useful.

perl Makefile.PL --with-apache2-apxs=/home/astronet/app/apache/bin/apxs
make 
make install

Enable it adding to httpd.conf:

LoadModule apreq_module modules/mod_apreq2.so

Performance testing

  • Siege is a good tool for testing httpd servers. We testing nginx as FE and apache2+modperl as BE.

Registry script - avg. load is 1.4

siege -c 80 -t 10m -d 1 http://127.0.0.1/mp/tdb.cgp
** siege 2.65
** Preparing 80 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.
Transactions:                  92673 hits
Availability:                 100.00 %
Elapsed time:                 600.29 secs
Data transferred:              41.89 MB
Response time:                  0.01 secs
Transaction rate:             154.38 trans/sec
Throughput:                     0.07 MB/sec
Concurrency:                    1.23
Successful transactions:       92673
Failed transactions:               0
Longest transaction:            0.41
Shortest transaction:           0.00

Plain cgi-script - load about 30 ! This is not surprising, since for every request httpd server had to run perl interpretator and connect to database, as it's seen in top output.

 siege -c 80 -t 10m -d 1 http://127.0.0.1/mp/tdb.cgi
** siege 2.65
** Preparing 80 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.
Transactions:                  12925 hits
Availability:                 100.00 %
Elapsed time:                 599.88 secs
Data transferred:               4.98 MB
Response time:                  3.24 secs
Transaction rate:              21.55 trans/sec
Throughput:                     0.01 MB/sec
Concurrency:                   69.80
Successful transactions:       12925
Failed transactions:               0
Longest transaction:            6.02
Shortest transaction:           1.38

As a result we see 7 times lesser requests were proceeded with very high concurrence 70 against 1.23 !

Настройки ядра

  • kern.ipc.zero_copy.send: 0
  • kern.ipc.zero_copy.receive: 0
  • kern.ipc.maxsockbuf: 262144
  • kern.ipc.somaxconn: 32767
  • kern.ipc.maxsockets: 67584
  • kern.ipc.shmmax: 1073741824
  • kern.ipc.shmall: 1073741824

To avoid warning when starting Apache:

kldload accf_http

or add to /boot/loader.conf:

accf_http_load=&#8221;YES&#8221;

See details

MSK2 Setup

  • Create user postgres.postgres'
  • Create cluster with default encoding UTF-8 and locale ru_RU.UTF-8
[postgres@msk2 ~]$ initdb -E UTF-8 -D /work/pgdata --locale=ru_RU.UTF-8
  • Configure restricted access using internal ip
# trust local (internal) network :)
host all all 172.16.172.0/24 trust

# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
#host    all         all         127.0.0.1/32          trust
# IPv6 local connections:
#host    all         all         ::1/128               trust
  • Configure postgresql.conf
[postgres@msk2 ~/save]$ grep -v -E '^#' postgresql.conf | grep -E '^[a-z]'
listen_addresses = '172.16.172.2,172.16.172.1' #'localhost'             # what IP address(es) to listen on;
max_connections = 40                    # (change requires restart)
shared_buffers = 512MB                  # min 128kB or max_connections*16kB
work_mem = 16MB                         # min 64kB
maintenance_work_mem = 32MB             # min 1MB
max_fsm_pages = 179200          # min max_fsm_relations*16, 6 bytes each
log_destination = 'stderr'              # Valid values are combinations of
redirect_stderr = on #off                       # Enable capturing of stderr into log
log_directory = '/var/log/pgsql' #'pg_log'              # Directory where log files are written
log_filename = 'pgsql.log.%Y.%m.%d' #'postgresql-%Y-%m-%d_%H%M%S.log' # Log file name pattern.
log_rotation_size = 0 #10MB             # Automatic rotation of logfiles will
silent_mode = on
log_duration = on #off
log_line_prefix = '%t [%p]: [%l-1] '   # Special values:
stats_block_level = on #off
stats_row_level = on #off
autovacuum = on
datestyle = 'iso, dmy'
lc_messages = 'C'                       # locale for system error message
lc_monetary = 'ru_RU.UTF-8'                     # locale for monetary formatting
lc_numeric = 'ru_RU.UTF-8'                      # locale for number formatting
lc_time = 'ru_RU.UTF-8'                         # locale for time formatting
  • Create writable /var/log/pgsql
  • Start server
pg_ctl -D /work/pgdata -l /dev/null start

Now, database is accessible from msk (ONLY):

[astronet@MSK ~]$ psql -h 172.16.172.2 -Upostgres -l
        List of databases
   Name    |  Owner   | Encoding
-----------+----------+----------
 postgres  | postgres | UTF8
 template0 | postgres | UTF8
 template1 | postgres | UTF8
(3 rows)

For production, I recommend to switch off stats_block_level and stats_row_level to reduce overhead of statistic collector (db reinited every night from scratch).

The format of log files is compatible with pgfouine, so it's possible to prepare database performance reports and identify slow queries.

Current modperl configuration contained in /usr/local/etc/apache22/httpd.conf, and startup.pl.

siege -c 10 -t 1m -d 1 'http://msk.socionet.ru/cgi/xml/discipline.cgi?h=economics'
** siege 2.65
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.                                         Transactions:                     220 hits
Availability:                 100.00 %
Elapsed time:                  60.27 secs
Data transferred:               0.33 MB
Response time:                  2.27 secs
Transaction rate:               3.65 trans/sec
Throughput:                     0.01 MB/sec
Concurrency:                    8.27
Successful transactions:         220
Failed transactions:               0
Longest transaction:            3.39
Shortest transaction:           0.85

Now, we use modperl version

 siege -c 20 -t 1m -d 1 'http://msk.socionet.ru/test/discipline.cgp?h=economics'
** siege 2.65
** Preparing 20 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.                                         Transactions:                    2393 hits
Availability:                 100.00 %
Elapsed time:                  60.02 secs
Data transferred:               0.68 MB
Response time:                  0.08 secs
Transaction rate:              39.87 trans/sec
Throughput:                     0.01 MB/sec
Concurrency:                    3.35
Successful transactions:        2393
Failed transactions:               0
Longest transaction:            1.11
Shortest transaction:           0.02

We got 10x performance gain ! We have one more optimization - persistent database connection. It's not easy, since it require rewrites of mysqlconnect.pm, which is not reentrant right now. Notice, real performance gain depends on cocoon performance, but it's outside of my task.

Testing original .xml processing

 siege -c 10 -t 1m -d 1 'http://msk.socionet.ru/discipline.xml?h=economics'
** siege 2.65
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.                                         Transactions:                     178 hits
Availability:                 100.00 %
Elapsed time:                  60.29 secs
Data transferred:               2.66 MB
Response time:                  2.73 secs
Transaction rate:               2.95 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                    8.06
Successful transactions:         178
Failed transactions:               0
Longest transaction:            5.25
Shortest transaction:           1.14

and modperl-enabled

 siege -c 10 -t 1m -d 1 'http://msk.socionet.ru/discipline_p.xml?h=economics'
** siege 2.65
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.                                         Transactions:                     202 hits
Availability:                 100.00 %
Elapsed time:                  60.16 secs
Data transferred:               2.99 MB
Response time:                  2.46 secs
Transaction rate:               3.36 trans/sec
Throughput:                     0.05 MB/sec
Concurrency:                    8.27
Successful transactions:         202
Failed transactions:               0
Longest transaction:            5.80
Shortest transaction:           0.84

We see, that problem mostly in cocoon.