![[Home]](/~megera/oddmuse/zen3.gif)
Server msk.socionet.ru ( /home/astronet/app/www )
Testing postgreSQL 8.2.3: about 530tps with fsync=0, 512Mb shared memory and statistics switched off (With statistics on we got only 350 tps, weird). Testing cmdline:
dropdb pgbench&&createdb pgbench&&pgbench -i pgbench&& \ pg_ctl -D /home/astronet/tmp -o "--fsync=on" restart && \ sleep 3 && pgbench -c 10 -t 3000 pgbench
Backend is apache2+modperl (dso)
AddHandler cgi-script .cgi
<Files *.cgp>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
#PerlOptions -GlobalRequest
Options +ExecCGI
Order allow,deny
Allow from all
</Files>
Also, since we won't to bloat server with a lot of heavy modperl backends, we should restrict MaxClients (that means no more than MaxClients database connections !). In conf/extra/httpd-mpm.conf:
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 40
MaxRequestsPerChild 0
</IfModule>
Backend should be accessed only from localhost, so in httpd.conf:
Listen 127.0.01:8100
We want to get original ip address from X-Forwarded-For header, so we need rpaf ( http://stderr.net/apache/rpaf/ ).
apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
Load it:
LoadModule rpaf_module modules/mod_rpaf-2.0.so RPAFEnable On RPAFsethostname On RPAFproxy_ips 127.0.0.1 193.232.194.24
Very simple script:
cat t.cgp #!/usr/bin/perl use strict; print "Content-type: text/html\n\n"; my $a = localtime(); print "Timestamp: $a\n";
cgi-version is the same script (actually, it's a symbolic link)
lrwxrwxr-x 1 astronet astronet 5 Mar 3 21:43 t.cgi@ -> t.cgp -rwxrwxr-x 1 astronet astronet 113 Mar 3 21:19 t.cgp*
For testing purposes we configure test location as follow:
# Testing directory
<Location /mp>
AllowOverride All
Options MultiViews Indexes SymLinksIfOwnerMatch Includes ExecCGI
</Location>
./ab -n 1000 -c 10 http://msk.socionet.ru:8100/mp/t.cgp Requests per second: 1064.99
./ab -n 1000 -c 10 http://msk.socionet.ru:8100/mp/t.cgi Requests per second: 141.64
Resume: We got about 700% performance gain.
Performance gain is much more when working with database:
our $dbh;
if ( !$dbh ) {
$dbh = DBI->connect("dbi:Pg:dbname=pgbench") or die $DBI::errstr;
print "Connect to db...\n";
} else {
print "Reuse connection...\n";
}
Registry script reuses database handler in addition to the perl interpetator preloaded. That gave us 1131.60 against 18.52 for old cgi script.
Add generic query ( \dt ):
print "<P><table border=1><th><td>Schema</td><td>Name</td><td>ObjType</td><td>Ow
ner</td></th>\n";
my $ary_ref = $dbh->selectall_arrayref($query);
foreach my $r ( @$ary_ref ) {
print "<tr><td>",join('</td><td>',@$r),"</td></tr>\n";
}
print "</table>\n";
Again, we use ab to test registry and cgi scripts, which are again, the same.
ab -n 1000 -c 10 http://msk.socionet.ru:8100/mp/tdb.cgp ab -n 1000 -c 10 http://msk.socionet.ru:8100/mp/tdb.cgi
Now, we have 556.70/17.31. We see, that performance of cgi script mostly defined by connection time. Still, we have 3000% performance gain.
Frontend is a fast and light nginx (http://sysoev.ru/nginx)
./configure --prefix=/home/astronet/app/apache \ --with-http_realip_module --with-pcre=../pcre-7.0 \ --http-log-path=logs/fe_access.log
We may serve many simultaneous connections with FE:
events {
worker_connections 1024;
}
We proxy all requests to the backend (BE) and pass original ip address using special header, so BE could get it.
location / {
#charset koi8-r;
proxy_pass http://127.0.0.1:8100/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
client_body_temp_path /var/nginx/client_body_temp;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_send_lowat 12000;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_temp_path /var/nginx/proxy_temp;
#access_log logs/host.access.log main;
root html;
index index.html index.htm;
}
We wont serve binary content by heavy BE, so we add:
location ~* ^.+\.(jpg|jpeg|gif)$ {
root htdocs;
access_log off;
expires 30d;
}
Images will be not logged , expired after 30 days and not served by BE.
./configure --prefix=/home/astronet/fe --enable-so \ --enable-proxy --enable-proxy_http --enable-expires --enable-headers \ --enable-info --enable-rewrite
in httpd.conf
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://127.0.0.1:8100/
ProxyPassReverse / http://127.0.0.1:8100/
To install Mason with mp2/apache2 we need libapreq2 (2.0.8 from CPAN). This page is very useful.
perl Makefile.PL --with-apache2-apxs=/home/astronet/app/apache/bin/apxs make make install
Enable it adding to httpd.conf:
LoadModule apreq_module modules/mod_apreq2.so
Registry script - avg. load is 1.4
siege -c 80 -t 10m -d 1 http://127.0.0.1/mp/tdb.cgp ** siege 2.65 ** Preparing 80 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 92673 hits Availability: 100.00 % Elapsed time: 600.29 secs Data transferred: 41.89 MB Response time: 0.01 secs Transaction rate: 154.38 trans/sec Throughput: 0.07 MB/sec Concurrency: 1.23 Successful transactions: 92673 Failed transactions: 0 Longest transaction: 0.41 Shortest transaction: 0.00
Plain cgi-script - load about 30 ! This is not surprising, since for every request httpd server had to run perl interpretator and connect to database, as it's seen in top output.
siege -c 80 -t 10m -d 1 http://127.0.0.1/mp/tdb.cgi ** siege 2.65 ** Preparing 80 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 12925 hits Availability: 100.00 % Elapsed time: 599.88 secs Data transferred: 4.98 MB Response time: 3.24 secs Transaction rate: 21.55 trans/sec Throughput: 0.01 MB/sec Concurrency: 69.80 Successful transactions: 12925 Failed transactions: 0 Longest transaction: 6.02 Shortest transaction: 1.38
As a result we see 7 times lesser requests were proceeded with very high concurrence 70 against 1.23 !
To avoid warning when starting Apache:
kldload accf_http
or add to /boot/loader.conf:
accf_http_load=”YES”
See details
[postgres@msk2 ~]$ initdb -E UTF-8 -D /work/pgdata --locale=ru_RU.UTF-8
# trust local (internal) network :) host all all 172.16.172.0/24 trust # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: #host all all 127.0.0.1/32 trust # IPv6 local connections: #host all all ::1/128 trust
[postgres@msk2 ~/save]$ grep -v -E '^#' postgresql.conf | grep -E '^[a-z]' listen_addresses = '172.16.172.2,172.16.172.1' #'localhost' # what IP address(es) to listen on; max_connections = 40 # (change requires restart) shared_buffers = 512MB # min 128kB or max_connections*16kB work_mem = 16MB # min 64kB maintenance_work_mem = 32MB # min 1MB max_fsm_pages = 179200 # min max_fsm_relations*16, 6 bytes each log_destination = 'stderr' # Valid values are combinations of redirect_stderr = on #off # Enable capturing of stderr into log log_directory = '/var/log/pgsql' #'pg_log' # Directory where log files are written log_filename = 'pgsql.log.%Y.%m.%d' #'postgresql-%Y-%m-%d_%H%M%S.log' # Log file name pattern. log_rotation_size = 0 #10MB # Automatic rotation of logfiles will silent_mode = on log_duration = on #off log_line_prefix = '%t [%p]: [%l-1] ' # Special values: stats_block_level = on #off stats_row_level = on #off autovacuum = on datestyle = 'iso, dmy' lc_messages = 'C' # locale for system error message lc_monetary = 'ru_RU.UTF-8' # locale for monetary formatting lc_numeric = 'ru_RU.UTF-8' # locale for number formatting lc_time = 'ru_RU.UTF-8' # locale for time formatting
pg_ctl -D /work/pgdata -l /dev/null start
Now, database is accessible from msk (ONLY):
[astronet@MSK ~]$ psql -h 172.16.172.2 -Upostgres -l
List of databases
Name | Owner | Encoding
-----------+----------+----------
postgres | postgres | UTF8
template0 | postgres | UTF8
template1 | postgres | UTF8
(3 rows)
For production, I recommend to switch off stats_block_level and stats_row_level to reduce overhead of statistic collector (db reinited every night from scratch).
The format of log files is compatible with pgfouine, so it's possible to prepare database performance reports and identify slow queries.
Current modperl configuration contained in /usr/local/etc/apache22/httpd.conf, and startup.pl.
siege -c 10 -t 1m -d 1 'http://msk.socionet.ru/cgi/xml/discipline.cgi?h=economics' ** siege 2.65 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 220 hits Availability: 100.00 % Elapsed time: 60.27 secs Data transferred: 0.33 MB Response time: 2.27 secs Transaction rate: 3.65 trans/sec Throughput: 0.01 MB/sec Concurrency: 8.27 Successful transactions: 220 Failed transactions: 0 Longest transaction: 3.39 Shortest transaction: 0.85
Now, we use modperl version
siege -c 20 -t 1m -d 1 'http://msk.socionet.ru/test/discipline.cgp?h=economics' ** siege 2.65 ** Preparing 20 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 2393 hits Availability: 100.00 % Elapsed time: 60.02 secs Data transferred: 0.68 MB Response time: 0.08 secs Transaction rate: 39.87 trans/sec Throughput: 0.01 MB/sec Concurrency: 3.35 Successful transactions: 2393 Failed transactions: 0 Longest transaction: 1.11 Shortest transaction: 0.02
We got 10x performance gain ! We have one more optimization - persistent database connection. It's not easy, since it require rewrites of mysqlconnect.pm, which is not reentrant right now. Notice, real performance gain depends on cocoon performance, but it's outside of my task.
Testing original .xml processing
siege -c 10 -t 1m -d 1 'http://msk.socionet.ru/discipline.xml?h=economics' ** siege 2.65 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 178 hits Availability: 100.00 % Elapsed time: 60.29 secs Data transferred: 2.66 MB Response time: 2.73 secs Transaction rate: 2.95 trans/sec Throughput: 0.04 MB/sec Concurrency: 8.06 Successful transactions: 178 Failed transactions: 0 Longest transaction: 5.25 Shortest transaction: 1.14
and modperl-enabled
siege -c 10 -t 1m -d 1 'http://msk.socionet.ru/discipline_p.xml?h=economics' ** siege 2.65 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 202 hits Availability: 100.00 % Elapsed time: 60.16 secs Data transferred: 2.99 MB Response time: 2.46 secs Transaction rate: 3.36 trans/sec Throughput: 0.05 MB/sec Concurrency: 8.27 Successful transactions: 202 Failed transactions: 0 Longest transaction: 5.80 Shortest transaction: 0.84
We see, that problem mostly in cocoon.
