Skip to content

Instantly share code, notes, and snippets.

@parhamr

parhamr/0: Magento 1.12 Enterprise multi-store cluster configuration.md

Last active October 13, 2023 14:20
Show Gist options
  • Save parhamr/6177160 to your computer and use it in GitHub Desktop.
Save parhamr/6177160 to your computer and use it in GitHub Desktop.
Download ZIP
A highly available, fault tolerant, distributed, and load balanced LEMP cluster.
Raw
0: Magento 1.12 Enterprise multi-store cluster configuration.md

Magento Enterprise 1.12 cluster

Scaling Magento presentation with 60 slides and notes, which was then reviewed by Alan Storm.

Table of Contents

Summary

  • Peak capacity of 100,000 orders in a day (4,000 an hour or 80/minute burst)
  • 806,000 lines of code (100,000 lines more than core + Zend)
  • 97% PHP
  • 1.6% Ruby
  • 0.45% Perl
  • 0.28% Shell
  • 0.12% Python
  • Admin users: 46
  • Nearly 2,000 internal users (managers and employees)
  • Categories: 2,450 (largest one has 2,400 products)
  • Product entities: 101,000
  • Combo products: 484
  • Product relations: 54,000
  • In stock and enabled configurable products: 10,100
  • CMS blocks: 3,100
  • CMS pages: 1,400

Traffic Statistics

August 2013 Traffic

  • 40 million monthly pageviews
  • 2.3 million unique visitors
  • 46,000 monthly checkouts
  • 89% of visitors from the USA
  • 34% of visits from iOS
  • 14% of visits from Android

November 2013 Traffic

  • 48 million monthly pageviews
  • 2.8 million unique visitors
  • 68,000 monthly checkouts
  • 89% of visitors from the USA
  • 39% of visits from iOS
  • 16% of visits from Android

December 2013 Traffic

  • 63 million monthly pageviews
  • 3.6 million unique visitors
  • 154,000 monthly checkouts (peak hour: 1,364 transactions)
  • 91% of visitors from the USA
  • 39% of visits from iOS
  • 17% of visits from Android

January 2014 Traffic

  • 42 million monthly pageviews
  • 2.5 million unique visitors
  • 58,000 monthly checkouts
  • 89% of visitors from the USA
  • 42% of visits from iOS
  • 17% of visits from Android

August 2014 Traffic

  • 59 million monthly pageviews
  • 3.4 million unique visitors
  • 97,000 monthly checkouts
  • 88% of visitors from the USA
  • 44% of visits from iOS
  • 20% of visits from Android

November 2014 Traffic

  • 59 million monthly pageviews
  • 3.6 million unique visitors
  • 106,000 monthly checkouts (peak hour: 826 transactions)
  • 88% of visitors from the USA
  • 41% of visits from iOS
  • 19% of visits from Android

December 2014 Traffic

  • 68 million monthly pageviews
  • 4.3 million unique visitors
  • 202,000 monthly checkouts (peak hour: 2,300 transactions)
  • 90% of visitors from the USA
  • 42% of visits from iOS
  • 21% of visits from Android

Web hosts

There are 10 hosts behind redundant, highly available hardware firewalls and hardware load balancers.

All product media assets are managed and served through Adobe’s Scene 7 service. All web media assets are managed through SFTP to a shared media host and distributed in production through Akamai CDNs. Some media assets are deployed to web servers through the application repository.

Performance

Over the first 90 days:

  • site-wide average response time: 282 ms
  • load average: 0.6 to 1.0 (in tests, performance degrades by 35% when load averages hit ~5.0)
  • CPU utilization averages 14%
  • disk IO averages 3%
  • network throughput averages 5 mbps transmitted and 17 mbps received

Configuration

  • Ubuntu 12.04 LTS 64-bit
  • 3.5.0-36-generic GNU/Linux kernel
  • Dual Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (4 cores each)
  • 32 GB DDR3 1333 MHz RAM
  • PHP-FPM, 5.3.10-1ubuntu3.7

Modules

Cache hosts

There are two hosts running Redis in a master-slave configuration with automated failover.

Three Redis instances are used to increase throughput and provide fine-tuning of persistence behaviors.

Performance

Over the first 90 days:

  • 3,000 commands per second
  • 0.7 ms average response time
  • load average of 1.0 to 1.5
  • CPU utilization averages 7%
  • disk IO averages 11%
  • network throughput averages 150 mbps transmitted and 30 mbps received

Sessions

  • average of 6–8 million keys (20–26 GB)
  • commands per second: ~400 (peaking at 1,500 under load)
  • expirations per second: ~40

Backend

  • average of ~1 million keys (6–10 GB)
  • commands per second: ~3,250 (peaking at 5,000 under load)
  • expirations per second: ~100
  • hit rate: ~85%

FPC

  • average of ~450,000 keys (4–8 GB)
  • commands per second: ~2,000 (peaking at 5,000 under load)
  • expirations per second: ~10
  • hit rate: ~80%

Configuration

  • Ubuntu 12.04 LTS 64-bit
  • 3.2.0-48-generic GNU/Linux kernel
  • Quad Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (6 cores each)
  • 128 GB buffered DDR3 1333 MHz RAM
  • Mechanical disks, RAID 1, hardware controller
  • Redis 2.6.11

Database hosts

There are two hosts running MySQL 5.6.11 in a master-slave configuration with warm failover.

Performance

Over the first 90 days:

  • 1,500 commands per second
  • 1.1 ms average response time
  • load average of 0.1 (master) and 0.4 (slave)
  • CPU utilization averages 0% (master) and 1.5% (slave)
  • disk IO averages 1% (master) and 0.5% (slave)
  • network throughput averages 40 mbps transmitted (slave) and 0.7 mbps received (master)

Statistics

Over the first 100 days:

  • 42 TB of transmitted data
  • 23 TB of innodb writes
  • 8.4 TB of innodb log churn
  • 100% thread cache hit rate
  • 99.996% table cache hit rate (large number of open tables possibly related to MySQL bugs #16244691 and #65384)
  • 99.9999993% of table locks are immediate (most are nightly processes)
  • Innodb_buffer_pool_wait_free: 0
  • Innodb_log_waits: 0
  • 85% query cache hit rate; this doesn’t really mean anything with such high churn rate of data
  • 94.5% of temp tables are in memory (only nightly processes require disk tables)
  • 99.97% of queries are faster than 200 ms; we’ve reached a plateau of optimization
  • The average row lock is a bit slow as a consequence of Magento indexing architecture and our background processing queues
  • Moderate rate of random and sequential reads (table scans!), but we can absorb that overhead with hardware and focus on improving PHP code

Configuration

Host:

  • Ubuntu 12.04 LTS 64-bit
  • 3.5.0-34-generic GNU/Linux
  • Quad Intel(R) Xeon(R) CPU E7- 2860 @ 2.27GHz (10 cores each)
  • 128 GB buffered DDR3 1333 MHz RAM
  • SSD, RAID 1+0, hardware controller
  • MySQL 5.6.11 with tcmalloc

See also: MySQL configurations

Utility hosts

These run cron and systems tasks. Daily backups are stored offsite and verified functional! :)

Scripts are in: Bash, Ruby, Perl, Python, and PHP.

Chef Server 10 for deployments and configuration management. Database tests, using Perl’s Test::DatabaseRow, are used to catch some human errors.

Raw
1: Magento core_config_data
mysql> select path, value, scope_id from core_config_data where path like 'web%' OR path like 'system%' or path like 'admin%' or path like 'api%' or path like 'cache%' or path like 'dev%' order by path\G;
*************************** 1. row ***************************
path: admin/captcha/enable
value: 0
scope_id: 0
*************************** 2. row ***************************
path: admin/dashboard/enable_charts
value: 1
scope_id: 0
*************************** 7. row ***************************
path: admin/security/lockout_failures
value: 6
scope_id: 0
*************************** 8. row ***************************
path: admin/security/lockout_threshold
value: 30
scope_id: 0
*************************** 9. row ***************************
path: admin/security/password_is_forced
value: 1
scope_id: 0
*************************** 10. row ***************************
path: admin/security/password_lifetime
value: 90
scope_id: 0
*************************** 11. row ***************************
path: admin/security/session_cookie_lifetime
value: 900
scope_id: 0
*************************** 12. row ***************************
path: admin/security/use_case_sensitive_login
value: 0
scope_id: 0
*************************** 13. row ***************************
path: admin/security/use_form_key
value: 1
scope_id: 0
*************************** 14. row ***************************
path: admin/startup/page
value: dashboard
scope_id: 0
*************************** 15. row ***************************
path: admin/url/use_custom
value: 0
scope_id: 0
*************************** 16. row ***************************
path: admin/url/use_custom_path
value: 0
scope_id: 0
*************************** 17. row ***************************
path: api/config/charset
value: UTF-8
scope_id: 0
*************************** 18. row ***************************
path: api/config/compliance_wsi
value: 0
scope_id: 0
*************************** 19. row ***************************
path: api/config/session_timeout
value: 86400
scope_id: 0
*************************** 20. row ***************************
path: api/config/wsdl_cache_enabled
value: 1
scope_id: 0
*************************** 21. row ***************************
path: cache/expires/catalog_category_view
value:
scope_id: 0
*************************** 22. row ***************************
path: cache/expires/catalog_product_list
value:
scope_id: 0
*************************** 23. row ***************************
path: cache/expires/cms_block
value:
scope_id: 0
*************************** 24. row ***************************
path: cache/expires/cms_page
value:
scope_id: 0
*************************** 25. row ***************************
path: cache/expires/lightspeed
value:
scope_id: 0
*************************** 26. row ***************************
path: cache/expires/page_cache
value:
scope_id: 0
*************************** 163. row ***************************
path: dev/css/merge_css_files
value: 0
scope_id: 0
*************************** 164. row ***************************
path: dev/debug/profiler
value: 0
scope_id: 0
*************************** 165. row ***************************
path: dev/js/deprecation
value: 0
scope_id: 0
*************************** 166. row ***************************
path: dev/js/merge_files
value: 0
scope_id: 0
*************************** 167. row ***************************
path: dev/log/active
value: 1
scope_id: 0
*************************** 168. row ***************************
path: dev/log/exception_file
value: exception.log
scope_id: 0
*************************** 169. row ***************************
path: dev/log/file
value: system.log
scope_id: 0
*************************** 173. row ***************************
path: dev/restrict/allow_ips
value: NULL
scope_id: 0
*************************** 174. row ***************************
path: dev/template/allow_symlink
value: 1
scope_id: 0
*************************** 175. row ***************************
path: dev/translate_inline/active
value: 0
scope_id: 0
*************************** 176. row ***************************
path: dev/translate_inline/active_admin
value: 0
scope_id: 0
*************************** 288. row ***************************
path: system/adminnotification/frequency
value: 24
scope_id: 0
*************************** 289. row ***************************
path: system/adminnotification/use_https
value: 1
scope_id: 0
*************************** 290. row ***************************
path: system/backup/enabled
value: 0
scope_id: 0
*************************** 291. row ***************************
path: system/cron/history_cleanup_every
value: 10
scope_id: 0
*************************** 292. row ***************************
path: system/cron/history_failure_lifetime
value: 600
scope_id: 0
*************************** 293. row ***************************
path: system/cron/history_success_lifetime
value: 60
scope_id: 0
*************************** 294. row ***************************
path: system/cron/schedule_ahead_for
value: 20
scope_id: 0
*************************** 295. row ***************************
path: system/cron/schedule_generate_every
value: 15
scope_id: 0
*************************** 296. row ***************************
path: system/cron/schedule_lifetime
value: 15
scope_id: 0
*************************** 297. row ***************************
path: system/currency/installed
value: AFN,ALL,DZD,AOA,ARS,AMD,AWG,AUD,AZN,AZM,BSD,BHD,BDT,BBD,BYR,BZD,BMD,BTN,BOB,BAM,BWP,BRL,GBP,BND,BGN,BUK,BIF,XOF,XPF,KHR,CAD,CVE,KYD,CLP,CNY,COP,KMF,CDF,CRC,HRK,CUP,CZK,DKK,DJF,DOP,XCD,EGP,SVC,GQE,ERN,EEK,ETB,EUR,FKP,FJD,GMD,GEK,GHS,GIP,GTQ,GNF,GYD,HTG,HNL,HKD,HUF,ISK,INR,IDR,IRR,IQD,JMD,JPY,JOD,KZT,KES,KWD,KGS,LAK,LVL,LBP,LSL,LRD,LYD,LTL,MOP,MKD,MGA,MWK,MYR,MVR,LSM,MRO,MUR,MXN,MDL,MNT,MAD,MZN,MMK,NAD,NPR,ANG,ILS,NZD,NIC,NGN,KPW,NOK,ROL,TRL,OMR,PKR,PAB,PGK,PYG,PEN,PHP,PLN,QAR,RHD,RON,RUB,RWF,SHP,STD,SAR,RSD,SCR,SLL,SGD,SKK,SBD,SOS,ZAR,KRW,LKR,SDG,SRD,SZL,SEK,CHF,SYP,TWD,TJS,TZS,THB,TOP,TTD,TND,TRY,TMM,USD,UGX,UAH,AED,UYU,UZS,VUV,VEB,VND,CHE,CHW,WST,YER,ZMK,ZWD
scope_id: 0
*************************** 298. row ***************************
path: system/discountoptimiser/disable_for_add_to_cart
value: 0
scope_id: 0
*************************** 299. row ***************************
path: system/discountoptimiser/enabled
value: 0
scope_id: 0
*************************** 300. row ***************************
path: system/guidance_cachebuster/is_enabled
value: 1
scope_id: 0
*************************** 301. row ***************************
path: system/log/clean_after_day
value: 180
scope_id: 0
*************************** 302. row ***************************
path: system/log/enabled
value: 0
scope_id: 0
*************************** 303. row ***************************
path: system/log/error_email
value:
scope_id: 0
*************************** 304. row ***************************
path: system/log/error_email_identity
value: general
scope_id: 0
*************************** 305. row ***************************
path: system/log/error_email_template
value: system_log_error_email_template
scope_id: 0
*************************** 306. row ***************************
path: system/log/frequency
value: D
scope_id: 0
*************************** 307. row ***************************
path: system/log/time
value: 00,00,00
scope_id: 0
*************************** 308. row ***************************
path: system/page_cache/allowed_depth
value: 4
scope_id: 0
*************************** 309. row ***************************
path: system/page_cache/max_cache_size
value: 8192
scope_id: 0
*************************** 310. row ***************************
path: system/page_crawl/enable
value: 0
scope_id: 0
*************************** 311. row ***************************
path: system/smtp/disable
value: 0
scope_id: 0
*************************** 312. row ***************************
path: system/smtp/host
value: localhost
scope_id: 0
*************************** 313. row ***************************
path: system/smtp/port
value: 25
scope_id: 0
*************************** 314. row ***************************
path: system/smtppro/option
value: smtp
scope_id: 0
*************************** 315. row ***************************
path: system/smtpsettings/authentication
value: login
scope_id: 0
*************************** 316. row ***************************
path: system/smtpsettings/host
value: smtp.sendgrid.net
scope_id: 0
*************************** 317. row ***************************
path: system/smtpsettings/password
value: [redacted]
scope_id: 0
*************************** 318. row ***************************
path: system/smtpsettings/port
value: 587
scope_id: 0
*************************** 319. row ***************************
path: system/smtpsettings/ssl
value: tls
scope_id: 0
*************************** 320. row ***************************
path: system/smtpsettings/username
value: company-name
scope_id: 0
*************************** 321. row ***************************
path: web/browser_capabilities/cookies
value: 1
scope_id: 0
*************************** 322. row ***************************
path: web/browser_capabilities/javascript
value: 1
scope_id: 0
*************************** 323. row ***************************
path: web/cookie/cookie_domain
value: NULL
scope_id: 0
*************************** 324. row ***************************
path: web/cookie/cookie_httponly
value: 1
scope_id: 0
*************************** 325. row ***************************
path: web/cookie/cookie_lifetime
value: 7776000
scope_id: 0
*************************** 326. row ***************************
path: web/cookie/cookie_path
value: NULL
scope_id: 0
*************************** 327. row ***************************
path: web/cookie/cookie_restriction
value: 0
scope_id: 0
*************************** 328. row ***************************
path: web/default/cms_home_page
value: home
scope_id: 0
*************************** 329. row ***************************
path: web/default/cms_home_page
value: internal_home
scope_id: 2
*************************** 330. row ***************************
path: web/default/cms_no_cookies
value: enable-cookies
scope_id: 0
*************************** 331. row ***************************
path: web/default/cms_no_route
value: page-not-found-flat
scope_id: 0
*************************** 332. row ***************************
path: web/default/cms_store_home_page
value: internal_home
scope_id: 0
*************************** 333. row ***************************
path: web/default/front
value: cms
scope_id: 0
*************************** 334. row ***************************
path: web/default/no_route
value: cms/index/noRoute
scope_id: 0
*************************** 335. row ***************************
path: web/default/show_cms_breadcrumbs
value: 1
scope_id: 0
*************************** 336. row ***************************
path: web/polls/poll_check_by_ip
value: 0
scope_id: 0
*************************** 337. row ***************************
path: web/secure/base_js_url
value: {{secure_base_url}}js/
scope_id: 0
*************************** 338. row ***************************
path: web/secure/base_js_url
value: https://internal.company-name.com/js/
scope_id: 2
*************************** 339. row ***************************
path: web/secure/base_link_url
value: {{secure_base_url}}
scope_id: 0
*************************** 340. row ***************************
path: web/secure/base_media_url
value: https://static.company-name.com/skin/frontend/company-name/default/
scope_id: 0
*************************** 341. row ***************************
path: web/secure/base_media_url
value: https://internal.company-name.com/skin/
scope_id: 2
*************************** 342. row ***************************
path: web/secure/base_media_url
value: https://static.company-name.com/skin/frontend/company-name/default/
scope_id: 2
*************************** 343. row ***************************
path: web/secure/base_skin_url
value: {{secure_base_url}}skin/
scope_id: 0
*************************** 344. row ***************************
path: web/secure/base_skin_url
value: {{secure_base_url}}skin/
scope_id: 2
*************************** 345. row ***************************
path: web/secure/base_url
value: https://www.company-name.com/
scope_id: 0
*************************** 346. row ***************************
path: web/secure/base_url
value: https://internal.company-name.com/
scope_id: 2
*************************** 347. row ***************************
path: web/secure/offloader_header
value: SSL_OFFLOADED
scope_id: 0
*************************** 348. row ***************************
path: web/secure/use_in_adminhtml
value: 1
scope_id: 0
*************************** 349. row ***************************
path: web/secure/use_in_frontend
value: 1
scope_id: 0
*************************** 350. row ***************************
path: web/seo/use_rewrites
value: 1
scope_id: 0
*************************** 351. row ***************************
path: web/session/use_frontend_sid
value: 0
scope_id: 0
*************************** 352. row ***************************
path: web/session/use_http_user_agent
value: 0
scope_id: 0
*************************** 353. row ***************************
path: web/session/use_http_via
value: 0
scope_id: 0
*************************** 354. row ***************************
path: web/session/use_http_x_forwarded_for
value: 1
scope_id: 0
*************************** 355. row ***************************
path: web/session/use_remote_addr
value: 0
scope_id: 0
*************************** 356. row ***************************
path: web/unsecure/base_js_url
value: {{unsecure_base_url}}js/
scope_id: 0
*************************** 357. row ***************************
path: web/unsecure/base_js_url
value: http://internal.company-name.com/js/
scope_id: 2
*************************** 358. row ***************************
path: web/unsecure/base_link_url
value: {{unsecure_base_url}}
scope_id: 0
*************************** 359. row ***************************
path: web/unsecure/base_media_url
value: http://static.company-name.com/skin/frontend/company-name/default/
scope_id: 0
*************************** 360. row ***************************
path: web/unsecure/base_media_url
value: http://static.company-name.com/skin/frontend/company-name/default/
scope_id: 2
*************************** 361. row ***************************
path: web/unsecure/base_skin_url
value: {{unsecure_base_url}}skin/
scope_id: 0
*************************** 362. row ***************************
path: web/unsecure/base_skin_url
value: {{unsecure_base_url}}skin/
scope_id: 2
*************************** 363. row ***************************
path: web/unsecure/base_url
value: http://www.company-name.com/
scope_id: 0
*************************** 364. row ***************************
path: web/unsecure/base_url
value: http://internal.company-name.com/
scope_id: 2
*************************** 365. row ***************************
path: web/url/redirect_to_base
value: 0
scope_id: 0
*************************** 366. row ***************************
path: web/url/use_store
value: 0
scope_id: 0
Raw
2: deployment
Open source Chef Server 10.18.2
The following steps are performed by a Shell script when executed as user deploy on a utility host.
1. Initialize variables
2. Get the latest revision hash from GitHub
3. Disable availability monitoring to prevent alerts over HTTP 503 status
4. Switch all web and utility hosts to maintenance mode
5. Deploy utility
1. Stop Magento’s cron and all Resque workers
2. Address filesystem dependencies
3. Chef checks out the defined revision as a new release
4. Address Magento dependencies
5. Update all cron tasks and scripts for automation
6. Deploy all modules
7. Clear cache
8. Run any migrations
9. Start the Magento cron
10. Start Resque workers
6. Deploy the first web host
1. Address filesystem dependencies
2. Chef checks out the defined revision as a new release
3. Address Magento dependencies
4. Deploy all Magento modules
7. Mark a new deployment in New Relic
8. Enable availability monitoring services
9. Disable load balancer services to all other web hosts
10. Continue deployments on web hosts, sequentially bringing them online
11. Perform Chef routines for the Endeca hosts
Raw
cache — etc:network:interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
bond-master bond0
auto eth1
iface eth1 inet manual
bond-master bond0
auto eth2
iface eth2 inet manual
bond-master bond0
auto eth3
iface eth3 inet manual
bond-master bond0
auto bond0
iface bond0 inet static
address 10.150.0.18
netmask 255.255.255.0
gateway 10.150.0.1
slaves all
bond-miimon 100
bond-mode 5
up /sbin/ifenslave bond0 eth0 eth1 eth2 eth3
down /sbin/ifenslave -d bond0 eth0 eth1 eth2 eth3
Raw
cache — etc:redis:6382.conf
# Redis configuration file for Magento sessions
daemonize yes
pidfile /var/run/redis/6382/redis_6382.pid
port 6382
bind 10.1.0.18
tcp-keepalive 15
timeout 0
loglevel notice
logfile /var/log/redis/6382.log
syslog-enabled no
syslog-ident redis-6382
syslog-facility local0
databases 16
save 900 1
save 300 10
save 60 40000
rdbcompression yes
dbfilename dump-6382.rdb
dir /var/lib/redis
maxclients 10000
maxmemory 32GB
maxmemory-policy volatile-lru
maxmemory-samples 3
appendonly no
appendfilename appendonly-6382.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
hz 100
Raw
cache — etc:redis:6383.conf
# Redis configuration file for Magento backend
daemonize yes
pidfile /var/run/redis/6383/redis_6383.pid
port 6383
bind 10.1.0.18
tcp-keepalive 15
timeout 0
loglevel notice
logfile /var/log/redis/6383.log
syslog-enabled no
syslog-ident redis-6383
syslog-facility local0
databases 16
# 17 minute save periods reduce overlap with sessions BGSAVE
save 1020 1
# 7 minute save periods reduce overlap with sessions BGSAVE
save 420 400000
rdbcompression yes
dbfilename dump-6383.rdb
dir /var/lib/redis
maxclients 10000
maxmemory 48GB
maxmemory-policy volatile-lru
maxmemory-samples 3
appendonly no
appendfilename appendonly-6383.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
hz 100
Raw
cache — etc:redis:6384.conf
# Redis configuration file for Magento FPC
daemonize yes
pidfile /var/run/redis/6384/redis_6384.pid
port 6384
bind 10.1.0.18
tcp-keepalive 15
timeout 0
loglevel notice
logfile /var/log/redis/6384.log
syslog-enabled no
syslog-ident redis-6384
syslog-facility local0
databases 16
# 17 minute save periods reduce overlap with sessions BGSAVE
save 1020 1
# 7 minute save periods reduce overlap with sessions BGSAVE
save 420 400000
rdbcompression yes
dbfilename dump-6384.rdb
dir /var/lib/redis
maxclients 10000
maxmemory 12GB
maxmemory-policy volatile-lru
maxmemory-samples 3
appendonly no
appendfilename appendonly-6384.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
hz 100
Raw
data etc:mysql:my.cnf
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr/local/mysql
# symlinks to SSD RAID 1+0 (ext4 mounted with noatime,data=writeback,noexec)
datadir = /usr/local/mysql/data
tmpdir = /usr/local/mysql/tmp
skip-external-locking
character-set-server = utf8
collation-server = utf8_general_ci
event_scheduler = 0
bind-address = 10.1.0.10
key_buffer = 512M
max_allowed_packet = 64M
thread_stack = 512K
thread_cache_size = 512
sort_buffer_size = 24M
read_buffer_size = 8M
read_rnd_buffer_size = 24M
join_buffer_size = 128M
# for some nightly processes client sessions set the join buffer to 8 GB
auto-increment-increment = 1
auto-increment-offset = 1
myisam-recover = BACKUP
max_connections = 1024
# max connect errors artificially high to support behaviors of NetScaler monitors
max_connect_errors = 999999
concurrent_insert = 2
connect_timeout = 5
wait_timeout = 180
net_read_timeout = 120
net_write_timeout = 120
back_log = 128
# this table_open_cache might be too low because of MySQL bugs #16244691 and #65384)
table_open_cache = 12288
tmp_table_size = 512M
max_heap_table_size = 512M
bulk_insert_buffer_size = 512M
open-files-limit = 8192
open-files = 1024
query_cache_type=1
# large query limit supports SOAP and REST API integrations
query_cache_limit = 4M
# larger than 512 MB query cache size is problematic; this is typically ~60% full
query_cache_size = 512M
# set to true on read slaves
read_only = false
slow_query_log_file = /var/log/mysql/slow.log
slow_query_log = 0
long_query_time = 0.2
expire_logs_days = 10
max_binlog_size = 1024M
binlog_cache_size = 32K
sync_binlog = 0
log-bin=mysql-bin
server-id=1
innodb_data_home_dir = /usr/local/mysql/data
innodb_log_group_home_dir = /usr/local/mysql/data
# SSD RAID10 technically has a write capacity of 10000 IOPS
innodb_io_capacity = 400
innodb_file_per_table
innodb_table_locks = true
innodb_lock_wait_timeout = 30
# These servers have 80 CPU threads; match 1:1
innodb_thread_concurrency = 80
innodb_commit_concurrency = 2
innodb_support_xa = true
innodb_buffer_pool_size = 32G
innodb_log_file_size = 512M
innodb_additional_mem_pool_size = 8M
innodb_data_file_path = ibdata1:1024M:autoextend
innodb_file_per_table
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 2G
skip-federated
[mysqldump]
quick
quote-names
single-transaction
max_allowed_packet = 64M
Raw
fastcgi_params
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param REMOTE_USER $remote_user;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param HTTPS $https;
# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;
# Additional params
fastcgi_param REDIRECT_QUERY_STRING $query_string;
fastcgi_param REDIRECT_URL $request_uri;
Raw
Magento Catalog Tests.md

Magento Catalog Tests

Perl

The Magento catalog tests are written in Perl and run through Ubuntu cron. The Test::DatabaseRow module is used as it provides convenient syntax to check values returned from SELECT statements. Here’s an excerpt of the test file:

#!/usr/bin/perl -w
use strict;
use warnings;
use DBI;
use DBD::mysql;
use POSIX;

# Testing dependencies
use Test::More;
use Test::Builder;
use Test::DatabaseRow;

my $Test = Test::Builder->new;

# turn on autoflush for progress
local $| = 1;

# define configurations here
#
# and then use the configurations…

my $dsn       = "dbi:mysql:$sourceDatabase:$sourceHost:$sourcePort";
our $sourceConn     = DBI->connect($dsn, $sourceUser, $sourcePass) || die $DBI::errstr;
isnt( $sourceConn, undef, "Source database connection is present." );

local $Test::DatabaseRow::dbh = $sourceConn;

# Simple counts
#over 100,000 records 7/23/2013
row_ok(sql => "SELECT COUNT(*) AS count FROM catalog_product_entity",
  tests => { '>' => { count => 80000 } },
  label => 'There are more than 80,000 product entity records.');
$Test->note($sourceConn->selectrow_array('SELECT COUNT(*) AS count FROM catalog_product_entity'));

# The tests continue…
#

# Finalize
done_testing();

$sourceConn->disconnect;

Tests in English

SQL has been written to validate these criteria:

  1. There are more than 80,000 product entity records.
  2. There are at least 2,200 categories.
  3. There are more than 80,000 catalog inventory records.
  4. There are at least 30 admin users.
  5. There are at least 2,800 CMS blocks.
  6. There are at least 1,200 CMS pages.
  7. There are at least 400 combo products.
  8. There are at least 50,000 product relations.
  9. There are more than 10,000 in stock and enabled configurable products.
  10. There are 669 EAV attributes.
  11. No catalog rules are active.
  12. Fewer than 0.1 percent of catalog inventory records are missing product records.
  13. Fewer than 1 percent of catalog product records are missing inventory records.
  14. The maximum category nesting level is 6.
  15. There is only one level 0 category.
  16. There are only two level 1 categories.
  17. The largest category has between 2,000 and 3,000 products.
  18. Less than 0.1 percent of categories have non-existent parent categories.
  19. Less than 0.1 percent of categories have non-existent parents.
  20. No simple products are missing SKUs.
  21. One configurable product is missing SKU.
  22. No products have negative inventory.
  23. No parents of EAV entities are missing.
  24. No simple products have prices of zero.
  25. All web stores have Ship to Store methods.
Raw
web — app:etc:local.xml
<?xml version="1.0"?>
<!--
/**
* Magento
*
* NOTICE OF LICENSE
*
* This source file is subject to the Academic Free License (AFL 3.0)
* that is bundled with this package in the file LICENSE_AFL.txt.
* It is also available through the world-wide-web at this URL:
* http://opensource.org/licenses/afl-3.0.php
* If you did not receive a copy of the license and are unable to
* obtain it through the world-wide-web, please send an email
* to [email protected] so we can send you a copy immediately.
*
* DISCLAIMER
*
* Do not edit or add to this file if you wish to upgrade Magento to newer
* versions in the future. If you wish to customize Magento for your
* needs please refer to http://www.magentocommerce.com for more information.
*
* @category Mage
* @package Mage_Core
* @copyright Copyright (c) 2008 Irubin Consulting Inc. DBA Varien (http://www.varien.com)
* @license http://opensource.org/licenses/afl-3.0.php Academic Free License (AFL 3.0)
*/
-->
<config>
<global>
<install>
<date><![CDATA[Thu, 14 Mar 2013 22:45:22 +0000]]></date>
</install>
<crypt>
<key><![CDATA[redacted]]></key>
</crypt>
<disable_local_modules>false</disable_local_modules>
<resources>
<db>
<table_prefix><![CDATA[]]></table_prefix>
</db>
<default_setup>
<connection>
<host><![CDATA[10.1.0.10]]></host>
<username><![CDATA[redacted]]></username>
<password><![CDATA[redacted]]></password>
<dbname><![CDATA[redacted]]></dbname>
<initStatements><![CDATA[SET NAMES utf8]]></initStatements>
<model><![CDATA[mysql4]]></model>
<type><![CDATA[pdo_mysql]]></type>
<pdoType><![CDATA[]]></pdoType>
<active>1</active>
</connection>
</default_setup>
<default_read>
<connection>
<use/>
<host><![CDATA[10.1.0.113]]></host>
<username><![CDATA[redacted]]></username>
<password><![CDATA[redacted]]></password>
<dbname><![CDATA[redacted]]></dbname>
<type>pdo_mysql</type>
<model>mysql4</model>
<initStatements><![CDATA[SET NAMES utf8]]></initStatements>
<active>1</active>
</connection>
</default_read>
</resources>
<session_save><![CDATA[db]]></session_save>
<session_cache_limiter><![CDATA[no-store, no-cache, must-revalidate, post-check=0, pre-check=0]]></session_cache_limiter>
<redis_session>
<host><![CDATA[10.1.0.18]]></host> <!-- Specify an absolute path if using a unix socket -->
<port><![CDATA[6382]]></port>
<password></password> <!-- Specify if your redis server requires authentication -->
<timeout><![CDATA[30]]></timeout> <!-- This is the Redis connection timeout, not the locking timeout -->
<persistent><![CDATA[]]></persistent> <!-- Specify unique string to enable persistent connections. E.g.: sess-db0 -->
<db><![CDATA[0]]></db>
<compression_threshold>0</compression_threshold><!-- 0 to disable -->
<compression_lib><![CDATA[lzf]]></compression_lib>
<log_broken_locks><![CDATA[1]]></log_broken_locks> <!-- Useful for debugging fatal errors or extremely slow pages -->
<max_concurrency><![CDATA[48]]></max_concurrency>
<break_after_frontend><![CDATA[6]]></break_after_frontend> <!-- Frontend arguably doesn't need locking as urgently -->
<break_after_adminhtml><![CDATA[15]]></break_after_adminhtml>
<bot_lifetime>300</bot_lifetime> <!-- Bots get shorter session lifetimes. 0 to disable -->
</redis_session>
<cache>
<backend><![CDATA[Cm_Cache_Backend_Redis]]></backend>
<backend_options>
<server><![CDATA[10.1.0.18]]></server>
<port><![CDATA[6383]]></port>
<persistent><![CDATA[]]></persistent>
<database><![CDATA[0]]></database>
<password></password>
<force_standalone><![CDATA[0]]></force_standalone><!-- 0 for phpredis, 1 for standalone PHP -->
<connect_retries><![CDATA[2]]></connect_retries>
<read_timeout><![CDATA[30]]></read_timeout>
<log_level><![CDATA[4]]></log_level>
<automatic_cleaning_factor>0</automatic_cleaning_factor>
<compress_data><![CDATA[1]]></compress_data> <!-- 0-9 for compression level, recommended: 0 or 1 -->
<compress_tags><![CDATA[1]]></compress_tags> <!-- 0-9 for compression level, recommended: 0 or 1 -->
<compression_threshold>2048</compression_threshold><!-- Strings below this size will not be compressed -->
<compression_lib><![CDATA[lzf]]></compression_lib>
</backend_options>
</cache>
<full_page_cache>
<backend><![CDATA[Cm_Cache_Backend_Redis]]></backend>
<backend_options>
<server><![CDATA[10.1.0.18]]></server>
<port><![CDATA[6384]]></port>
<persistent><![CDATA[]]></persistent>
<database><![CDATA[0]]></database>
<password></password>
<force_standalone><![CDATA[0]]></force_standalone><!-- 0 for phpredis, 1 for standalone PHP -->
<connect_retries><![CDATA[2]]></connect_retries>
<log_level><![CDATA[4]]></log_level>
<lifetimelimit><![CDATA[86400]]></lifetimelimit> <!-- 24 hours of lifetime for cache record -->
<compress_data><![CDATA[0]]></compress_data> <!-- DISABLE compression for EE FPC since it already uses compression -->
</backend_options>
</full_page_cache>
</global>
<admin>
<routers>
<adminhtml>
<args>
<frontName><![CDATA[admin]]></frontName>
</args>
</adminhtml>
</routers>
</admin>
<frontend>
<events>
<controller_action_predispatch>
<observers><log><type>disabled</type></log></observers>
</controller_action_predispatch>
<controller_action_postdispatch>
<observers><log><type>disabled</type></log></observers>
</controller_action_postdispatch>
<sales_quote_save_after>
<observers><log><type>disabled</type></log></observers>
</sales_quote_save_after>
<checkout_quote_destroy>
<observers><log><type>disabled</type></log></observers>
</checkout_quote_destroy>
</events>
</frontend>
</config>
Raw
web — etc:nginx:nginx.conf
user www-data;
# the web hosts have 8 CPU cores
worker_processes 8;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
# multi_accept on;
}
http {
# Basic Settings
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# decrease side channel leaks
server_tokens off;
root /var/www/www.company-name.com/current;
index index.php;
# add forwarded for to aid debugging
log_format main '$remote_addr - $remote_user $time_local '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for';
# allow odd HTTP headers from Barracuda devices http://stackoverflow.com/questions/8300136/where-does-the-cuda-cliip-http-header-come-from
underscores_in_headers on;
# requires nginx 1.2.6 or later
# https://newrelic.com/docs/features/request-queuing-and-tracking-front-end-time
fastcgi_param HTTP_X_REQUEST_START "t=${msec}000";
proxy_buffering off;
# PHP back end
upstream backend {
# matches path defined in /etc/php5/fpm/pool.d/www.conf
server unix:/tmp/php-cgi.socket;
}
upstream admin-backend {
# Magento admin requests are passed through to admin-specific hosts
# This keeps long/complex requests out of the public-facing resources
# Admin hosts have higher memory_limit values in php.ini
server 10.1.0.104;
}
server {
server_name www.client-name.com production.alternate-domain.com;
access_log /var/log/nginx/www.company-name.com.access.log main;
error_log /var/log/nginx/www.company-name.com.error.log info;
# 504 is a PHP timeout and must be static
# 502 is momentary during a PHP restart and should be treated like maintenance
# other 50x errors are handled by Magento
error_page 502 504 /errors/company-name/504.html;
listen 80;
listen 443 ssl;
# pass all Magento admin requests to the dedicated admin web hosts
location ~* /(index\.php/admin|admin) {
proxy_pass http://admin-backend;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_set_header Accept-Encoding "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# this header prevents Magento from redirecting out of admin
add_header Front-End-Https on;
proxy_redirect off;
}
location ~ ^/(.*)sitemap([0-9]*).xml$ {
root /var/www/www.company-name.com/current/sitemaps;
}
location /sitemaps/sitemap.xml { allow all; }
location /sitemaps/image-sitemap.xml { allow all; }
ssl_certificate /etc/ssl/certs/_.company-name.com.crt;
ssl_certificate_key /etc/ssl/private/_.company-name.com.key;
# PCI specified ciphers
ssl_ciphers RC4:HIGH:!aNULL:!MD5:!kEDH;
# PCI specified protocols
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
# anti BEAST
ssl_prefer_server_ciphers on;
# the hardware load balancers statefully route SSL
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# header from the hardware load balancers
real_ip_header X-Forwarded-For;
# trust this header from anything inside the subnet; possibly risky
set_real_ip_from 10.1.0.0/24;
# the header is a comma-separated list; the left-most IP is the end user
real_ip_recursive on;
# ensure zero calls are written to disk
client_max_body_size 16m;
client_body_buffer_size 2m;
client_header_buffer_size 16k;
large_client_header_buffers 8 8k;
# default ENV vars
fastcgi_param MAGE_RUN_CODE base;
fastcgi_param MAGE_RUN_TYPE website;
fastcgi_read_timeout 90s;
fastcgi_send_timeout 60s;
fastcgi_index index.php;
# ensures Magento works and provides some protection against malicious files
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
include fastcgi_params;
# ensure zero calls are written to disk
fastcgi_buffers 512 16k;
fastcgi_buffer_size 512k;
fastcgi_busy_buffers_size 512k;
# Maintenance HTML page can override Magento
index index.html index.php;
# rewrite API2 calls to api.php (REST only)
rewrite ^/api/rest(.*) /api.php?$1&type=rest;
# Deny protected Magento files
location /app/ { deny all; access_log off; }
location /cert/ { deny all; access_log off; }
location /chef/ { deny all; access_log off; }
location /cron.php { deny all; access_log off; }
location /db_back/ { deny all; access_log off; }
location /downloader/ { deny all; access_log off; }
location /includes/ { deny all; access_log off; }
location /index.php.dev.php { deny all; access_log off; }
location /install.php { deny all; access_log off; }
location /lib/ { deny all; access_log off; }
location /media/downloadable/ { deny all; access_log off; }
location /pagoda/ { deny all; access_log off; }
location /pkginfo/ { deny all; access_log off; }
location /prototype/ { deny all; access_log off; }
location /report/config.xml { deny all; access_log off; }
location /utility/ { deny all; access_log off; }
location /util/ { deny all; access_log off; }
location /var/ { deny all; access_log off; }
# deny access to dotfiles
location ~ /\. {
deny all;
access_log off;
log_not_found off;
}
# Disable PHP execution in var and media
location /var { location ~ \.php$ {return 403;} }
location /media { location ~ \.php$ {return 403;} }
# remove the cache-busting timestamp
location ~* (.+)\.(\d+)\.(js|css|png|jpg|jpeg|gif)$ {
try_files $uri $1.$3;
access_log off;
log_not_found off;
expires 21d;
add_header Cache-Control "public";
}
# do not log static files; regexp should capture alternate cache-busting timestamps
location ~* \.(jpg|jpeg|gif|css|png|js|ico|txt|swf|xml|svg|svgz|mp4|ogg|ogv)(\?[0-9]+)?$ {
access_log off;
log_not_found off;
expires 21d;
add_header Cache-Control "public";
}
# CSS and JS
rewrite ^/minify/([0-9]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1 last;
rewrite ^/skin/m/([0-9]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1 last;
location /lib/minify/ {
allow all;
}
# the javascript compressor
location ^~ /js/index.php {
access_log off;
expires 30d;
fastcgi_pass backend;
}
# use fastcgi for all php files
location ~ \.php$ {
expires off;
# 404 if the file does not exist
try_files $uri =404;
fastcgi_pass backend;
}
# pass everything else over to PHP-FPM
location / {
# 404 if the file does not exist
try_files $uri $uri/ /index.php =404;
fastcgi_pass backend;
}
}
server {
server_name internal.company-name.com internal.alternate-domain.com;
access_log /var/log/nginx/internal.company-name.com.access.log main;
error_log /var/log/nginx/internal.company-name.com.error.log info;
# 504 is a PHP timeout and must be static
# 502 is momentary during a PHP restart and should be treated like maintenance
# other 50x errors are handled by Magento
error_page 502 504 /errors/company-name/504.html;
listen 80;
listen 443 ssl;
location ~* /(index\.php/admin|admin) {
proxy_pass http://admin-backend;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_set_header Accept-Encoding "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
add_header Front-End-Https on;
proxy_redirect off;
}
location ~ ^/(.*)sitemap([0-9]*).xml$ {
root /var/www/www.company-name.com/current/sitemaps;
}
location /sitemaps/sitemap.xml { allow all; }
location /sitemaps/image-sitemap.xml { allow all; }
ssl_certificate /etc/ssl/certs/_.company-name.com.crt;
ssl_certificate_key /etc/ssl/private/_.company-name.com.key;
# PCI specified ciphers
ssl_ciphers RC4:HIGH:!aNULL:!MD5:!kEDH;
# PCI specified protocols
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
# anti BEAST
ssl_prefer_server_ciphers on;
#
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
real_ip_header X-Forwarded-For;
set_real_ip_from 10.1.0.0/24;
real_ip_recursive on;
client_max_body_size 16m;
client_body_buffer_size 2m;
client_header_buffer_size 16k;
large_client_header_buffers 8 8k;
# default ENV vars
fastcgi_param MAGE_RUN_CODE internal;
fastcgi_param MAGE_RUN_TYPE website;
fastcgi_read_timeout 90s;
fastcgi_send_timeout 60s;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
include fastcgi_params;
fastcgi_buffers 512 16k;
fastcgi_buffer_size 512k;
fastcgi_busy_buffers_size 512k;
# Maintenance HTML page can override PHP
index index.html index.php;
# rewrite API2 calls to api.php (REST only)
rewrite ^/api/rest(.*) /api.php?$1&type=rest;
# Deny protected Magento files
location /app/ { deny all; access_log off; }
location /cert/ { deny all; access_log off; }
location /chef/ { deny all; access_log off; }
location /cron.php { deny all; access_log off; }
location /db_back/ { deny all; access_log off; }
location /downloader/ { deny all; access_log off; }
location /includes/ { deny all; access_log off; }
location /index.php.dev.php { deny all; access_log off; }
location /install.php { deny all; access_log off; }
location /lib/ { deny all; access_log off; }
location /media/downloadable/ { deny all; access_log off; }
location /pagoda/ { deny all; access_log off; }
location /pkginfo/ { deny all; access_log off; }
location /prototype/ { deny all; access_log off; }
location /report/config.xml { deny all; access_log off; }
location /utility/ { deny all; access_log off; }
location /util/ { deny all; access_log off; }
location /var/ { deny all; access_log off; }
# deny access to dotfiles
location ~ /\. {
deny all;
access_log off;
log_not_found off;
}
# Disable PHP execution in var and media
location /var { location ~ \.php$ {return 403;} }
location /media { location ~ \.php$ {return 403;} }
location ~* (.+)\.(\d+)\.(js|css|png|jpg|jpeg|gif)$ {
try_files $uri $1.$3;
access_log off;
log_not_found off;
expires 21d;
add_header Cache-Control "public";
}
# do not log static files; regexp should capture cache-busting timestamps
location ~* \.(jpg|jpeg|gif|css|png|js|ico|txt|swf|xml|svg|svgz|mp4|ogg|ogv)(\?[0-9]+)?$ {
access_log off;
log_not_found off;
expires 21d;
add_header Cache-Control "public";
}
# CSS and JS
rewrite ^/minify/([0-9]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1 last;
rewrite ^/skin/m/([0-9]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1 last;
location /lib/minify/ {
allow all;
}
# the javascript compressor
location ^~ /js/index.php {
access_log off;
expires 30d;
fastcgi_pass backend;
}
# use fastcgi for all php files
location ~ \.php$ {
expires off;
# 404 if the file does not exist
try_files $uri =404;
fastcgi_pass backend;
}
# pass everything else over to PHP-FPM
location / {
# 404 if the file does not exist
try_files $uri $uri/ /index.php =404;
fastcgi_pass backend;
}
}
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 48 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
#include /etc/nginx/sites-enabled/*;
}
Raw
web — etc:php5:conf.d:*.ini
/etc/php5/fpm/conf.d/apc.ini
[apc]
apc.stat = "0"
apc.max_file_size = "4M"
apc.localcache = "1"
apc.localcache.size = "512"
apc.shm_segments = "1"
apc.ttl = "3600"
apc.user_ttl = "7200"
apc.gc_ttl = "3600"
apc.cache_by_default = "1"
apc.filters = ""
apc.write_lock = "1"
apc.num_files_hint= "0"
apc.user_entries_hint="0"
apc.shm_size = "256M"
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.include_once_override = "0"
apc.file_update_protection="2"
apc.canonicalize = "1"
apc.report_autofilter="0"
apc.stat_ctime="0"
/etc/php5/fpm/conf.d/curl.ini
/etc/php5/fpm/conf.d/gd.ini
/etc/php5/fpm/conf.d/lzf.ini
/etc/php5/fpm/conf.d/mcrypt.ini
/etc/php5/fpm/conf.d/mysql.ini
/etc/php5/fpm/conf.d/mysqli.ini
/etc/php5/fpm/conf.d/newrelic.ini
/etc/php5/fpm/conf.d/pdo.ini
/etc/php5/fpm/conf.d/pdo_mysql.ini
/etc/php5/fpm/conf.d/redis.ini
/etc/php5/fpm/conf.d/suhosin.ini
[suhosin]
suhosin.executor.include.whitelist="phar"
suhosin.session.cryptua = off
suhosin.get.max_value_length = 2000
suhosin.post.max_vars = 1000
suhosin.request.max_vars = 1000
/etc/php5/fpm/conf.d/xmlrpc.ini
Raw
web — etc:php5:fpm:php-fpm.conf
;;;;;;;;;;;;;;;;;;;;;
; FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;
[global]
pid = /var/run/php5-fpm.pid
error_log = /var/log/php/php5-fpm.error.log
log_level = notice
;emergency_restart_threshold = 0
;emergency_restart_interval = 0
;process_control_timeout = 0
process.max = 128
daemonize = yes
;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;
include=/etc/php5/fpm/pool.d/*.conf
Raw
web — etc:php5:fpm:php.ini
[PHP]
engine = On
short_open_tag = On
asp_tags = Off
precision = 14
y2k_compliance = On
output_buffering = 4096
;output_handler =
zlib.output_compression = Off
;zlib.output_compression_level = -1
;zlib.output_handler =
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
allow_call_time_pass_reference = Off
safe_mode = Off
safe_mode_gid = Off
safe_mode_include_dir =
safe_mode_exec_dir =
safe_mode_allowed_env_vars = PHP_
safe_mode_protected_env_vars = LD_LIBRARY_PATH
;open_basedir =
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,
disable_classes =
; user abort is difficult to detect behind load balancers; for safety, complete full requests
ignore_user_abort = On
; realpath_cache_size is much larger than default to support the large file structure of Magento
realpath_cache_size = 256k
realpath_cache_ttl = 300
zend.enable_gc = On
zend.ze1_compatibility_mode = off
expose_php = Off
max_execution_time = 120
max_input_time = 60
; max input vars is 8000 on admin servers
max_input_vars = 1000
; memory limit is 8 GB on admin servers
memory_limit = 512M
error_reporting = E_ALL & ~E_DEPRECATED
display_errors = Off
display_startup_errors = On
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = Off
error_log = /var/log/php/php5-fpm.error.log
variables_order = "GPCS"
request_order = "GP"
register_globals = Off
register_long_arrays = Off
register_argc_argv = Off
auto_globals_jit = On
; large POST size to support SOAP and REST APIs
post_max_size = 16M
magic_quotes_gpc = Off
magic_quotes_runtime = Off
magic_quotes_sybase = Off
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
enable_dl = Off
cgi.force_redirect = 1
cgi.redirect_status_env = ;
cgi.fix_pathinfo=1
file_uploads = On
upload_max_filesize = 16M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 300
[Pdo]
; Whether to pool ODBC connections. Can be one of "strict", "relaxed" or "off"
; http://php.net/pdo-odbc.connection-pooling
;pdo_odbc.connection_pooling=strict
;pdo_odbc.db2_instance_name
[Pdo_mysql]
; If mysqlnd is used: Number of cache slots for the internal result set cache
; http://php.net/pdo_mysql.cache_size
pdo_mysql.cache_size = 2000
; Default socket name for local MySQL connects. If empty, uses the built-in
; MySQL defaults.
; http://php.net/pdo_mysql.default-socket
pdo_mysql.default_socket=
[MySQLi]
mysqli.max_persistent = -1
;mysqli.allow_local_infile = On
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[bcmath]
bcmath.scale = 0
[Session]
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
# crontab cleans sessions
session.gc_probability = 0
session.gc_divisor = 1000
session.gc_maxlifetime = 3600
session.bug_compat_42 = Off
session.bug_compat_warn = Off
session.referer_check =
session.entropy_length = 0
;session.entropy_file = /dev/urandom
session.cache_limiter = private
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
Raw
web — etc:php5:fpm:pool.d:www.conf
[www]
user = www-data
group = www-data
listen = /tmp/php-cgi.socket
listen.backlog = 4096
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
; only allow localhost to connect
listen.allowed_clients = 127.0.0.1
; a fixed number of child processes
pm = static
; 6:1 ratio of PHP processes to CPU cores found through trial and error
; This is the lowest process count we could deploy without socket resets under crush loads
pm.max_children = 48
; with static process manager, these values don’t matter:
pm.start_servers = 8
pm.min_spare_servers = 8
pm.max_spare_servers = 8
; restart child processes intermittently to address minor memory leaks
pm.max_requests = 40000
;pm.status_path = /status
;ping.path = /ping
;ping.response = pong
;access.log = /var/log/php/$pool.access.log
access.format = %R - %u %t "%m %r%Q%q " %s %f %{mili}d %{megabytes}M %C%%
;slowlog = /var/log/php/$pool.slow.log
request_slowlog_timeout = 0
; 2 minutes chosen to allow for graceful performance degradation and some slow API/admin calls
request_terminate_timeout = 120
;chroot =
;chdir = /
; log PHP warnings and notices to aid in debugging
catch_workers_output = yes
security.limit_extensions = .php .html .phtml
Raw
web — etc:sysctl.conf
# Examples:
# http://www.thern.org/projects/sysctl.conf
# http://serverfault.com/questions/415844/linux-webhost-security-settings-in-etc-sysctl-conf
# Backlog of incoming connections; high values ensure HTTP stability under load
kern.ipc.somaxconn=4096
net.core.somaxconn=4096
net.core.netdev_max_backlog=4096
net.ipv4.tcp_max_syn_backlog=4096
#
# net.inet.tcp.msl defines the Maximum Segment Life -
#
# FROM: http://silverwraith.com/papers/freebsd-ddos.php
# This is the maximum amount of time to wait for an ACK in reply
# to a SYN-ACK or FIN-ACK, in milliseconds. If an ACK is not received
# in this time, the segment can be considered "lost" and the network
# connection is freed.
net.inet.tcp.msl=7500
# Decrease TCP keepalive time
# Default: 7200
net.ipv4.tcp_keepalive_time=1800
# max size of shared memory segment
# possibly kernel.shmmax
kern.ipc.shmmax=536870912
kern.ipc.shmall=32768
# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_syncookies = 1
# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
# No source routed packets here
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Make sure no one can alter the routing tables
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
# Don't act as a router
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Don't Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
# This will increase the amount of memory available for socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824
# Limit of open files
fs.file-max = 131072
Raw
web — packages
curl
lzf
mcelog
modman
php-apc
php5-cli
php5-curl
php5-dev
php5-fpm
php5-gd
php5-mcrypt
php5-mysql
php5-suhosin
php5-xmlrpc
phpredis
ssmtp
unzip
@rsidhaarth
Copy link

Hi,

I am looking for Dev like you to optimize our server and website.

I am impressed by the speed of wellghosh.com, not sure whether it is using community or enterprise. We are having a community store which need serious optimization.

We have 11 dedicated servers and did a setup, not sure what we miss.

@rsidhaarth
Copy link

This is our infra, can you please email me to [email protected] , so we can work together.

https://prnt.sc/tj89jn

@parhamr
Copy link
Author

parhamr commented Jul 17, 2020

@rsidhaarth I do not work freelance, sorry. I have no referrals to offer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment