2015-03-18

针对Ejabberd的操作系统C1000K优化

本文基于Ubuntu Server 14.04进行优化, 不同的Linux/Unix系统有不同的细节.

查看系统当前支持的最大打开文件数:

root@ci:~# cat /proc/sys/fs/nr_open
1048576

查看硬性限制和软性限制

ulimit -Hn
ulimit -Sn

如果该值小于1000K, 请增大如下设置, 否则达不到100W并发连接.

fs.file-max = 1024000
net.ipv4.ip_conntrack_max = 1024000
net.ipv4.netfilter.ip_conntrack_max = 1024000

所有进程打开的文件描述符数不能超过/proc/sys/fs/file-max
单个进程打开的文件描述符数不能超过user limit中nofile的soft limit
nofile的soft limit不能超过其hard limit
nofile的hard limit不能超过/proc/sys/fs/nr_open

查看服务器TCP状态:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

参考资料

1. 构建C1000K的服务器(1) – 基础
2. Linux Increase The Maximum Number Of Open Files / File Descriptors (FD)
3. Linux系统优化加固

fs.file-max=65535000

net.nf_conntrack_max = 1000000

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

net.core.netdev_max_backlog = 3000000

net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216

net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

#net.ipv4.tcp_congestion_control = HTCP

#net.ipv4.tcp_mtu_probing = 1

net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_buckets = 32768

#net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120

解决 nf_conntrack: table full, dropping packet 的几种思路
http://jaseywang.me/2012/08/16/%E8%A7%A3%E5%86%B3-nf_conntrack-table-full-dropping-packet-%E7%9A%84%E5%87%A0%E7%A7%8D%E6%80%9D%E8%B7%AF/

关于Erlang的一些限制
http://youthyblog.com/2014/08/05/erlang%E6%9C%89%E5%85%B3%E6%95%88%E7%8E%87%E7%9A%84%E4%B8%80%E4%BA%9Blimit/

Ejabberd

Optimization