8.0-RELEASE-p2 a zpomaleni TCP/IP prenosu z Apache proti 7.3 [long]

[Tomas Hlavacek]

Dobry den!

Narazil jsem na problem s rychlosti prenosu z Apache a i z vsftpd na
FreeBSD 8.0-RELEASE-p2, ktery se na FreeBSD 7.3 nevyskytuje a to pri
stejne konfiguraci Apache/vsftpd, HW i site. Pri prenosu pres nc(1) je
rychlost ovsem stejna (a je bliz te vyssi hodnote) jak na 7.3 i na 8.0.

Problem se vyskytuje pri stahovani souboru na linkach s horsi latenci a
na asymetrickych linkach. Naopak vykon je OK po lokalni siti resp. i v
jinych L3 subnetech, ktere jsou ale routovane wire-speed zarizenim. Oba
pokusne servery, mezi kterymi je rozdil v rychlosti, jsou pripojeny do
stejneho access-switche a ani na jeden nebyl pozorovan packetloss bez
zateze z zadne z destinaci.

Konkretne jde o 1/3 pokles rychlosti pri prenosu v CR (AS29134 ->
AS6830, rtt je ~10ms) a o 2/3 az 4/5 pokles pri stahovani mimo CR
(AS29134 -> AS29655 rtt >25ms a <50ms) v porovnani se 7.3. Absolutni
cisla jsou v ramci CR: 8.0 2.1MB/s vs. 7.3 3.5MB/s a v zahranici to je
2MB/s vs. 12MB/s. Taky jsem pozoroval, ze pripojeni na 8.0 casto
zpomaluje a zrychluje zatimco 7.3 po slow-startu uz rychlost stahovani
drzi. Jako klienty jsem pouzival FreeBSD 7.2 a Linux 2.6.32, programy
wget a curl a to v obou destinacich (CR i zahranici). Ve vsech pripadech
se delo totez.

Oba servery jsem mel bez jine zateze, soubory ctu z UFS (ovsem zkousel
jsem i tmpfs a zadna zmena), pripojeni obou stroju je uplne stejne.
Nastaveni apache 2.2.14_5 je defaultni, jak to nainstaloval port
www/apache22. Pouzita sitovka je v obou pripadech:

igb0 at pci0:1:0:0:	class=0x020000 card=0x010015d9 chip=0x10c98086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    class      = network
    subclass   = ethernet


Nastaveni:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>


Vystup sysctl -a | grep net je na 8.0 nasledujici:

kern.random.sys.harvest.ethernet: 1
net.local.stream.recvspace: 8192
net.local.stream.sendspace: 8192
net.local.dgram.recvspace: 4096
net.local.dgram.maxdgram: 2048
net.local.taskcount: 0
net.local.recycled: 0
net.local.inflight: 0
net.inet.ip.portrange.randomtime: 45
net.inet.ip.portrange.randomcps: 10
net.inet.ip.portrange.randomized: 1
net.inet.ip.portrange.reservedlow: 0
net.inet.ip.portrange.reservedhigh: 1023
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.first: 10000
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.forwarding: 0
net.inet.ip.redirect: 1
net.inet.ip.ttl: 64
net.inet.ip.rtexpire: 3600
net.inet.ip.rtminexpire: 10
net.inet.ip.rtmaxcache: 128
net.inet.ip.sourceroute: 0
net.inet.ip.intr_queue_maxlen: 256
net.inet.ip.intr_queue_drops: 0
net.inet.ip.accept_sourceroute: 0
net.inet.ip.keepfaith: 0
net.inet.ip.gifttl: 30
net.inet.ip.same_prefix_carp_only: 0
net.inet.ip.subnets_are_local: 0
net.inet.ip.random_id_total: 0
net.inet.ip.random_id_collisions: 0
net.inet.ip.random_id_period: 8192
net.inet.ip.mcast.loop: 1
net.inet.ip.mcast.maxsocksrc: 128
net.inet.ip.mcast.maxgrpsrc: 512
net.inet.ip.fastforwarding: 0
net.inet.ip.maxfragpackets: 800
net.inet.ip.output_flowtable_size: 2048
net.inet.ip.maxfragsperpacket: 16
net.inet.ip.fragpackets: 0
net.inet.ip.check_interface: 0
net.inet.ip.random_id: 0
net.inet.ip.sendsourcequench: 0
net.inet.ip.process_options: 1
net.inet.icmp.maskrepl: 0
net.inet.icmp.icmplim: 200
net.inet.icmp.bmcastecho: 0
net.inet.icmp.quotelen: 8
net.inet.icmp.reply_from_interface: 0
net.inet.icmp.reply_src: 
net.inet.icmp.icmplim_output: 1
net.inet.icmp.log_redirect: 0
net.inet.icmp.drop_redirect: 0
net.inet.icmp.maskfake: 0
net.inet.igmp.gsrdelay: 10
net.inet.igmp.default_version: 3
net.inet.igmp.legacysupp: 0
net.inet.igmp.v2enable: 1
net.inet.igmp.v1enable: 1
net.inet.igmp.sendlocal: 1
net.inet.igmp.sendra: 1
net.inet.igmp.recvifkludge: 1
net.inet.tcp.rfc1323: 1
net.inet.tcp.mssdflt: 512
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 32768
net.inet.tcp.recvspace: 65536
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.v6mssdflt: 1024
net.inet.tcp.hostcache.purge: 0
net.inet.tcp.hostcache.prune: 300
net.inet.tcp.hostcache.expire: 3600
net.inet.tcp.hostcache.count: 3
net.inet.tcp.hostcache.bucketlimit: 30
net.inet.tcp.hostcache.hashsize: 512
net.inet.tcp.hostcache.cachelimit: 15360
net.inet.tcp.read_locking: 1
net.inet.tcp.recvbuf_max: 262144
net.inet.tcp.recvbuf_inc: 16384
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.insecure_rst: 0
net.inet.tcp.ecn.maxretries: 1
net.inet.tcp.ecn.enable: 0
net.inet.tcp.abc_l_var: 2
net.inet.tcp.rfc3465: 1
net.inet.tcp.rfc3390: 1
net.inet.tcp.rfc3042: 1
net.inet.tcp.drop_synfin: 0
net.inet.tcp.delayed_ack: 1
net.inet.tcp.blackhole: 0
net.inet.tcp.log_in_vain: 0
net.inet.tcp.sendbuf_max: 262144
net.inet.tcp.sendbuf_inc: 8192
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.tso: 1
net.inet.tcp.newreno: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.reass.overflows: 0
net.inet.tcp.reass.maxqlen: 48
net.inet.tcp.reass.cursegments: 0
net.inet.tcp.reass.maxsegments: 1600
net.inet.tcp.sack.globalholes: 0
net.inet.tcp.sack.globalmaxholes: 65536
net.inet.tcp.sack.maxholes: 128
net.inet.tcp.sack.enable: 1
net.inet.tcp.inflight.stab: 20
net.inet.tcp.inflight.max: 1073725440
net.inet.tcp.inflight.min: 6144
net.inet.tcp.inflight.rttthresh: 10
net.inet.tcp.inflight.debug: 0
net.inet.tcp.inflight.enable: 1
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.pcbcount: 6
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.log_debug: 0
net.inet.tcp.minmss: 216
net.inet.tcp.syncache.rst_on_sock_fail: 1
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.count: 0
net.inet.tcp.syncache.cachelimit: 15360
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncookies_only: 0
net.inet.tcp.syncookies: 1
net.inet.tcp.timer_race: 0
net.inet.tcp.finwait2_timeout: 60000
net.inet.tcp.fast_finwait2_recycle: 0
net.inet.tcp.always_keepalive: 1
net.inet.tcp.rexmit_slop: 200
net.inet.tcp.rexmit_min: 30
net.inet.tcp.msl: 30000
net.inet.tcp.nolocaltimewait: 0
net.inet.tcp.maxtcptw: 5120
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 42080
net.inet.udp.blackhole: 0
net.inet.udp.log_in_vain: 0
net.inet.sctp.vtag_time_wait: 60
net.inet.sctp.nat_friendly_init: 0
net.inet.sctp.enable_sack_immediately: 0
net.inet.sctp.udp_tunneling_port: 0
net.inet.sctp.udp_tunneling_for_client_enable: 0
net.inet.sctp.mobility_fasthandoff: 0
net.inet.sctp.mobility_base: 0
net.inet.sctp.default_frag_interleave: 1
net.inet.sctp.default_cc_module: 0
net.inet.sctp.log_level: 0
net.inet.sctp.max_retran_chunk: 30
net.inet.sctp.min_residual: 1452
net.inet.sctp.strict_data_order: 0
net.inet.sctp.abort_at_limit: 0
net.inet.sctp.hb_max_burst: 4
net.inet.sctp.do_sctp_drain: 1
net.inet.sctp.max_chained_mbufs: 5
net.inet.sctp.abc_l_var: 1
net.inet.sctp.nat_friendly: 1
net.inet.sctp.auth_disable: 0
net.inet.sctp.asconf_auth_nochk: 0
net.inet.sctp.early_fast_retran_msec: 250
net.inet.sctp.early_fast_retran: 0
net.inet.sctp.cwnd_maxburst: 1
net.inet.sctp.cmt_pf: 0
net.inet.sctp.cmt_use_dac: 0
net.inet.sctp.nr_sack_on_off: 0
net.inet.sctp.cmt_on_off: 0
net.inet.sctp.outgoing_streams: 10
net.inet.sctp.add_more_on_output: 1452
net.inet.sctp.path_rtx_max: 5
net.inet.sctp.assoc_rtx_max: 10
net.inet.sctp.init_rtx_max: 8
net.inet.sctp.valid_cookie_life: 60000
net.inet.sctp.init_rto_max: 60000
net.inet.sctp.rto_initial: 3000
net.inet.sctp.rto_min: 1000
net.inet.sctp.rto_max: 60000
net.inet.sctp.secret_lifetime: 3600
net.inet.sctp.shutdown_guard_time: 180
net.inet.sctp.pmtu_raise_time: 600
net.inet.sctp.heartbeat_interval: 30000
net.inet.sctp.asoc_resource: 10
net.inet.sctp.sys_resource: 1000
net.inet.sctp.sack_freq: 2
net.inet.sctp.delayed_sack_time: 200
net.inet.sctp.chunkscale: 10
net.inet.sctp.min_split_point: 2904
net.inet.sctp.pcbhashsize: 256
net.inet.sctp.tcbhashsize: 1024
net.inet.sctp.maxchunks: 3200
net.inet.sctp.maxburst: 4
net.inet.sctp.peer_chkoh: 256
net.inet.sctp.strict_init: 1
net.inet.sctp.loopback_nocsum: 1
net.inet.sctp.strict_sacks: 1
net.inet.sctp.ecn_nonce: 0
net.inet.sctp.ecn_enable: 1
net.inet.sctp.auto_asconf: 1
net.inet.sctp.recvspace: 233016
net.inet.sctp.sendspace: 233016
net.inet.raw.recvspace: 9216
net.inet.raw.maxdgram: 9216
net.inet.accf.unloadable: 0
net.inet.flowtable.nmbflows: 4096
net.inet.flowtable.tcp_expire: 86400
net.inet.flowtable.fin_wait_expire: 600
net.inet.flowtable.udp_expire: 300
net.inet.flowtable.syn_expire: 300
net.inet.flowtable.collisions: 0
net.inet.flowtable.max_depth: 0
net.inet.flowtable.free_checks: 3979
net.inet.flowtable.frees: 332
net.inet.flowtable.misses: 333
net.inet.flowtable.lookups: 287490
net.inet.flowtable.hits: 287158
net.inet.flowtable.enable: 1
net.inet.flowtable.debug: 0
net.link.generic.system.ifcount: 3
net.link.ether.inet.log_arp_permanent_modify: 1
net.link.ether.inet.log_arp_movements: 1
net.link.ether.inet.log_arp_wrong_iface: 1
net.link.ether.inet.proxyall: 0
net.link.ether.inet.useloopback: 1
net.link.ether.inet.maxtries: 5
net.link.ether.inet.max_age: 1200
net.link.ether.ipfw: 0
net.link.gif.parallel_tunnels: 0
net.link.gif.max_nesting: 1
net.link.log_link_state_change: 1
net.link.tun.devfs_cloning: 1
net.inet6.ip6.forwarding: 0
net.inet6.ip6.redirect: 1
net.inet6.ip6.hlim: 64
net.inet6.ip6.maxfragpackets: 6400
net.inet6.ip6.accept_rtadv: 1
net.inet6.ip6.keepfaith: 0
net.inet6.ip6.log_interval: 5
net.inet6.ip6.hdrnestlimit: 15
net.inet6.ip6.dad_count: 1
net.inet6.ip6.auto_flowlabel: 1
net.inet6.ip6.defmcasthlim: 1
net.inet6.ip6.gifhlim: 30
net.inet6.ip6.kame_version: FreeBSD
net.inet6.ip6.use_deprecated: 1
net.inet6.ip6.rr_prune: 5
net.inet6.ip6.v6only: 1
net.inet6.ip6.rtexpire: 3600
net.inet6.ip6.rtminexpire: 10
net.inet6.ip6.rtmaxcache: 128
net.inet6.ip6.use_tempaddr: 0
net.inet6.ip6.temppltime: 86400
net.inet6.ip6.tempvltime: 604800
net.inet6.ip6.auto_linklocal: 1
net.inet6.ip6.prefer_tempaddr: 0
net.inet6.ip6.use_defaultzone: 0
net.inet6.ip6.maxfrags: 6400
net.inet6.ip6.mcast_pmtu: 0
net.inet6.ip6.mcast.loop: 1
net.inet6.ip6.mcast.maxsocksrc: 128
net.inet6.ip6.mcast.maxgrpsrc: 512
net.inet6.icmp6.rediraccept: 1
net.inet6.icmp6.redirtimeout: 600
net.inet6.icmp6.nd6_prune: 1
net.inet6.icmp6.nd6_delay: 5
net.inet6.icmp6.nd6_umaxtries: 3
net.inet6.icmp6.nd6_mmaxtries: 3
net.inet6.icmp6.nd6_useloopback: 1
net.inet6.icmp6.nodeinfo: 3
net.inet6.icmp6.errppslimit: 100
net.inet6.icmp6.nd6_maxnudhint: 0
net.inet6.icmp6.nd6_debug: 0
net.inet6.icmp6.nd6_maxqueuelen: 1
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0
net.inet6.mld.v1enable: 1
net.inet6.mld.gsrdelay: 10
net.bpf.zerocopy_enable: 0
net.bpf.maxinsns: 512
net.bpf.maxbufsize: 524288
net.bpf.bufsize: 4096
net.isr.numthreads: 1
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 0
net.isr.maxthreads: 1
net.isr.direct: 1
net.isr.direct_force: 1
net.raw.recvspace: 8192
net.raw.sendspace: 8192
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 1
net.route.netisr_maxqlen: 256


Samozrejme jsem googloval vsechny mozne apache tuning guide a zkousel
manipulovat s inflight, ecn, send a receive buffery, newreno a
sack.enable, bez vysledku, resp. vysledky byly +- 0.1MB/s . Zkousel jsem
to v ruznou dobu, koukal tcpdumpem, jestli tam netece neco vedle a pod.

Btw. samozrejme me i napadlo, ze to muze byt sitovkou ci jejim driverem
a tak jsem zkusil neco podobneho i na jinem stroji s touhle sitovkou:

em0 at pci0:13:0:0:	class=0x020000 card=0x108c15d9 chip=0x108c8086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel Corporation 82573E Gigabit Ethernet Controller (Copper) (82573E)'
    class      = network
    subclass   = ethernet

Delo se totez (prestoze to nejde tak dobre srovnavat, nebot ten server
uz byl pripojen jinde a jinak...). Jeste jsem zkousel pripojit se tam z
vic mist a udelat soucet a vysledek je, ze kazde jednotlive spojeni bylo
pomalejsi na 8.0 nez na 7.3 a ve vysledku byl soucet 19MB/s pro 7.3 vs.
8MB/s, ktere zvladlo 8.0. Samozrejme, ze vsechny pokusy jsem opakoval
vickrat.

Rad dodam ev. dalsi informace ci vyzkousim nejake napady. Uvital bych
zejmena ev. informaci, ze/jestli to je znamy problem, nebo napady co za
tim muze byt a navrhy reseni a samozrejme uvitam i nejaky odkaz na cteni
o profilovani a nebo na pribuzne problemy. Samozrejme jsem cetl, ze 8.0
ma prepsany TCP/IP stack, ale vsude se rika, ze v dusledku toho je
rychlejsi a nikde jsem nevidel zminku, ze by to tak nebylo.

Mejte se,
Tomas Hlavacek


-- 
Tomáš Hlaváček <tomas.hlavacek at elfove.cz>

Be strict when sending and tolerant when receiving. (RFC1958)