It seems that Francesco pointed me in the right direction in a private
message. Cyrus in OpenBSD can't use mmap since it assumes that changes
in an mmaped file are inmediately seen by VFS system calls. OpenBSD
hasn't an unified buffer and page cache so this semantic requirement
of mmap isn't met (requires and explicit msync()). Another
implementation based in lseek and xmalloc must be used, which performs
much worse than the mmap based.
I think this explains memory usage, and the high number of IOPS.
Now, I must switch the IMAP server or the operating system.
Thanks to everyone, specially Francesco.
2007/10/16, Samuel Moqux <smonux@gmail.com>:
quoted text > Hello everyone,
>
> I running into some problems with a Cyrus Imap server. A year ago I
> asked about sizing the server and it was defined as "overkill" (Dell
> 1850, Perc 4e/DC, 4 300GB 10krpm disks in Raid 10) for 300 users, but
> now I'm suffering of really high loads and performance problems. I
> think it's due I/O contention (more than a thousand t/s in iostat),
> but don't know why Cyrus is so intensive in I/O that this hardware
> can't handle it. Memory consumption is also very high.
>
> When I started with ~60 users everything was ok, but after surpasing
> the one hundred, performance problems arised. I added a imap proxy
> which sited between the server and the webmail(Squirrel) and improved
> overall performance, but now we have 215 users and the server has
> severe problems during peak hours, when load can reach values of 12 or
> more. Users perceive degraded response times.
>
> Top usually lota of imap processes waiting for getblk, biowait, lockf,
> or select. Vmstat also shows always proceses blocked waiting for I/O.
>
> I had to tweak default kernel values because server crashed when
> importing old mailboxes:
> maxusers 64 # estimated number of users
> option NKMEMPAGES_MAX=65535
> option BUFCACHEPERCENT=15
>
> Cyrus was compiled from ports, with default options. The server is
> running Sendmail and a lightly loaded openldap.
>
> I don't know where to look at or what to tweak. Any ideas will be welcome.
>
> Best regards.
>
> Dmesg and other relevant data (non peak hours).
>
> load averages: 10.36, 6.97, 5.79
> 17:59:50
> 259 processes: 258 idle, 1 on processor
> CPU states: 3.7% user, 0.0% nice, 20.6% system, 1.9% interrupt, 73.8%
idle
quoted text > Memory: Real: 1135M/1702M act/tot Free: 312M Swap: 47M/2196M used/tot
>
> PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
> 9503 _cyrus 2 0 2716K 4392K sleep select 0:08 1.12% imapd
> 16618 _cyrus -5 0 3976K 5728K sleep biowai 0:08 1.03% imapd
> 14158 _cyrus 2 0 3544K 5568K sleep select 0:01 0.68% imapd
> 23082 _cyrus -5 0 11M 11M sleep getblk 0:06 0.54% imapd
> 4397 _cyrus 2 0 1664K 3396K sleep select 0:00 0.54% imapd
> 831 _cyrus -5 0 14M 16M sleep getblk 0:01 0.34% imapd
> 30096 _cyrus 2 0 6136K 7464K sleep select 0:03 0.29% imapd
> 10307 _cyrus -5 0 5572K 5908K sleep getblk 0:00 0.29% imapd
> 28758 _cyrus -5 0 7664K 9412K sleep biowai 0:02 0.20% imapd
> 27091 _cyrus -5 0 10M 12M sleep getblk 0:02 0.20% imapd
> 15191 _cyrus 2 0 1740K 3664K sleep select 0:00 0.20% imapd
> 17387 _cyrus 2 0 2216K 4056K sleep select 0:02 0.15% imapd
> 25614 _cyrus 2 0 2056K 3952K sleep select 0:02 0.15% imapd
>
> # iostat 5 5
> tty sd0 sd1 cd0
> fd0 cpu
> tin tout KB/t t/s MB/s KB/t t/s MB/s KB/t t/s MB/s KB/t t/s
> MB/s us ni sy in id
> 0 8 10.77 18 0.19 14.30 242 3.38 0.00 0 0.00 0.00 0
> 0.00 18 0 5 1 77
> 0 54 9.93 34 0.33 15.00 956 14.00 0.00 0 0.00 0.00 0
> 0.00 3 0 13 0 84
> 0 18 12.07 80 0.95 14.33 615 8.61 0.00 0 0.00 0.00 0
> 0.00 7 0 15 1 77
> 0 18 8.68 9 0.08 14.03 575 7.88 0.00 0 0.00 0.00 0
> 0.00 1 0 6 1 92
> 0 18 10.21 30 0.30 13.26 709 9.19 0.00 0 0.00 0.00 0
> 0.00 1 0 9 2 88
>
> # vmstat 5 5
> procs memory page disks traps cpu
> r b w avm fre flt re pi po fr sr sd0 sd1 int sys cs us sy
id
quoted text > 7 2 01206040 321392 1719 0 0 11 0 174 12 430 573 4294967005
> 446 18 6 77
> 1 7 01200372 329816 7117 0 0 0 0 0 28 1497 1488 8354 1469 11
17 72
quoted text > 0 2 01198584 331560 4741 0 0 0 0 0 15 1770 1390 5770 1202 3
14 83
quoted text > 0 2 01189932 341820 2549 0 0 0 0 0 37 778 800 2981 592 2 8
90
quoted text > 1 5 01196632 334756 5348 0 0 0 0 0 56 1622 1455 6145 1247 3
18 78
quoted text >
>
> /etc/sysctl.conf :
>
> kern.maxproc=1024
> kern.maxfiles=8000
> net.inet.tcp.sendspace=65535
> net.inet.tcp.recvspace=65535
>
> # mount
> /dev/sd0a on / type ffs (local)
> /dev/sd0i on /home type ffs (local, nodev, softdep)
> /dev/sd0d on /tmp type ffs (local, nodev, nosuid)
> /dev/sd0f on /usr type ffs (local, nodev)
> /dev/sd0e on /var type ffs (local, nodev, nosuid, softdep)
> /dev/sd0h on /var/spool type ffs (local, noatime, nodev, nosuid, softdep)
> /dev/sd1a on /das0 type ffs (local, noatime, nodev, nosuid, softdep)
>
> # df -h
> Filesystem Size Used Avail Capacity Mounted on
> /dev/sd0a 500M 99.9M 375M 21% /
> /dev/sd0i 50.7G 42.4G 5.8G 88% /home
> /dev/sd0d 124M 16.0K 117M 0% /tmp
> /dev/sd0f 7.9G 3.3G 4.2G 44% /usr
> /dev/sd0e 2.0G 344M 1.5G 18% /var
> /dev/sd0h 3.9G 55.7M 3.7G 1% /var/spool
> /dev/sd1a 550G 78.3G 444G 15% /das0
>
> # cat /etc/cyrus.conf
> START {
> # do not delete this entry!
> recover cmd="ctl_cyrusdb -r"
>
> # this is only necessary if using idled for IMAP IDLE
> idled cmd="idled"
> }
>
> # UNIX sockets start with a slash and are put into /var/imap/socket
> SERVICES {
> # add or remove based on preferences
> imap cmd="imapd -U 30" listen="imap" prefork=10
> # imaps cmd="imapd -s" listen="imaps" prefork=0
> pop3 cmd="pop3d" listen="pop3" prefork=1
> # pop3s cmd="pop3d -s" listen="pop3s" prefork=0
> sieve cmd="timsieved" listen="sieve" prefork=1
>
> # these are only necessary if receiving/exporting usenet via NNTP
> # nntp cmd="nntpd" listen="nntp" prefork=0
> # nntps cmd="nntpd -s" listen="nntps" prefork=0
>
> # at least one LMTP is required for delivery
> # lmtp cmd="lmtpd" listen="lmtp" prefork=0
> #lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1
> maxchild=40
> lmtpunix cmd="lmtpd -a" listen="/var/imap/socket/lmtp"
> prefork=5 maxchild=10
>
> # this is only necessary if using notifications
> notify cmd="notifyd" listen="/var/imap/socket/notify"
> proto="udp" prefork=1
> }
>
> EVENTS {
> # this is required
> checkpoint cmd="ctl_cyrusdb -c" period=30
>
> # this is only necessary if using duplicate delivery suppression,
> # Sieve or NNTP
> delprune cmd="cyr_expire -E 3" at=0400
>
> squatter cmd="squatter -s -r user" at=0100
> # this is only necessary if caching TLS sessions
> # tlsprune cmd="tls_prune" at=0400
> }
>
>
>
> OpenBSD 3.9-stable (GENERIC) #3: Sat Aug 26 23:30:07 CEST 2006
> ctl@imap.carreras.sa:/usr/src/sys/arch/i386/compile/GENERIC
> cpu0: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.20 GHz
> cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
quoted text > real mem = 2146807808 (2096492K)
> avail mem = 1779486720 (1737780K)
> using 4278 buffers containing 280363008 bytes (273792K) of memory
> mainbus0 (root)
> bios0 at mainbus0: AT/286+(00) BIOS, date 01/09/06, BIOS32 rev. 0 @ 0xffe90
> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
> pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb260/304 (17 entries)
> pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC" rev
0x00)
quoted text > pcibios0: PCI bus #11 is the last bus
> bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x1000 0xcc000/0x1000
> 0xcd000/0x2200 0xcf800/0x2200 0xec000/0x4000!
> ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca8/8 spacing 4
> cpu0 at mainbus0
> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x09
> ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x09
> pci1 at ppb0 bus 1
> ppb1 at pci1 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x06
> pci2 at ppb1 bus 2
> ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: irq 7 Dell 16c
32b
quoted text > ami0: FW 521X, BIOS vH430, 256MB RAM
> ami0: 1 channels, 0 FC loops, 1 logical drives
> scsibus0 at ami0: 40 targets
> sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
> sd0: 69880MB, 69880 cyl, 64 head, 32 sec, 512 bytes/sec, 143114240 sec
total
quoted text > scsibus1 at ami0: 16 targets
> safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x2 SCSI BP, 1.0> SCSI2
> 3/processor fixed
> ppb2 at pci1 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x06
> pci3 at ppb2 bus 3
> ppb3 at pci0 dev 4 function 0 "Intel MCH PCIE" rev 0x09
> pci4 at ppb3 bus 4
> ppb4 at pci4 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x07
> pci5 at ppb4 bus 5
> ami1 at pci5 dev 14 function 0 "Symbios Logic MegaRAID 320-2E" rev
> 0x07: irq 7 Dell 2 32b
> ami1: FW 521X, BIOS vH430, 128MB RAM
> ami1: 2 channels, 0 FC loops, 1 logical drives
> scsibus2 at ami1: 40 targets
> sd1 at scsibus2 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
> sd1: 572160MB, 572160 cyl, 64 head, 32 sec, 512 bytes/sec, 1171783680 sec
total
quoted text > scsibus3 at ami1: 16 targets
> ses0 at scsibus3 targ 6 lun 0: <DELL, PV22XS, E.19> SCSI3 3/processor fixed
> scsibus4 at ami1: 16 targets
> ppb5 at pci4 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x07
> pci6 at ppb5 bus 6
> ppb6 at pci0 dev 5 function 0 "Intel MCH PCIE" rev 0x09
> pci7 at ppb6 bus 7
> ppb7 at pci7 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
> pci8 at ppb7 bus 8
> em0 at pci8 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
> irq 11, address 00:13:72:5c:87:e9
> ppb8 at pci7 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
> pci9 at ppb8 bus 9
> em1 at pci9 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
> irq 3, address 00:13:72:5c:87:ea
> ppb9 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x09
> pci10 at ppb9 bus 10
> uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: irq 11
> usb0 at uhci0: USB revision 1.0
> uhub0 at usb0
> uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
> uhub0: 2 ports with 2 removable, self powered
> uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: irq 10
> usb1 at uhci1: USB revision 1.0
> uhub1 at usb1
> uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
> uhub1: 2 ports with 2 removable, self powered
> uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: irq 7
> usb2 at uhci2: USB revision 1.0
> uhub2 at usb2
> uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1
> uhub2: 2 ports with 2 removable, self powered
> ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: irq 5
> usb3 at ehci0: USB revision 2.0
> uhub3 at usb3
> uhub3: Intel EHCI root hub, rev 2.00/1.00, addr 1
> uhub3: 6 ports with 6 removable, self powered
> ppb10 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
> pci11 at ppb10 bus 11
> vga1 at pci11 dev 13 function 0 "ATI Radeon VE QY" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
> pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
> DMA, channel 0 configured to compatibility, channel 1 configured to
> compatibility
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus5 at atapiscsi0: 2 targets
> cd0 at scsibus5 targ 0 lun 0: <TEAC, CD-ROM CD-224E-N, 3.AB> SCSI0
> 5/cdrom removable
> cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
> pciide0: channel 1 disabled (no drives)
> isa0 at ichpcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> npx0 at isa0 port 0xf0/16: using exception 16
> pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> biomask ffe5 netmask ffed ttymask ffef
> pctr: user-level cycle counter enabled
> uhub4 at uhub3 port 3
> uhub4: Dell product 0xa001, rev 2.00/0.00, addr 2
> uhub4: 2 ports with 2 removable, self powered, multiple transaction
translators
quoted text > dkcsum: sd0 matches BIOS drive 0x80
> dkcsum: sd1 matches BIOS drive 0x81
> root on sd0a
> rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
> WARNING: / was not properly unmounted
> arplookup: unable to enter address for 192.168.10.229
> arplookup: unable to enter address for 192.168.10.243
> uvm_mapent_alloc: out of static map entries
> uvm_mapent_alloc: out of static map entries
> uvm_mapent_alloc: out of static map entries
> uvm_mapent_alloc: out of static map entries
