quoted text > It seems that Francesco pointed me in the right direction in a private
> message. Cyrus in OpenBSD can't use mmap since it assumes that changes
> in an mmaped file are inmediately seen by VFS system calls. OpenBSD
> hasn't an unified buffer and page cache so this semantic requirement
> of mmap isn't met (requires and explicit msync()). Another
> implementation based in lseek and xmalloc must be used, which performs
> much worse than the mmap based.
>
> I think this explains memory usage, and the high number of IOPS.
>
> Now, I must switch the IMAP server or the operating system.
>
> Thanks to everyone, specially Francesco.
>
> 2007/10/16, Samuel Moqux <smonux@gmail.com>:
> > Hello everyone,
> >
> > I running into some problems with a Cyrus Imap server. A year ago I
> > asked about sizing the server and it was defined as "overkill" (Dell
> > 1850, Perc 4e/DC, 4 300GB 10krpm disks in Raid 10) for 300 users, but
> > now I'm suffering of really high loads and performance problems. I
> > think it's due I/O contention (more than a thousand t/s in iostat),
> > but don't know why Cyrus is so intensive in I/O that this hardware
> > can't handle it. Memory consumption is also very high.
> >
> > When I started with ~60 users everything was ok, but after surpasing
> > the one hundred, performance problems arised. I added a imap proxy
> > which sited between the server and the webmail(Squirrel) and improved
> > overall performance, but now we have 215 users and the server has
> > severe problems during peak hours, when load can reach values of 12 or
> > more. Users perceive degraded response times.
> >
> > Top usually lota of imap processes waiting for getblk, biowait, lockf,
> > or select. Vmstat also shows always proceses blocked waiting for I/O.
> >
> > I had to tweak default kernel values because server crashed when
> > importing old mailboxes:
> > maxusers 64 # estimated number of users
> > option NKMEMPAGES_MAX=65535
> > option BUFCACHEPERCENT=15
> >
> > Cyrus was compiled from ports, with default options. The server is
> > running Sendmail and a lightly loaded openldap.
> >
> > I don't know where to look at or what to tweak. Any ideas will be welcome.
> >
> > Best regards.
> >
> > Dmesg and other relevant data (non peak hours).
> >
> > load averages: 10.36, 6.97, 5.79
> > 17:59:50
> > 259 processes: 258 idle, 1 on processor
> > CPU states: 3.7% user, 0.0% nice, 20.6% system, 1.9% interrupt, 73.8%
> idle
> > Memory: Real: 1135M/1702M act/tot Free: 312M Swap: 47M/2196M used/tot
> >
> > PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
> > 9503 _cyrus 2 0 2716K 4392K sleep select 0:08 1.12% imapd
> > 16618 _cyrus -5 0 3976K 5728K sleep biowai 0:08 1.03% imapd
> > 14158 _cyrus 2 0 3544K 5568K sleep select 0:01 0.68% imapd
> > 23082 _cyrus -5 0 11M 11M sleep getblk 0:06 0.54% imapd
> > 4397 _cyrus 2 0 1664K 3396K sleep select 0:00 0.54% imapd
> > 831 _cyrus -5 0 14M 16M sleep getblk 0:01 0.34% imapd
> > 30096 _cyrus 2 0 6136K 7464K sleep select 0:03 0.29% imapd
> > 10307 _cyrus -5 0 5572K 5908K sleep getblk 0:00 0.29% imapd
> > 28758 _cyrus -5 0 7664K 9412K sleep biowai 0:02 0.20% imapd
> > 27091 _cyrus -5 0 10M 12M sleep getblk 0:02 0.20% imapd
> > 15191 _cyrus 2 0 1740K 3664K sleep select 0:00 0.20% imapd
> > 17387 _cyrus 2 0 2216K 4056K sleep select 0:02 0.15% imapd
> > 25614 _cyrus 2 0 2056K 3952K sleep select 0:02 0.15% imapd
> >
> > # iostat 5 5
> > tty sd0 sd1 cd0
> > fd0 cpu
> > tin tout KB/t t/s MB/s KB/t t/s MB/s KB/t t/s MB/s KB/t t/s
> > MB/s us ni sy in id
> > 0 8 10.77 18 0.19 14.30 242 3.38 0.00 0 0.00 0.00 0
> > 0.00 18 0 5 1 77
> > 0 54 9.93 34 0.33 15.00 956 14.00 0.00 0 0.00 0.00 0
> > 0.00 3 0 13 0 84
> > 0 18 12.07 80 0.95 14.33 615 8.61 0.00 0 0.00 0.00 0
> > 0.00 7 0 15 1 77
> > 0 18 8.68 9 0.08 14.03 575 7.88 0.00 0 0.00 0.00 0
> > 0.00 1 0 6 1 92
> > 0 18 10.21 30 0.30 13.26 709 9.19 0.00 0 0.00 0.00 0
> > 0.00 1 0 9 2 88
> >
> > # vmstat 5 5
> > procs memory page disks traps cpu
> > r b w avm fre flt re pi po fr sr sd0 sd1 int sys cs us sy
> id
> > 7 2 01206040 321392 1719 0 0 11 0 174 12 430 573 4294967005
> > 446 18 6 77
> > 1 7 01200372 329816 7117 0 0 0 0 0 28 1497 1488 8354 1469 11
> 17 72
> > 0 2 01198584 331560 4741 0 0 0 0 0 15 1770 1390 5770 1202 3
> 14 83
> > 0 2 01189932 341820 2549 0 0 0 0 0 37 778 800 2981 592 2 8
> 90
> > 1 5 01196632 334756 5348 0 0 0 0 0 56 1622 1455 6145 1247 3
> 18 78
> >
> >
> > /etc/sysctl.conf :
> >
> > kern.maxproc=1024
> > kern.maxfiles=8000
> > net.inet.tcp.sendspace=65535
> > net.inet.tcp.recvspace=65535
> >
> > # mount
> > /dev/sd0a on / type ffs (local)
> > /dev/sd0i on /home type ffs (local, nodev, softdep)
> > /dev/sd0d on /tmp type ffs (local, nodev, nosuid)
> > /dev/sd0f on /usr type ffs (local, nodev)
> > /dev/sd0e on /var type ffs (local, nodev, nosuid, softdep)
> > /dev/sd0h on /var/spool type ffs (local, noatime, nodev, nosuid, softdep)
> > /dev/sd1a on /das0 type ffs (local, noatime, nodev, nosuid, softdep)
> >
> > # df -h
> > Filesystem Size Used Avail Capacity Mounted on
> > /dev/sd0a 500M 99.9M 375M 21% /
> > /dev/sd0i 50.7G 42.4G 5.8G 88% /home
> > /dev/sd0d 124M 16.0K 117M 0% /tmp
> > /dev/sd0f 7.9G 3.3G 4.2G 44% /usr
> > /dev/sd0e 2.0G 344M 1.5G 18% /var
> > /dev/sd0h 3.9G 55.7M 3.7G 1% /var/spool
> > /dev/sd1a 550G 78.3G 444G 15% /das0
> >
> > # cat /etc/cyrus.conf
> > START {
> > # do not delete this entry!
> > recover cmd="ctl_cyrusdb -r"
> >
> > # this is only necessary if using idled for IMAP IDLE
> > idled cmd="idled"
> > }
> >
> > # UNIX sockets start with a slash and are put into /var/imap/socket
> > SERVICES {
> > # add or remove based on preferences
> > imap cmd="imapd -U 30" listen="imap" prefork=10
> > # imaps cmd="imapd -s" listen="imaps" prefork=0
> > pop3 cmd="pop3d" listen="pop3" prefork=1
> > # pop3s cmd="pop3d -s" listen="pop3s" prefork=0
> > sieve cmd="timsieved" listen="sieve" prefork=1
> >
> > # these are only necessary if receiving/exporting usenet via NNTP
> > # nntp cmd="nntpd" listen="nntp" prefork=0
> > # nntps cmd="nntpd -s" listen="nntps" prefork=0
> >
> > # at least one LMTP is required for delivery
> > # lmtp cmd="lmtpd" listen="lmtp" prefork=0
> > #lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1
> > maxchild=40
> > lmtpunix cmd="lmtpd -a" listen="/var/imap/socket/lmtp"
> > prefork=5 maxchild=10
> >
> > # this is only necessary if using notifications
> > notify cmd="notifyd" listen="/var/imap/socket/notify"
> > proto="udp" prefork=1
> > }
> >
> > EVENTS {
> > # this is required
> > checkpoint cmd="ctl_cyrusdb -c" period=30
> >
> > # this is only necessary if using duplicate delivery suppression,
> > # Sieve or NNTP
> > delprune cmd="cyr_expire -E 3" at=0400
> >
> > squatter cmd="squatter -s -r user" at=0100
> > # this is only necessary if caching TLS sessions
> > # tlsprune cmd="tls_prune" at=0400
> > }
> >
> >
> >
> > OpenBSD 3.9-stable (GENERIC) #3: Sat Aug 26 23:30:07 CEST 2006
> > ctl@imap.carreras.sa:/usr/src/sys/arch/i386/compile/GENERIC
> > cpu0: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.20 GHz
> > cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
> H,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
> > real mem = 2146807808 (2096492K)
> > avail mem = 1779486720 (1737780K)
> > using 4278 buffers containing 280363008 bytes (273792K) of memory
> > mainbus0 (root)
> > bios0 at mainbus0: AT/286+(00) BIOS, date 01/09/06, BIOS32 rev. 0 @ 0xffe90
> > pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
> > pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb260/304 (17 entries)
> > pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC" rev
> 0x00)
> > pcibios0: PCI bus #11 is the last bus
> > bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x1000 0xcc000/0x1000
> > 0xcd000/0x2200 0xcf800/0x2200 0xec000/0x4000!
> > ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca8/8 spacing 4
> > cpu0 at mainbus0
> > pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> > pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x09
> > ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x09
> > pci1 at ppb0 bus 1
> > ppb1 at pci1 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x06
> > pci2 at ppb1 bus 2
> > ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: irq 7 Dell 16c
> 32b
> > ami0: FW 521X, BIOS vH430, 256MB RAM
> > ami0: 1 channels, 0 FC loops, 1 logical drives
> > scsibus0 at ami0: 40 targets
> > sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
> > sd0: 69880MB, 69880 cyl, 64 head, 32 sec, 512 bytes/sec, 143114240 sec
> total
> > scsibus1 at ami0: 16 targets
> > safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x2 SCSI BP, 1.0> SCSI2
> > 3/processor fixed
> > ppb2 at pci1 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x06
> > pci3 at ppb2 bus 3
> > ppb3 at pci0 dev 4 function 0 "Intel MCH PCIE" rev 0x09
> > pci4 at ppb3 bus 4
> > ppb4 at pci4 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x07
> > pci5 at ppb4 bus 5
> > ami1 at pci5 dev 14 function 0 "Symbios Logic MegaRAID 320-2E" rev
> > 0x07: irq 7 Dell 2 32b
> > ami1: FW 521X, BIOS vH430, 128MB RAM
> > ami1: 2 channels, 0 FC loops, 1 logical drives
> > scsibus2 at ami1: 40 targets
> > sd1 at scsibus2 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
> > sd1: 572160MB, 572160 cyl, 64 head, 32 sec, 512 bytes/sec, 1171783680 sec
> total
> > scsibus3 at ami1: 16 targets
> > ses0 at scsibus3 targ 6 lun 0: <DELL, PV22XS, E.19> SCSI3 3/processor fixed
> > scsibus4 at ami1: 16 targets
> > ppb5 at pci4 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x07
> > pci6 at ppb5 bus 6
> > ppb6 at pci0 dev 5 function 0 "Intel MCH PCIE" rev 0x09
> > pci7 at ppb6 bus 7
> > ppb7 at pci7 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
> > pci8 at ppb7 bus 8
> > em0 at pci8 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
> > irq 11, address 00:13:72:5c:87:e9
> > ppb8 at pci7 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
> > pci9 at ppb8 bus 9
> > em1 at pci9 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
> > irq 3, address 00:13:72:5c:87:ea
> > ppb9 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x09
> > pci10 at ppb9 bus 10
> > uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: irq 11
> > usb0 at uhci0: USB revision 1.0
> > uhub0 at usb0
> > uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub0: 2 ports with 2 removable, self powered
> > uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: irq 10
> > usb1 at uhci1: USB revision 1.0
> > uhub1 at usb1
> > uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub1: 2 ports with 2 removable, self powered
> > uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: irq 7
> > usb2 at uhci2: USB revision 1.0
> > uhub2 at usb2
> > uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub2: 2 ports with 2 removable, self powered
> > ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: irq 5
> > usb3 at ehci0: USB revision 2.0
> > uhub3 at usb3
> > uhub3: Intel EHCI root hub, rev 2.00/1.00, addr 1
> > uhub3: 6 ports with 6 removable, self powered
> > ppb10 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
> > pci11 at ppb10 bus 11
> > vga1 at pci11 dev 13 function 0 "ATI Radeon VE QY" rev 0x00
> > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> > wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> > ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
> > pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
> > DMA, channel 0 configured to compatibility, channel 1 configured to
> > compatibility
> > atapiscsi0 at pciide0 channel 0 drive 0
> > scsibus5 at atapiscsi0: 2 targets
> > cd0 at scsibus5 targ 0 lun 0: <TEAC, CD-ROM CD-224E-N, 3.AB> SCSI0
> > 5/cdrom removable
> > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
> > pciide0: channel 1 disabled (no drives)
> > isa0 at ichpcib0
> > isadma0 at isa0
> > pckbc0 at isa0 port 0x60/5
> > pckbd0 at pckbc0 (kbd slot)
> > pckbc0: using irq 1 for kbd slot
> > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > pcppi0 at isa0 port 0x61
> > midi0 at pcppi0: <PC speaker>
> > spkr0 at pcppi0
> > npx0 at isa0 port 0xf0/16: using exception 16
> > pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> > fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> > biomask ffe5 netmask ffed ttymask ffef
> > pctr: user-level cycle counter enabled
> > uhub4 at uhub3 port 3
> > uhub4: Dell product 0xa001, rev 2.00/0.00, addr 2
> > uhub4: 2 ports with 2 removable, self powered, multiple transaction
> translators
> > dkcsum: sd0 matches BIOS drive 0x80
> > dkcsum: sd1 matches BIOS drive 0x81
> > root on sd0a
> > rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
> > WARNING: / was not properly unmounted
> > arplookup: unable to enter address for 192.168.10.229
> > arplookup: unable to enter address for 192.168.10.243
> > uvm_mapent_alloc: out of static map entries
> > uvm_mapent_alloc: out of static map entries
> > uvm_mapent_alloc: out of static map entries
> > uvm_mapent_alloc: out of static map entries
--
"Give me the power to issue a nation's money, and I care not who writes the
laws" -- Mayer Amschel Rothschild
