This is coming along very nicely indeed!
One suggestion I have is that as more metrics are added it becomes

important for an "at a glance" overview of changes so we can monitor for

performance improvements and regressions among many workloads.
One way to do this would be a matrix of each metric with its change

compared to recent samples.  e.g. you could do a student's T comparison

of today's numbers with those from yesterday, or from a week ago, and

colour-code those that show a significant deviation from "no change".

This might be a bit noisy on short timescales, so you could aggregrate

data into larger bins and compare e.g. moving 1-week aggregates.

Fluctuations on short timescales won't stand out, but if there is a real

change then it will show up less than a week later.
These significant events could also be graphed themselves and/or a

history log maintained (or automatically annotated on the individual

graphs) so historical changes can also be pinpointed.
At some point the ability to annotate the data will become important

(e.g. "We understand the cause of this, it was r1.123 of foo.c, which

was corrected in r1.124.  The developer responsible has been shot.")
Kris
P.S. If I understand correctly, the float test shows a regression?  The

metric is calculations/second, so higher = better?

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

(Note: I hope this reply shows up correctly. I subscribed to the

mailing list after the fact and had to "forge" the subject line.)
Hey everyone,

I'm having serious trouble with the same thing, and just found this

thread while looking for the correct place to post. Looks like I found

it. (I wrote most of the post before finding the thread, so some of it

will seem a bit odd.)
I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old

80GB 7200rpm disk.
My problem is that I get completely unacceptable latency on console IO

(both via SSH and serial console) when the system is performing disk

IO. The worst case I've noticed yet was when I tried copying a core

dump from a lzjb compressed ZFS file system to a gzip-9 compressed

one, to compare the file size/compression ratio. screen(1) took at

LEAST ten seconds - probably a bit more - I'm not exaggerating here -

to switch to another window, and an "ls" in an empty directory also

about 5-10 seconds.

Doing some silly CPU load with two instances of "yes >/dev/null" (on a

single core, remember) doesn't change anything, the system remains

very responsive. "cat /dev/zero > /uncompressed_fs/..." however

produces the extreme slowdown. (On a gzip-1 FS it doesn't, since the

file ends up extremely small - a kilobyte or so - even after a while,

thus performing minimal IO).
I'm thinking about switching to FreeBSD on my beefier "production"

system (dual-core amd64, 4GB RAM, 4x1TB disks, compared to this one,

single-core, 2GB RAM, 80GB disk), but unless I feel assured this won't

happen there as well, I'm not so sure anymore. I can do any kind of

heavy IO/compilation/whatever on that box, currently running Linux,

and it's always unnoticable. In this case it's impossible *not* to

notice that your key input is lagging behind 5-10 seconds... I thought

multiple times that the box must have panicked.

I do realize that the hardware isn't the best, especially the disks,

but this is far wor...
[ message continues ]

Michelle,
just found this in the mail.  However, it looks like the VPN to work is down.  I can't get to the machine right now.  I'll mail on Monday.
P.
________________________________

From: Michelle Li <michelle_li_001@yahoo.com>

To: freebsd-performance@freebsd.org

Sent: Saturday, December 20, 2008 7:25:02 PM

Subject: Re: ZFS, NFS and Network tuning (Paul Patterson) 
...and the dmesg?
please post
freebsd-performance-request@freebsd.org wrote: Send freebsd-performance mailing list submissions to

freebsd-performance@freebsd.org
To subscribe or unsubscribe via the World Wide Web, visit

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

or, via email, send a message with subject or body 'help' to

freebsd-performance-request@freebsd.org
You can reach the person managing the list at

freebsd-performance-owner@freebsd.org
When replying, please edit your Subject line so it is more specific

than "Re: Contents of freebsd-performance digest..."
Today's Topics:
   1. Re: ZFS, NFS and Network tuning (Paul Patterson)

   2. Re: ZFS, NFS and Network tuning (Paul Patterson)

   3. Re: ZFS, NFS and Network tuning (Paul Patterson)

   4. intel i7 and Hyperthreading (Mike Tancsa)
----------------------------------------------------------------------
Message: 1

Date: Fri, 19 Dec 2008 06:47:59 -0800 (PST)

From: Paul Patterson 
Subject: Re: ZFS, NFS and Network tuning

To: Paul Patterson

,

freebsd-performance@freebsd.org

Message-ID: <15723.22980.qm@web110511.mail.gq1.yahoo.com>

Content-Type: text/plain; charset=us-ascii
Hi,
as promised, the parameter tuning I have on the box (does anyone see anything wrong?)
/boot/loader.conf
kern.hz="100"

vm.kmem_size_max="1536M"

vm.kmem_size="1536M"

vfs.zfs.prefetch_disble=1
/etc/sysctl.conf
kern.ipc.maxsockbuf=16777216

kern.ipc.nmbclusters=32768

kern.ipc.somaxconn=8192

kern.maxfiles=65536

kern.maxfilesperproc=32768

kern.mxvnodes=600000

net.inet.tcp.delayed_ack=0

net.inet.tcp.infli...
[ message continues ]

You might find the "RELENG_7 heavy disk = system crawls" thread interesting:
} I didn't actually solve it or do anything.

} I just upgraded to RELENG_8.

}

} Now it's behaving more like FreeBSD should.

} I can do sequential reads/writes and still

} use kbd/mouse/X11/buildworld and so on.

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = stable/

8. Been running current/stable-8 since May.
Regards,

Thomas

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Is this really ZFS-bound? I also saw (and still see!) massive

performance impacts when a lot of disk activities, especially when

compiling large packages (done on UFS2 disks). Copying data from one ZFS

drive to (on different harddrive) another ZFS drive (which is

compressed) doesn't impact performance that much.
When the box hit this performance issue, console, X11 and other

activities tend to be stuck for minutes! This happens on an UP box

(Athlon64 3500+, 2 GB ECC RAM, 2x 250 GB WD SATA-II disks and 1 TB WD

SATA-II disk (1x 250 GB UFS2 for system, 1x 250 GB ZFS compressed for

backup, 1x 1 TB ZFS holding /home).
But this impact is also noticable on my lab's machine: Quad core Intel

Q6600 CPU at 3,0 GHZ, 8 GB RAM, 1x 320 GB SATA-II UFS2 disk for system,

2x SATA-II 500 ZFS and 1x 750 GB disks ZFS and even on a 8 core, two

socket Dell PowerEdge 1950-III box with two XEON CPUs running at 2,5

GHz, 16 GB RAM and two SATA-II harddrives attached to the built in SAS

controller.
Maybe this is of interest:

http://www.phoronix.com/scan.php?page=article&item=freebsd8_ubuntu910&num=1
Watch the threaded I/O impact ...
Regards,

Oliver

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

It doesn't appear to be an ZFS issue, no. (Note that I wasn't the one

who added it to the subject line :)

I just tried it to my one UFS filesystem, and while screen(1) DID

remain 100% responsive, everything didn't. vim worked OK most of the

time, but other FS ops on the same disk... oh no. I aborted the "file /

etc/*" after 57 seconds. Tried it again after stopping the disk IO

(cat /dev/zero > /bootdir/filename) - 1.52 seconds.
This raises the question: is this some kind of hardware specific

issue? If so, what hardware? I can't think of anything my computer

would have in common with the ones you've listed!

I mean, come on, FreeBSD's IO performance can't be this abysmal for

everybody or nobody would use it for serious applications. Something

must be wrong for a few of us, but where and why?
Regards,

Thomas

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

And now for some publicity :)
I am a big fan of Orca (www.orcaware.com) which I have used regularly

with Solaris *and*

FreeBSD. I wrote a performance data collector for FreeBSD, actually.
The advantage of Orca is that it gets simple text files with the

performance data

and then it generates graphs and web pages periodically, keeping the

data in RRD

databases. I think Orca would be really useful for this project.
And of course, if anyone wants to give "Devilator" a try, it currently

graphs

CPU usage in system, interrupt, user time, disk I/O bandwidth, memory

usage,

system and SWI CPU usage, process sleeping reasons... So far it's been

really

helpful for me in order to identify bottlenecks.
Regards,
Borja.
_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

I have a suggestion to make the graphs more readable: if a long period=20

was chosen by the user (e.g. > 100 days / plot points), don't plot=20

points and error bars, plot a simple line through the points. Also, set=20

all date strings on the X-axis to empty strings except for the dates on=20

1/10ths of the interval.
Did you remove WITNESS,INVARIANTS and malloc debugging for the benchmarks=

?

The kernel configuration file has:

include GENERIC

PERFMON	nomakeoptions	DEBUG

PERFMON	nooptions	INVARIANTS

PERFMON	nooptions	GDB

PERFMON	nooptions	DDB

PERFMON	nooptions	KDB

PERFMON	nooptions	WITNESS

PERFMON	nooptions	WITNESS_SKIPSPIN

PERFMON	nooptions	INVARIANT_SUPPORT
I also ship the images with a GENERIC kernel in case debugging is needed.
I haven't touched malloc.conf but realize that I should. What's the

official recommendation on malloc settings?
Erik

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

You'd have to patch /usr/src/lib/libc/stdlib/malloc.c and define

MALLOC_PRODUCTION. Yes, it's not elegant.

Actually the error bars are quite important to see what is going on.

Some of the metrics are very (too) noisy and if you only look at the

data points they sometimes appear to have a signal when they don't.

Ultimately that just means more data points should be taken per run for 
For benchmarking you should enable MALLOC_PRODUCTION in

src/lib/libc/stdlib/malloc.c.  Anyway, for now I am not worried about

the particular benchmarks you are running, because they are just

demonstrators for the framework and we can easily go back and add more

in later on.
Kris

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Of course they are useful, but do we really need to see them when there

are hundreds of samples plotted 1 pixel apart? (As I've said: keep them

when when the user "zooms in" on an appropriate small number of

samples). But this is an aesthetics-oriented idea, functionality won't

be impaired either way so feel free to ignore it.

I think so, yeah.  Otherwise I'd waste time zooming in only to find the

trends are measurement noise.
Kris
_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

I agree that there's a need for an overview and some sort of

notification. I've been collecting historical data to get a baseline for 
There's a field in the database for this sort of thing. I just think it 
The documentation on Unixbench is scarce, but I would think so.
BTW if anyone's interested my SVN repo is online at:
svn://littlebit.dk/website/trunk    (Pylons project)

svn://littlebit.dk/tracker/trunk    (sh/Python scripts for runnning the

server and slaves)
Be careful with your eyes - this is my first attempt at both shell

scripting and Python :-)
Erik

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

At Wed, 23 Jan 2008 14:49:14 +0100,
BTW This is excellent work and, if I get the time, I'll be using some

of this at my day job.
Thanks,

George

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Sounds great -- it would be nice to be able to have a few annotations such as

"RELENG_7 branchpoint", "7.0 release", that could then appear as vertical

lines in the graphs, and likewise things like "netisr made default", "libthr

becomes default".
Finally, in the interests of making your life more complicated, it would be

neat to graph performance across a set of FreeBSD branches overlaid or

vertically offset so you could monitor, say, MySQL performance on 8-CURRENT,

7-STABLE, and 6-STABLE over time.
Robert N M Watson

Computer Laboratory

University of Cambridge

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

I'd like to do that, but I'm a bit afraid of hitting the webserver too

hard. Data is changing constantly, so graphs are created on the fly. I

could probably cache some small versions though and put them on an 
It's supported by the tools, database and website but currently I just

have 2 (slow) machines to work with at the university. The server is

churning out images for the slave to consume every 2 hours which is an

acceptable sample rate if the intent is to monitor CURRENT development

and provide performance alerts much like the tinderboxes do now. Adding

releases would seriously hurt that rate.
My focus has been strictly on comparing CVS versions because trying to

cover anything else quickly makes the configuration space explode and

the sample rate go down the drain. Lots of interesting comparisons could

be made, however, provided sufficient hardware supplies. Comparing

branches would on the top of the list.
Erik

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Quoting Erik Cederstrand <erik@cederstrand.dk> (from Wed, 23 Jan 2008  
In case you have some space left for more machines, maybe someone is

willing to help out by sponsoring some. Just tell us how many machines

you can handle (space/power/...) and if you are interested that we put

it up on our wantlist.
Bye,

Alexander.
--

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7

http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

I'm sharing an office with 4 others and I'm leaving for Real Work in a

month, so I'm not really in a position to expand at the moment. However,

I'd like to do so later on, and also get a more permanent and reliable

location for the machines.
After weeding out the worst bugs the system runs pretty much by itself,

but since this is CURRENT there's always the possibility of the slaves

needing attention.
Erik

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

I don't see machine availability being a problem once we are ready to

"take this live".
Kris
_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Quoting Kris Kennaway <kris@FreeBSD.org> (from Thu, 24 Jan 2008  
Me too, but it's in the development stage and Robert asks for some new

features, and I've read the answer in a way that I thought that it

would be good to have 2-3 more machines available for development.
Erik, would some more machines be helpful for development?
Bye,

Alexander.
--

Lie, n.:

	A very poor substitute for the truth, but the only one

	discovered to date.
http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7

http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

As Kris noted elsewhere I'm still just demonstrating the framework, and

the hardware I have available for this is sufficient. When the framework

goes live, more computing power is probably necessary.
Erik

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

Interesting.  Some candidate changes from 2007-10-02:
   Modified files:

     contrib/gcc          opts.c

   Log:

   Do not imply -ftree-vrp with -O2 and above.  One must implicitly specify

   '-ftree-vrp' if one wants it.

   Some bad code generation has been tracked to -ftree-vrp.  jdk1{5,6} are

   notable examples.
     sys/kern             sched_ule.c

   Log:

    - Move the rebalancer back into hardclock to prevent potential softclock

      starvation caused by unbalanced interrupt loads.

    - Change the rebalancer to work on stathz ticks but retain

randomization.

    - Simplify locking in tdq_idled() to use the tdq_lock_pair() rather than

      complex sequences of locks to avoid deadlock.
     sys/kern             sched_ule.c

   Log:

    - Reassign the thread queue lock to newtd prior to switching.  Assigning

      after the switch leads to a race where the outgoing thread still owns

      the local queue lock while another cpu may switch it in.  This race

      is only possible on machines where cpu_switch can take significantly

      longer on different cpus which in practice means HTT machines with

      unfair thread scheduling algorithms.
Kris

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

OK, so it was this one.  The other interesting events seem to be:
2007-10-20: drop in super-smack performance and context switch

benchmarks.  This is due to the switch from SCHED_4BSD to SCHED_ULE

(super-smack is largely a context switch benchmark due to retarded

design).  There are uncommitted patches that reduce ULE context switch

overhead though, so it will be interesting to see how they affect this.
2007-12-30: file read/pipe read/pipe ping-pong/syscall overhead

performance increases.  This is due to Jeff's lockless struct file

changes (syscall overhead is only affected because unixbench uses the

dup2() syscall which is not in fact just a measure of syscall overhead

but now has reduced non-syscall cost).
Kris

_______________________________________________

freebsd-performance@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-performance

To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"