This is coming along very nicely indeed! One suggestion I have is that as more metrics are added it becomes important for an "at a glance" overview of changes so we can monitor for performance improvements and regressions among many workloads. One way to do this would be a matrix of each metric with its change compared to recent samples. e.g. you could do a student's T comparison of today's numbers with those from yesterday, or from a week ago, and colour-code those that show a significant deviation from "no change". This might be a bit noisy on short timescales, so you could aggregrate data into larger bins and compare e.g. moving 1-week aggregates. Fluctuations on short timescales won't stand out, but if there is a real change then it will show up less than a week later. These significant events could also be graphed themselves and/or a history log maintained (or automatically annotated on the individual graphs) so historical changes can also be pinpointed. At some point the ability to annotate the data will become important (e.g. "We understand the cause of this, it was r1.123 of foo.c, which was corrected in r1.124. The developer responsible has been shot.") Kris P.S. If I understand correctly, the float test shows a regression? The metric is calculations/second, so higher = better? _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
Michelle, just found this in the mail. However, it looks like the VPN to work is down. I can't get to the machine right now. I'll mail on Monday. P. ________________________________ From: Michelle Li <michelle_li_001@yahoo.com> To: freebsd-performance@freebsd.org Sent: Saturday, December 20, 2008 7:25:02 PM Subject: Re: ZFS, NFS and Network tuning (Paul Patterson) ...and the dmesg? please post freebsd-performance-request@freebsd.org wrote: Send freebsd-performance mailing list submissions to freebsd-performance@freebsd.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.freebsd.org/mailman/listinfo/freebsd-performance or, via email, send a message with subject or body 'help' to freebsd-performance-request@freebsd.org You can reach the person managing the list at freebsd-performance-owner@freebsd.org When replying, please edit your Subject line so it is more specific than "Re: Contents of freebsd-performance digest..." Today's Topics: 1. Re: ZFS, NFS and Network tuning (Paul Patterson) 2. Re: ZFS, NFS and Network tuning (Paul Patterson) 3. Re: ZFS, NFS and Network tuning (Paul Patterson) 4. intel i7 and Hyperthreading (Mike Tancsa) ---------------------------------------------------------------------- Message: 1 Date: Fri, 19 Dec 2008 06:47:59 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <15723.22980.qm@web110511.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.infli...
And now for some publicity :) I am a big fan of Orca (www.orcaware.com) which I have used regularly with Solaris *and* FreeBSD. I wrote a performance data collector for FreeBSD, actually. The advantage of Orca is that it gets simple text files with the performance data and then it generates graphs and web pages periodically, keeping the data in RRD databases. I think Orca would be really useful for this project. And of course, if anyone wants to give "Devilator" a try, it currently graphs CPU usage in system, interrupt, user time, disk I/O bandwidth, memory usage, system and SWI CPU usage, process sleeping reasons... So far it's been really helpful for me in order to identify bottlenecks. Regards, Borja. _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
I have a suggestion to make the graphs more readable: if a long period=20 was chosen by the user (e.g. > 100 days / plot points), don't plot=20 points and error bars, plot a simple line through the points. Also, set=20 all date strings on the X-axis to empty strings except for the dates on=20 1/10ths of the interval. Did you remove WITNESS,INVARIANTS and malloc debugging for the benchmarks= ?
The kernel configuration file has: include GENERIC PERFMON nomakeoptions DEBUG PERFMON nooptions INVARIANTS PERFMON nooptions GDB PERFMON nooptions DDB PERFMON nooptions KDB PERFMON nooptions WITNESS PERFMON nooptions WITNESS_SKIPSPIN PERFMON nooptions INVARIANT_SUPPORT I also ship the images with a GENERIC kernel in case debugging is needed. I haven't touched malloc.conf but realize that I should. What's the official recommendation on malloc settings? Erik _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
You'd have to patch /usr/src/lib/libc/stdlib/malloc.c and define MALLOC_PRODUCTION. Yes, it's not elegant.
Actually the error bars are quite important to see what is going on. Some of the metrics are very (too) noisy and if you only look at the data points they sometimes appear to have a signal when they don't. Ultimately that just means more data points should be taken per run for For benchmarking you should enable MALLOC_PRODUCTION in src/lib/libc/stdlib/malloc.c. Anyway, for now I am not worried about the particular benchmarks you are running, because they are just demonstrators for the framework and we can easily go back and add more in later on. Kris _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
Of course they are useful, but do we really need to see them when there are hundreds of samples plotted 1 pixel apart? (As I've said: keep them when when the user "zooms in" on an appropriate small number of samples). But this is an aesthetics-oriented idea, functionality won't be impaired either way so feel free to ignore it.
I think so, yeah. Otherwise I'd waste time zooming in only to find the trends are measurement noise. Kris _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
I agree that there's a need for an overview and some sort of notification. I've been collecting historical data to get a baseline for There's a field in the database for this sort of thing. I just think it The documentation on Unixbench is scarce, but I would think so. BTW if anyone's interested my SVN repo is online at: svn://littlebit.dk/website/trunk (Pylons project) svn://littlebit.dk/tracker/trunk (sh/Python scripts for runnning the server and slaves) Be careful with your eyes - this is my first attempt at both shell scripting and Python :-) Erik _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
At Wed, 23 Jan 2008 14:49:14 +0100, BTW This is excellent work and, if I get the time, I'll be using some of this at my day job. Thanks, George _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
Sounds great -- it would be nice to be able to have a few annotations such as "RELENG_7 branchpoint", "7.0 release", that could then appear as vertical lines in the graphs, and likewise things like "netisr made default", "libthr becomes default". Finally, in the interests of making your life more complicated, it would be neat to graph performance across a set of FreeBSD branches overlaid or vertically offset so you could monitor, say, MySQL performance on 8-CURRENT, 7-STABLE, and 6-STABLE over time. Robert N M Watson Computer Laboratory University of Cambridge _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
I'd like to do that, but I'm a bit afraid of hitting the webserver too hard. Data is changing constantly, so graphs are created on the fly. I could probably cache some small versions though and put them on an It's supported by the tools, database and website but currently I just have 2 (slow) machines to work with at the university. The server is churning out images for the slave to consume every 2 hours which is an acceptable sample rate if the intent is to monitor CURRENT development and provide performance alerts much like the tinderboxes do now. Adding releases would seriously hurt that rate. My focus has been strictly on comparing CVS versions because trying to cover anything else quickly makes the configuration space explode and the sample rate go down the drain. Lots of interesting comparisons could be made, however, provided sufficient hardware supplies. Comparing branches would on the top of the list. Erik _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
Quoting Erik Cederstrand <erik@cederstrand.dk> (from Wed, 23 Jan 2008 In case you have some space left for more machines, maybe someone is willing to help out by sponsoring some. Just tell us how many machines you can handle (space/power/...) and if you are interested that we put it up on our wantlist. Bye, Alexander. -- http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
I'm sharing an office with 4 others and I'm leaving for Real Work in a month, so I'm not really in a position to expand at the moment. However, I'd like to do so later on, and also get a more permanent and reliable location for the machines. After weeding out the worst bugs the system runs pretty much by itself, but since this is CURRENT there's always the possibility of the slaves needing attention. Erik _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
I don't see machine availability being a problem once we are ready to "take this live". Kris _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
Quoting Kris Kennaway <kris@FreeBSD.org> (from Thu, 24 Jan 2008 Me too, but it's in the development stage and Robert asks for some new features, and I've read the answer in a way that I thought that it would be good to have 2-3 more machines available for development. Erik, would some more machines be helpful for development? Bye, Alexander. -- Lie, n.: A very poor substitute for the truth, but the only one discovered to date. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
As Kris noted elsewhere I'm still just demonstrating the framework, and the hardware I have available for this is sufficient. When the framework goes live, more computing power is probably necessary. Erik _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
Interesting. Some candidate changes from 2007-10-02:
Modified files:
contrib/gcc opts.c
Log:
Do not imply -ftree-vrp with -O2 and above. One must implicitly specify
'-ftree-vrp' if one wants it.
Some bad code generation has been tracked to -ftree-vrp. jdk1{5,6} are
notable examples.
sys/kern sched_ule.c
Log:
- Move the rebalancer back into hardclock to prevent potential softclock
starvation caused by unbalanced interrupt loads.
- Change the rebalancer to work on stathz ticks but retain
randomization.
- Simplify locking in tdq_idled() to use the tdq_lock_pair() rather than
complex sequences of locks to avoid deadlock.
sys/kern sched_ule.c
Log:
- Reassign the thread queue lock to newtd prior to switching. Assigning
after the switch leads to a race where the outgoing thread still owns
the local queue lock while another cpu may switch it in. This race
is only possible on machines where cpu_switch can take significantly
longer on different cpus which in practice means HTT machines with
unfair thread scheduling algorithms.
Kris
_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"OK, so it was this one. The other interesting events seem to be: 2007-10-20: drop in super-smack performance and context switch benchmarks. This is due to the switch from SCHED_4BSD to SCHED_ULE (super-smack is largely a context switch benchmark due to retarded design). There are uncommitted patches that reduce ULE context switch overhead though, so it will be interesting to see how they affect this. 2007-12-30: file read/pipe read/pipe ping-pong/syscall overhead performance increases. This is due to Jeff's lockless struct file changes (syscall overhead is only affected because unixbench uses the dup2() syscall which is not in fact just a measure of syscall overhead but now has reduced non-syscall cost). Kris _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
| Adrian Bunk | If you want me to quit I will quit |
| Bryan Woods | Stardom SATA HSM violation |
| Borislav Petkov | [PATCH] ide-floppy fix |
| Ingo Molnar | Re: [Announce] [patch] Modular Scheduler Core and Completely Fair Scheduler [CFS] |
git: | |
| walt | [VOTE] git versus mercurial |
| Nicolas Pitre | Re: [PATCH] gc: call "prune --expire 2.weeks.ago" |
| cte | linking libgit.a in C++ projects |
| Marco Costalba | [ANNOUNCE] qgit4 aka qgit ported to Windows |
| qw er | OpenBSD sucks |
| GVG GVG | ssh_exchange_identification: Connection closed by remote host |
| cgc | Re: RES: RES: Filtering outgoing connections in pf |
| Karel Kulhavy | lookup option in /etc/resolv.conf ignored |
| David Miller | Re: [PATCH] pkt_sched: Destroy gen estimators under rtnl_lock(). |
| Christopher Snook | RFC: Nagle latency tuning |
| Matt Mackall | [PATCH] Stop scaring users with "treason uncloaked!" |
| John Heffner | Re: A Linux TCP SACK Question |
