Tasty » Adam Sampson's reproducibility bookmarks

Rigorous Benchmarking in Reasonable Time - Kent Academic Repository edit / delete

Tomas Kalibera and Richard Jones' paper on how to do benchmarking that's actually meaningful -- presenting results as confidence intervals for effect sizes, with techniques to establish i.i.d. results and work out how many repetitions you need to do. Very nice work for a pretty short paper! (I've spent most of today chasing references from this in the interests of understanding the maths behind it...)

to benchmarking compiler confidence effect-size independence java performance reproducibility statistics vm ... on 26 March 2014

Producing wrong data without doing anything obviously wrong! edit / delete

Lots of examples of how environmental factors (e.g. environment variable size, room temperature, link order, ASLR...) can affect experimental results, to the tune of 20% or more. Basically: why pretty much any benchmark you've seen in a paper where the effect size isn't huge is probably nonsense.

to benchmarking compiler performance reproducibility research statistics ... on 26 March 2014