Adam Sampson's reproducibility bookmarkshttps://bookmarks.offog.org/ats/reproducibilityAdam Sampson2014-03-26T17:25:54ZRigorous Benchmarking in Reasonable Time - Kent Academic Repositoryhttps://bookmarks.offog.org/edit?url=http%3A%2F%2Fkar.kent.ac.uk%2F33611%2F2014-03-26T17:25:54ZTomas Kalibera and Richard Jones' paper on how to do benchmarking that's actually meaningful -- presenting results as confidence intervals for effect sizes, with techniques to establish i.i.d. results and work out how many repetitions you need to do. Very nice work for a pretty short paper! (I've spent most of today chasing references from this in the interests of understanding the maths behind it...)Producing wrong data without doing anything obviously wrong!https://bookmarks.offog.org/edit?url=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D15082752014-03-26T17:22:30ZLots of examples of how environmental factors (e.g. environment variable size, room temperature, link order, ASLR...) can affect experimental results, to the tune of 20% or more. Basically: why pretty much any benchmark you've seen in a paper where the effect size isn't huge is probably nonsense.