Does anyone have any quantifiable measures for determining how accurate a model is ?

Hello All,

I've been asked about what type of assessments people undertake to show that a sewer model is suitably calibrated.

Here in Victoria, Australia, we review individual sites (flow, depth and velocity) to determine the variation from the observed data is within a tolerance, but this is done at each individual site.

Is it possible to aggregate these up to provide one number that shows the fitness of the model as a whole ?

Has anyone done this type of holistic reporting on a model's accuracy ?

Interested to hear back about how/whether this is done in other parts of the world.

Thanks - Ann

What you do sounds pretty much like what we do as well - it would be possible to devise some scoring scheme to aggregate them up, but I'm not sure it would prove anything. You could conceivably use it to compare different versions of the same model to see if your efforts are going in the right direction, but to say that the model of city A is more "accurate" than the model for city B doesn't really get you anywhere! And that's leaving aside the question of whether the word "accuracy" applies at all - fitness, yes, but accuracy?

Hi Ann,
I have always like the Nash–Sutcliffe model efficiency coefficient as it can be interpreted similarly to a n R or R^2 square value in regression analysis (good is close to 1). We use a variety of comparison methods in the Calibrator addon on of H2OMap SWMM and InfoSWMM to match the model to the monitored results. Here are the methods we allow the user to select as the goal for the Genetic Algorithm parameter estimation. Most of the criteria are forms of least square difference comparisons which are used to calculate the spread between the monitored and modelled data points. It helps to try to look at both volume and peak comparisons when you try to come up with a multivariate comparison.
Attachment 36

The use of the Nash–Sutcliffe model efficiency coefficient was discussed by Jenny Hill & Ally Martin of Richard Allitt Associates at last year's Innovyze European User Conference. Jenny and Ally presented a paper titled Quantifying the Quality: A Numeric Score of Model Confidence on the first day of the conference.

Link to the presentation Andrew mentioned Quantifying the Quality: A Numeric Score of Model Confidence by Jenny Hill and Ally Martin on the Innnovyze Blog.

http://blog.innovyze.com/2012/09/28/...er-conference/