Results 1 to 5 of 5

Thread: Does anyone have any quantifiable measures for determining how accurate a model is ?

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1

    Innovyze Employee



    Join Date
    Feb 2013
    Posts
    13

    Does anyone have any quantifiable measures for determining how accurate a model is ?

    Hello All,

    I've been asked about what type of assessments people undertake to show that a sewer model is suitably calibrated.

    Here in Victoria, Australia, we review individual sites (flow, depth and velocity) to determine the variation from the observed data is within a tolerance, but this is done at each individual site.

    Is it possible to aggregate these up to provide one number that shows the fitness of the model as a whole ?

    Has anyone done this type of holistic reporting on a model's accuracy ?

    Interested to hear back about how/whether this is done in other parts of the world.

    Thanks - Ann

  2. #2
    Senior Member
    Join Date
    Feb 2013
    Posts
    125
    What you do sounds pretty much like what we do as well - it would be possible to devise some scoring scheme to aggregate them up, but I'm not sure it would prove anything. You could conceivably use it to compare different versions of the same model to see if your efforts are going in the right direction, but to say that the model of city A is more "accurate" than the model for city B doesn't really get you anywhere! And that's leaving aside the question of whether the word "accuracy" applies at all - fitness, yes, but accuracy?

  3. #3
    Senior Member

    Innovyze Employee



    Join Date
    Feb 2013
    Posts
    146
    Hi Ann,
    I have always like the Nash–Sutcliffe model efficiency coefficient as it can be interpreted similarly to a n R or R^2 square value in regression analysis (good is close to 1). We use a variety of comparison methods in the Calibrator addon on of H2OMap SWMM and InfoSWMM to match the model to the monitored results. Here are the methods we allow the user to select as the goal for the Genetic Algorithm parameter estimation. Most of the criteria are forms of least square difference comparisons which are used to calculate the spread between the monitored and modelled data points. It helps to try to look at both volume and peak comparisons when you try to come up with a multivariate comparison.
    2013-04-23_0912.jpg
    Attached Images Attached Images
    Last edited by Robert Dickinson; April 23, 2013 at 06:13 AM.

  4. #4
    Forum Moderator

    Innovyze Employee



    Join Date
    Feb 2013
    Posts
    18
    The use of the Nash–Sutcliffe model efficiency coefficient was discussed by Jenny Hill & Ally Martin of Richard Allitt Associates at last year's Innovyze European User Conference. Jenny and Ally presented a paper titled Quantifying the Quality: A Numeric Score of Model Confidence on the first day of the conference.
    Andrew Walker
    Client Service Manager | Innovyze | Wallingford, UK
    Web: www.innovyze.com | Twitter: @innovyze

  5. #5
    Forum Moderator

    Innovyze Employee



    Join Date
    Jan 2013
    Posts
    2
    Link to the presentation Andrew mentioned Quantifying the Quality: A Numeric Score of Model Confidence by Jenny Hill and Ally Martin on the Innnovyze Blog.

    http://blog.innovyze.com/2012/09/28/...er-conference/
    Last edited by Anthony Wisdom; April 24, 2013 at 01:40 AM.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •