An Update: Faulty Catastrophe Models?

January 8th, 2007

Posted by: Roger Pielke, Jr.

Last April we discussed at length the profound significance to hurricane risk estimation of changes made by a leading company, Risk Management Solutions or RMS, to the implementation of catastrophe models used by insurance, reinsurance, among others in the risk management business. A news story from yesterday’s Tampa Tribune provides a perspective that underscores our original analysis.


Last April we wrote:

It does not seem to me that RMS recognizes how profoundly revolutionary this perspective is, or its potential consequences for their own business. What they are say is that the historical climatology of hurricane activity is no longer a valid basis for estimating future risks. This means that the catastrophe models that they provide are untethered from experience. Imagine if you are playing a game of poker, and the dealer tells you that the composition of the deck has been completely changed – now you don’t know whether there are 4 aces in the deck or 20. It would make gambling based on probabilities a pretty dodgy exercise. If RMS is correct, then it has planted the seed that has potential to completely transform its business and the modern insurance and reinsurance industries.

Yesterday’s Tampa Tribune has an article on the changes to the RMS model, which includes comments from scientists consulted by RMS who suggest that the changes to the model are scientifically unsupportable. Here is an excerpt from the news story:

The leading computer model used by the insurance industry to justify huge rate increases in coastal areas nationwide relies on faulty science, says an expert credited with helping develop it.

“I think it points to a problem with the way these modeling groups are operating,” said Jim Elsner, a professor of geography at Florida State University.

Elsner was one of four experts on a panel assembled in late 2005 to provide input for the computer model by Risk Management Solutions of Newark, Calif.

He said the results, details of which were brought to his attention by the Tribune, contain assumptions that are “actually unscientific.”

The flaws identified by Elsner and another panelist have nationwide implications. The expert input was used to justify loss estimates that have prompted major insurance companies to request homeowners rate increases of up to 40 percent.

The problem: RMS took a consensus of experts that there will be more storms across the Atlantic, then added its own projections about which U.S. regions would be most affected.

In an interview Saturday, Gov. Charlie Crist called RMS’s actions “apparent misrepresentations” that are stunning and appalling, but in a way, part of a pattern.

“It almost doesn’t shock me because this industry has been taking remarkable advantage of our people,” Crist said. “Big insurance is about to face a new day in Florida.”

The article reveals that the changes made by RMS apparently did not reflect what they were told by a panel of scientists that they convened to provide an informal expert elicitation:

In March, RMS surprised the insurance industry with a dramatic change in the benchmark catastrophe software model it sells access to. Instead of using historical models based on more than 100 years of storm data, RMS announced a “medium-term” five-year model for 2006 through 2010.

The models contain specific data on tens of millions of homes, allowing insurers to estimate risk based on computer simulations of possible storms.

Based on the new model, RMS said hurricane losses would increase by 40 percent over the Gulf Coast and 25 percent to 30 percent in the other regions.

Consumer advocates tried to raise alarms at the time, with little success.

Robert Hunter, a former Texas insurance commissioner now with the Consumer Federation of America, said the primary reason for the change to the five-year model appeared to be pressure from the insurance industry.

Thomas R. Knutson, a research meteorologist with the National Oceanic and Atmospheric Administration in Princeton, N.J., and another RMS expert panelist, said the five-year timeline didn’t come from the experts.

“I think that question was driven more by the needs of the insurance industry as opposed to the science,” he said.

In March, RMS said the five-year model was developed in cooperation with the expert panel that included Elsner and Knutson, and that based on their perspective: “Increases in hurricane frequency should be expected along the entire U.S. coast, but will be highest in the Gulf, Florida, and the Southeast, while lower in the Mid-Atlantic and the Northeast.”

“I didn’t make any such statement of that type,” Knutson said Friday.
Elsner said he warned RMS about flaws in the model. “I said that’s not a good way of doing it,” he recalled, and said RMS exaggerated the basic science “well beyond what we expected.”

Though RMS said in March that the expert panel “agreed unanimously that a forward-looking view of risk should reflect a higher probability of landfalling hurricanes,” Elsner said there was no consensus.

It doesn’t sound like we’ve heard the end of this issue:

Other experts in the catastrophe-modeling business have questions, too.

Long-term historical data are still the most credible, given the sparse data available for projecting the next five years, Karen Clark, chief executive officer of AIR Worldwide, said in a speech in the summer. Her company is an RMS competitor. Clark encouraged insurance companies not to replace the long-term model with the short-term one. Still, AIR has launched its own version of a five-year program for customers.

The details of how RMS arrives at its projections are considered a trade secret.

“We have never been able to get what they call the information out of the black box to review their models,” said Bob Lotane, a spokesman for Florida’s Office of Insurance Regulation. He said a public modeling system the state is working on should provide a way to verify the RMS projections.

Crist said information from RMS might be subpoenaed.

As we concluded last April,

From the perspective of the basic functioning of the insurance and reinsurance industries, the change in approach by RMS is an admission that the future is far more uncertain than has been the norm for this community. Such uncertainty may call into question the very basis of hurricane insurance and reinsurance which lies in an ability to quantify and anticipate risks. If the industry can’t anticipate risks, or simply come to a consensus on how to calculate risks (even if inaccurate), then this removes one of the key characteristics of successful insurance. Debate on this issue has only just begun.

15 Responses to “An Update: Faulty Catastrophe Models?”

    1
  1. Tim Clear Says:

    I wonder how much profit the insurance companies made last year?

  2. 2
  3. JamesG Says:

    From thestreet.com
    “JUPITER, Fla., April 17, 2006 – Last year’s (2005) devastating hurricane season took its toll on the property and casualty industry, walloping insurers with a $4.2 billion underwriting loss in 2005, according to Weiss Ratings, Inc., the nation’s leading independent provider of ratings and analyses of financial services companies, mutual funds, and stocks. The loss erased the record underwriting profit of $6.4 billion reported by the industry in 2004. ”

    From palmbeachpost.com.
    “The property insurance industry in hurricane-battered Florida will make $3 billion in profits in 2006.”

    Of course, even if storms don’t increase in number or ferocity, the number of people moving to Florida is still increasing and the houses being built are still not exactly appropriate for a hurricane zone, are they? I’d say buy/build a hurricane-proof concrete house to get cheaper insurance.

  4. 3
  5. Robert Muir-Wood Says:

    It might be useful to provide some more measured background to this story than is to be found in the Tampa Tribune.

    The idea for holding an expert elicitation on hurricane activities emerged at RMS during the summer of 2005. Expert elicitations are commonplace in the earthquake community, but, this was the first time (we believe) one had been attempted among climatologists. All those invited to the Oct 2005 meeting were told in the invitation that the purpose of the meeting was ‘to predict the activity rate of hurricanes, relevant to impact and loss modeling .. over the next 3-5 years’. Four scientists agreed to attend; Jim Elsner, Mark Saunders, Kerry Emanuel and Tom Knutson. Through the meeting, and in email exchanges in the days thereafter, a consensus was achieved around expected rates of Cat1-5 and Cat3-5 storms in the Atlantic Basin and at US landfall for the period 2006-2010. This consensus does not mean that everyone walks out of the meeting having agreed an identical answer but that everyone’s view has been equally weighted in arriving at an expected activity rate.

    RMS then took these findings and prepared to implement them in the RMS Hurricane Cat model. In the model Atlantic hurricanes are split into five separate populations according to the area of formation and track. The research to determine which track types were expected to show predominant increases was undertaken by Manuel Lonfat and based on his findings the ‘increment of activity’ was distributed among the track types to preserve the overall activity rate budget at landfall. There are alternative perspectives on regionalization (as emphasized by Jim Elsner), but as such a high proportion of intense hurricanes affect Florida, the Gulf and the Southeast, for the same increase in activity rates, modeled loss results in these regions are relatively insensitive to reasonable alternative regionalizations.

    At the end of this process (in March 2006) a press release was issued along with a white paper describing all the work that had been undertaken – both after being checked with the four experts. Ultimately the results of the implementation of the increase in activity rates were the responsibility of RMS and we did not look to get the experts to endorse the outcome around changes in modeled losses. A scientific paper describing the whole procedure is now in process of being published in a peer reviewed journal.

    In October 2006 the expert elicitation was repeated to cover the period 2007-2011. All four original experts were invited and only Jim Elsner declined, citing that he was ‘under contract’ with another modeling organisation. At the second expert elicitation there were seven climatologists, who were presented with results from twenty statistical/climatological forecast models, each being assigned 100c of probability to be assigned among the different models. The results from this exercise (in terms of expected levels of Cat1-5 and Cat 3-5 landfalling activities) were within 1-2% of the mean expected activity rates of the first expert elicitation. Again all the models, their results and the outcome of the elicitation will be published in scientific journals.

    The political response to the ‘insurance crisis’ currently underway in Florida is looking for someone to blame. Cat modelers are simply the messengers relaying news concerning the significance of a period of significantly higher hurricane activity that has persisted in 9 out of the last 12 years and that climatologists, as polled at the most recent expert elicitation, expect to continue for a decade or more longer. There is a need to get journalists and politicians in Florida to focus more attention on the reasons for the increase in hurricane activity and, in particular, the role of climate change.

    Robert Muir-Wood
    RMS

  6. 4
  7. JS Says:

    Indeed, it’s all about the money.

  8. 5
  9. DG Says:

    Mr. Muir-Wood’s response has a glaring error….in explaining that “CAT modelers are the messengers”…he implies that ALL modelers share a similar view of the hurricane risk. This is indeed not the case. Only RMS has replaced their standard model with the elevated-activity 5-year outlook model. Their clients should be wondering why they continue to use a model that forces them to accept dramatic changes to their modeled losses.

  10. 6
  11. Mark Bahner Says:

    “I’d say buy/build a hurricane-proof concrete house to get cheaper insurance.”

    I’d say spend an additional ***$15 to $60*** to have your house built (or even retrofitted?) with Popular Science Grand Prize Award-winning Bostitch “HurriQuake” nails:

    http://www.popsci.com/popsci/flat/bown/2006/product_75.html

    P.S. The author of this post has no connection at all to Bostitch, Popular Science, or anyone who builds homes in Florida. The author, as a former mechanical engineer, merely appreciates a beautiful piece of design to address a truly significant problem.

    P.P.S. And don’t forget to tell your Congressperson and/or President to fund “Mark Bahner’s Spectacular Water-Tube Storm Surge Protection System.” (In development…but progress is very slow, due to lack of funding. ;-) )

    P.P.P.S. The author does have a significant relationship with Mark Bahner (of “Mark Bahner’s Spectacular Water-Tube Storm Surge Protection System” fame).
    :-)

  12. 7
  13. James Annan Says:

    To point out the obvious, if someone thinks the models are faulty, then there is a great opportunity to make a big fat profit on the back of it.

    So who’s going to put their money where their mouth is?

  14. 8
  15. Roger Pielke, Jr. Says:

    James-

    You write, “if someone thinks the models are faulty, then there is a great opportunity to make a big fat profit on the back of it.”

    How would that happen? There is no relationship between model accuracy and profits (either for modelers or re/insurance) as no one actually knows how accurate the models are . . .

  16. 9
  17. Dan Says:

    “How would that happen? There is no relationship between model accuracy and profits”

    Of course.

    But that’s not the premise, Roger.

    The premise is that someone will pay bigbucks to trumpet the fact that the existing models are incorrect, and this new model X is correct (see how wrong the alarmists were? See?).

  18. 10
  19. Mark Bahner Says:

    Hi Roger,

    James Annan wrote, “You write, “if someone thinks the models are faulty, then there is a great opportunity to make a big fat profit on the back of it.”

    You responded, “How would that happen? There is no relationship between model accuracy and profits (either for modelers or re/insurance) as no one actually knows how accurate the models are . . .”

    The short answer is, “Because superior knowledge equals money.”

    That’s exactly why worldwide economic growth will accelerate dramatically as this century progresses, rendering all end-of-the-century AGW concerns totally moot:

    http://markbahner.typepad.com/random_thoughts/2005/11/why_economic_gr.html

    http://markbahner.typepad.com/random_thoughts/2004/10/implications_of.html

    That’s also why, in the movie “Time Cop,” people tried to go back to the Depression to short stocks that would collapse, and buy stocks that would do well:

    http://imdb.com/title/tt0111438/

    I don’t know how re-insurance actually works, but let’s say that the insurance companies and the customers both believe a model that over-predicts damages.

    The insurance company will collect big-time, because their damages will be much less than expected. But if they RE-insure themselves, they will OVER-pay on those payments, because they will honestly expect damages to be higher than they really end up to be.

    So the RE-insurance company can collect big-time, by charging huge rates, but never having to pay out.

    Knowledge that other people don’t possess is almost always a huge advantage in market situations. (For example, how did I know that Jeremy Rivkin was going to whip up mass hysteria about harmless Calgene tomatoes? But if I had known that, I would have shorted Calgene, instead of putting a ridiculous amount of money into what seemed like a really neat idea. ;-) )

    (Or to view things from a more cheerful perspective, when I was working at Babcock and Wilcox, and Consumers Power in Michigan decided to pull the plug–so to speak–on their B&W nuclear plant, and convert it to natural gas, I knew that was going to be a very wise decision. ;-) )

  20. 11
  21. Roger Pielke, Jr. Says:

    Dan, Mark- Thanks for your comments. A few replies-

    1. Dan, there already is a diversity of models out there on the market. Different companies are already criticizing each other. No one knows who is right, so many customers buy access to all the major models.

    2. Mark, there is no market signal for performance. Insurance companies and reinsurance companies do negotiate over their rates, but both benefit from being able to charge large premiums.

    I agree that there is money to be made in this field;-) There is not (yet) and efficient information market, that is for sure.

    Thanks.

  22. 12
  23. James Annan Says:

    Roger,

    If the insurance premiums are too high, another company can step in, undercut them and steal all the business. Assuming they are confident that the premiums really are too high, of course…

    If the premiums are too low then I suppose it’s not so easy. Save up your money to step into the market after they all go bust, perhaps.

  24. 13
  25. Herve Grenier Says:

    Who knows what the right premium is? A 5 year average? Sure would not look too good for Floridians in the path of Charley, Jeanne, Ivan and people in New Orleans, but would be great for people in NY. That’s exactly what these models are about : go beyond a 5 year average. Should companies ultimately carrying the hurricane risk in the US neglect the undisputable evidence that there are simply on average more storms in the Atlantic today than there was in the 1970s and 1980s? Shouldn’t that come into play in pricing the business and computing the probability to get bankrupt?

  26. 14
  27. Roger Pielke, Jr. Says:

    Hi Herve-

    Thanks for your comments.

    I have argued for about 10 years that the model understate the actual risks, in part because they have not (heretofore) been able to deal with variability (whether ENSO or something else) or change.

    So it makes sense to me that risk estimates have increased. But this makes it all the more important that such increases are well justified and defensible, as regulators, customers, and consumer advocate groups are going to be all over the increases (as they are).

    When you have scientists complaining that their views in an elicitation are not represented in the results, as we see here, then it is not a good situation for the modelers to be in.

    Thanks!

  28. 15
  29. JamesG Says:

    Mark B

    I have visions of the entire house flying away, Perfectly held together by those marvellous screws. Give me concrete like the do in the Bahamas or the Dominican Republic (when they can afford it that is). Keeps you cool too! Why can’t the USA see beyond shoddy wooden shacks?

    PS. I’m a mechanical engineer too, also ex-Babcock, also ex-nuclear as fate would have it.