An Update: Faulty Catastrophe Models?January 8th, 2007
Posted by: Roger Pielke, Jr.
Last April we discussed at length the profound significance to hurricane risk estimation of changes made by a leading company, Risk Management Solutions or RMS, to the implementation of catastrophe models used by insurance, reinsurance, among others in the risk management business. A news story from yesterday’s Tampa Tribune provides a perspective that underscores our original analysis.
Last April we wrote:
It does not seem to me that RMS recognizes how profoundly revolutionary this perspective is, or its potential consequences for their own business. What they are say is that the historical climatology of hurricane activity is no longer a valid basis for estimating future risks. This means that the catastrophe models that they provide are untethered from experience. Imagine if you are playing a game of poker, and the dealer tells you that the composition of the deck has been completely changed – now you don’t know whether there are 4 aces in the deck or 20. It would make gambling based on probabilities a pretty dodgy exercise. If RMS is correct, then it has planted the seed that has potential to completely transform its business and the modern insurance and reinsurance industries.
Yesterday’s Tampa Tribune has an article on the changes to the RMS model, which includes comments from scientists consulted by RMS who suggest that the changes to the model are scientifically unsupportable. Here is an excerpt from the news story:
The leading computer model used by the insurance industry to justify huge rate increases in coastal areas nationwide relies on faulty science, says an expert credited with helping develop it.
“I think it points to a problem with the way these modeling groups are operating,” said Jim Elsner, a professor of geography at Florida State University.
Elsner was one of four experts on a panel assembled in late 2005 to provide input for the computer model by Risk Management Solutions of Newark, Calif.
He said the results, details of which were brought to his attention by the Tribune, contain assumptions that are “actually unscientific.”
The flaws identified by Elsner and another panelist have nationwide implications. The expert input was used to justify loss estimates that have prompted major insurance companies to request homeowners rate increases of up to 40 percent.
The problem: RMS took a consensus of experts that there will be more storms across the Atlantic, then added its own projections about which U.S. regions would be most affected.
In an interview Saturday, Gov. Charlie Crist called RMS’s actions “apparent misrepresentations” that are stunning and appalling, but in a way, part of a pattern.
“It almost doesn’t shock me because this industry has been taking remarkable advantage of our people,” Crist said. “Big insurance is about to face a new day in Florida.”
The article reveals that the changes made by RMS apparently did not reflect what they were told by a panel of scientists that they convened to provide an informal expert elicitation:
In March, RMS surprised the insurance industry with a dramatic change in the benchmark catastrophe software model it sells access to. Instead of using historical models based on more than 100 years of storm data, RMS announced a “medium-term” five-year model for 2006 through 2010.
The models contain specific data on tens of millions of homes, allowing insurers to estimate risk based on computer simulations of possible storms.
Based on the new model, RMS said hurricane losses would increase by 40 percent over the Gulf Coast and 25 percent to 30 percent in the other regions.
Consumer advocates tried to raise alarms at the time, with little success.
Robert Hunter, a former Texas insurance commissioner now with the Consumer Federation of America, said the primary reason for the change to the five-year model appeared to be pressure from the insurance industry.
Thomas R. Knutson, a research meteorologist with the National Oceanic and Atmospheric Administration in Princeton, N.J., and another RMS expert panelist, said the five-year timeline didn’t come from the experts.
“I think that question was driven more by the needs of the insurance industry as opposed to the science,” he said.
In March, RMS said the five-year model was developed in cooperation with the expert panel that included Elsner and Knutson, and that based on their perspective: “Increases in hurricane frequency should be expected along the entire U.S. coast, but will be highest in the Gulf, Florida, and the Southeast, while lower in the Mid-Atlantic and the Northeast.”
“I didn’t make any such statement of that type,” Knutson said Friday.
Elsner said he warned RMS about flaws in the model. “I said that’s not a good way of doing it,” he recalled, and said RMS exaggerated the basic science “well beyond what we expected.”
Though RMS said in March that the expert panel “agreed unanimously that a forward-looking view of risk should reflect a higher probability of landfalling hurricanes,” Elsner said there was no consensus.
It doesn’t sound like we’ve heard the end of this issue:
Other experts in the catastrophe-modeling business have questions, too.
Long-term historical data are still the most credible, given the sparse data available for projecting the next five years, Karen Clark, chief executive officer of AIR Worldwide, said in a speech in the summer. Her company is an RMS competitor. Clark encouraged insurance companies not to replace the long-term model with the short-term one. Still, AIR has launched its own version of a five-year program for customers.
The details of how RMS arrives at its projections are considered a trade secret.
“We have never been able to get what they call the information out of the black box to review their models,” said Bob Lotane, a spokesman for Florida’s Office of Insurance Regulation. He said a public modeling system the state is working on should provide a way to verify the RMS projections.
Crist said information from RMS might be subpoenaed.
As we concluded last April,
From the perspective of the basic functioning of the insurance and reinsurance industries, the change in approach by RMS is an admission that the future is far more uncertain than has been the norm for this community. Such uncertainty may call into question the very basis of hurricane insurance and reinsurance which lies in an ability to quantify and anticipate risks. If the industry can’t anticipate risks, or simply come to a consensus on how to calculate risks (even if inaccurate), then this removes one of the key characteristics of successful insurance. Debate on this issue has only just begun.