Bank AVMs Are As Wildly Inaccurate As A Zestimate – But Will Be Regulated As Legitimate Values

June 24, 2024
  • AVMs are incredibly inaccurate and are being misused in property valuation
  • AVMs don’t consider the condition of a home
  • The mortgage industry’s push toward automation has reduced valuation accuracy

AVM or Automated Valuation Models have been incorrectly seen as the human-less way to value property. The technology has been drifting into mortgage lending reliance for more than a decade because it has been marketed as having the ease of “pushing a button.”

The Zestimate product by Zillow introduced the consumer to the concept nearly twenty years ago. When Zillow‘s website launched circa early 2006, there were no listings. The service was initially a toy for consumers (a.k. real estate porn) to value their own homes or the homes of their friends, neighbors, relatives, or celebrities. It was viewed as a threat by appraisers like me, who value real estate for a living, for its significant lack of accuracy. It has been the scourge of real estate agents and brokers, particularly when a homeowner who arrived at an inflated listing price using the Zestimate tool, becomes anchored to that inflated price. When I was part of a national start-up designed to compete with Case Shiller, I saw empirical evidence that it takes 12-18 months to decouple a seller anchored to an incorrect price. Then think about those millions of consumers inputting their addresses, pressing a button, and arriving at a value much higher than what their home is worth.

The Journey Is The Reward?

My journey to the Zestimate started in 2006 when I was invited to be on the industry advisory panel for Trulia by the founders just before their website launched. I was flown out to their San Francisco headquarters twice a year to provide feedback and a potentially different perspective than the real estate brokers on the panel. As one would expect, they had the obligatory volleyball court within their cavernous raw office loft, a ton of excitement held by a lot of super nice smart people accepting nominal salaries but would share in the upside when Trulia went public. I remember having lunch with the two co-founders early on when they shared their rental car management strategies to save money over owning one. Their pragmatism was impressive. I loved the Trulia culture and the good guy position against Zillow, similar to the Mac versus PC ads that were running at the time (I’m a Mac devotee). Later in their evolution, they built a valuation tool like the Zestimate. After their acquisition in 2014 by Zillow, the advisory board was dissolved and one of the co-founders (I think it was Pete), told me I was the only continuous member from start to finish). Yay.

Rhymes With Pillow

I’ve shared this story before, but the evening before Zillow’s launch, I was attending an Inman Connect afterparty in New York and was chatting with Lockhart Steele, the founder of Curbed, the neighborhood blog that was later to become part of Vox and CNBC. I wrote a column for Curbed for years and learned a lot about blogging and getting some thicker skin by taking a combination of loads of petty criticism and constructive criticism. Who knew that eliminating the use of gradients and shadows while relying on sans-serif fonts mattered?

Lockhart introduced me to Rich Barton, who was very nice to speak with. I asked him what he did, and he mentioned he was the co-founder of Expedia and after some time off he was launching his next product the following morning: Zillow. He followed with something like “Rhymes with Pillow.” Little did I realize how much he was going to change real estate, forever.

Zestimate Accuracy Metrics When Considering “Median”

AVMs are not credible yet in my opinion, because they are dependent on the slow improvement of public records, which varies significantly across municipalities. Think of Zestimates as a consumer-facing AVM. In my own experience with a startup and market report series, I find that public record is cleaner moving westward. For example, it has been my experience that public record is dramatically cleaner in California than in New York, but this may just be random.

For this piece, I went to update myself on the accuracy of Zestimates on the Zillow website. Their national accuracy metrics have deteriorated since last year. But I have to give Zillow credit since they disclose their metrics. Here is the latest summary followed by an explanation of what the results really mean.

The Zestimate’s accuracy depends on location and the availability of data in an area. The Zestimate’s median error rate for on-market homes nationwide is 3.2%, meaning Zestimates for half of all on-market homes are within 3.2% of the ultimate sale price, and half are not. For off-market homes, the median error rate is 7.52%. Check your home value estimate for free by entering your home address. You can improve the accuracy of your Zestimate by claiming the address as your home and updating the home details.

Zillow

The keyword used is “median” in their accuracy rate descriptions because in my view it makes Zestimates unbelievably inaccurate. Zestimates are truly an excuse to lurk.

  • If a property is listed for sale, half of the Zestimates are within 3.2% of the ultimate sales price and half are not.
  • If a property is not listed for sale, half of the Zestimates are within 7.52% of the ultimate sales price and half are not.

Translation The term “median” splits the results in half so half the properties analyzed could be off by 500% and the other half could be off by 3.2% making the median accuracy rate 3.2%. The consumer reads this as accurate within 3.2% since they don’t understand the application of the word “median.” If I was appraising a home and told the client, half the time I am usually within 3.2% but the other half of the time I am off an average of 500%, I would never be taken seriously. The Zestimate is literally so bad that when the AVM punches out what ends up being accurate results, it comes down to luck.

In the early days, one of the co-founders of Zillow and champion of the Zestimate came to meet with me in our New York office hoping to have access to our historical co-op data which was not in public record since a lot of banks were headquartered in Manhattan and they needed to impress the bank CEOs. I brought up an example of a Zestimate scenario in a cookie-cutter suburb in suburban Detroit where my father-in-law lived that generated a Zestimate that was way off to the low side. I knew about recent identical sales and heard about some contracts. The executive’s response was something like: “The consumer knows when the Zestimate is wrong.” My response was “No, consumers really don’t know when the Zestimate is wrong.” A property owner can probably tell when a Zestimate is 100%, 200%, or even 500% too high or low. However, if the Zestimate is off by 10%, it can be the difference between selling and not selling a home.

Just before the pandemic, I was asked to speak at NAR headquarters in Washington D.C. in front of most of the stakeholders in the valuation process including the chief appraisers of Fannie Mae and Freddie Mac. The speaker before me represented an AVM service and was bragging about how accurate they were, extolling that they were within 90% of the actual sales price showing lots of charts and graphs while gushing. I was embarrassed for him. Thankfully he left before I began my presentation. But AVMs are now ruling the day as the appraisal industry had no actual leadership in those formative years, legitimizing AVMs without pushback from the industry that values property.

Initially, AVMs were used for second-tier lending like home equity loans. An example of selecting an AVM over an appraisal might go like this:

  • Borrow has a $1,000,000 mortgage on a property that sold 5 years ago for $2,000,000.
  • They are applying for a home equity loan of $200,000.
  • An AVM would be used to consider the odds there was still equity left in the property.

Here is such a chart currently shared on the AVMetrics website, a good source for pro-AVM information. I remember meeting their founder a few times at some Washington hearings over the years. Note that 92.4% of the AVM results in California are within 10% of the sales price. According to Census, the average sales price of a US home a year ago was $495,100. If it is rounded it to $500,000, AVM providers think being within $50,000 of the eventual sales price is a reliable result.

The recent ruling to regulate the credibility of AVMs by the OCC and FDIC essentially legitimized the use of AVMs in lending. The driver behind this final rule was to eliminate potential bias in valuations by replacing appraisers with AVMS. Yet the Urban Institute study Revisiting Automated Valuation Model Disparities in Majority-Black Neighborhoods said:

But even with data improvement and artificial intelligence, we still find evidence that the percentage magnitude of AVM error is greater in majority-Black neighborhoods. This finding indicates that we cannot reject the role historic discrimination has played in the evaluation of home values. 

Urban Institute

When comparing the accuracy of AVMs and the actual sales price, while there are significantly greater inaccuracies in majority-black neighborhoods, AVMS are disconnected from the actual sales prices by 20% to +40%. This example shows how AVM results are garbage and shouldn’t be relied on for first mortgages.

AVM software is built by humans who have inherent biases. The void in representation by the appraisal industry over the past decade on the AVM issue, to talk about those 200 feral cats living in the house being valued, has enabled AVMs to be legitimized by the federal government.

During my career, I have observed that valuation accuracy has become weaker as technology has expanded in the mortgage process. The wiz-bang concept that the appraisal of a property can be completed at the push of a button is missing the realities of valuation.

Incidentally, this post was written on a Mac.

Did you miss Friday’s Housing Notes?

June 21, 2024

Outsized Rate Reactions Expected: The First Cut Is The Deepest

Housing Notes Reads

Market Reports

Extra Curricular Reads