Researchers use AI to 'see' beyond a structure's facade in Google Street View

Researchers at the University of Toronto have developed a method that uses artificial intelligence (AI) and Google Maps’ Street View images to glean more detailed information about buildings – such as their age and floor area.

This additional data can then be used to assess building stock, construction material flows and embodied greenhouse gases, which are estimates of the emissions generated by the production and transportation of goods.

The study was published in the Journal of Industrial Ecology.

“This is the first paper we know of where people took a picture that shows you the front of the building and then predicts things that you can’t see in the picture,” says Shoshanna Saxe, an associate professor in the department of civil and mineral engineering in the Faculty of Applied Science & Engineering who led the research team through the Centre for the Sustainable Built Environment. 

“My motivations were very focused on embodied carbon research use, but this will be useful for lots of different people. I’ve talked to researchers who are looking at understanding water usage for future planning or resilience assessments.”

Because Google Street View is widely available, the method offers a cost-effective way to generate large-scale building data.

“We spent maybe $1,000 on photos to get data that would otherwise cost millions of dollars to obtain,” Saxe says. “Nobody has millions of dollars to spend on just building dimensions, so this is the difference between being able to work on these problems and not. Having methods that can let us understand neighbourhoods and buildings at scale is really useful.” 

The team trained the AI to estimate building attributes based on exterior images of the structure, achieving 70 per cent accuracy for age prediction and 80 per cent accuracy for floor area prediction.

“Being able to assess the exteriors allows a sort of educated guess at the interiors and the kinds of uses the occupants put on local infrastructure,” says co-author Alex Olson, a senior AI researcher at U of T’s Centre for Analytics & Artificial Intelligence Engineering.

“It gives a strong estimate of the resources used in building, maintaining and operating the buildings.”  

Saxe adds that insights gained through their approach can’t be derived from maps or building plans alone.

“You need to see structures,” she says. “One of the distinctions is we’re predicting what the internal square footage of the building is. And, although obviously that tracks with the size of the outside of the building, it’s actually harder to predict. And you also can’t see how old the building is from the outside.

“If you have experience, you can walk around and say, that building looks about this old to me, this building looks about that old to me, and so on. But there’s all kinds of things about it that make it hard, including renovations. The front can be different from the back. And is the frontage brick, glass or is it concrete? Knowing the age of the building is important, as it tells you what materials were used and what embodied carbon there is. And, also, how it performs.”

Using AI to look beyond building facades could help urban planners better understand cities’ resource needs and prioritize future infrastructure projects. 

“You want to understand where there’s underused resources or infrastructure in your city,” says Olson. “It sounds like we should already have the data, but we really don’t. With this, while it doesn’t model the future, it does quite accurately describe what the current situation is and allows us to use the data for planning our resource uses and what we want to do in the future.”