Screenshot of Azure Custom Vision Service Portal

Using Custom Vision Service to Find the Perfect Home

Sam RuebyMachine Learning Leave a Comment

This post was originally published on LinkedIn: you can find it here.

Last week I attended Microsoft Build 2018 where they announced several cool new services and features. One that caught my eye was Custom Vision Service. Custom Vision Service allows you to create a classifier by uploading example images of what you’re looking for. After training a model, you can provide Custom Vision Service an image and it will return a confidence level that this image is also an example of the images you used to train the classifier.

Soon my wife and I will be looking to buy a house. Everyone has an idea of what they’d have in their dream house: a big yard, a reading nook, a basement movie theater. Some of these are nice-to-haves and others are requirements. I really like balconies but could live without one if I have to. My wife however, has a requirement. Our future house must have a third-story window. The reason is obvious: houses with third-story windows do not have ghosts in the attic.

House hunting can be difficult. At least there’s a number of search filters they provide you. How many square feet? How many bathrooms? How many bedrooms? Unfortunately, there isn’t a filter for ghost-preventing third-story windows. Fortunately, there’s a way we can apply machine learning to house hunting so that we are able to filter out available homes with requirements that aren’t easily searchable. Today I am going to show you how I used Custom Vision Service to only find homes that have a third-story window.

Usually when you’re searching for houses the first image you see is the front of the house. This is perfect because this is all we need to determine if the house has our criteria and it’s very likely that every listing will have such an image. So first thing I did was search for example images of houses that both have examples of no third-story window and ones that do and saved them to separate folders. After a quick jaunt around the Internet I found that it’s more common to find houses that don’t: I ended up with a small set of 10 examples of houses that meet our criteria of third-story window and 34 that do not. Next I headed over to the Custom Vision portal and created a new project.

After creating a new Custom Vision project, I uploaded the images and used three separate tags:

  • house – so that we can at least agree that a given test image is a house at all
  • has-third-story-widow – this will be the tag we’re looking for.
  • no-third-story-window – this can let us know that it may not be sure a house has a third-story window, but is confident that it doesn’t.

When we submit a test image we will receive each of these tags with a percent-confidence level. This will allow us to determine if a given image is a positive match, and therefor include it in the results.

Custom Vision Service portal with tagged images

Custom Vision Service portal with tagged images

From here you click the green ‘Train’ button in the upper-right to train you model. Only after this are you able to run a ‘Quick Test’ where you can upload an image and determine the the confidence the image contains each of your tags.

Next, we need to integrate this with an API that allows you to search for for-sale homes. Ironically this was the most difficult part of the project: I am unable to find a free API that allows you search for homes. But this doesn’t mean we can’t pretend!

Say we have an API that given some search criteria, returns to us homes that are for sale as a JSON object. Included in this object, in an array of homes, are URLs to images of the home. Using these images we can submit an image of the front of the house (likely the first one) and Custom Vision Service will tell us if thinks it has a third-story window. From here, we use this information to filter our own results and only show us the houses that we are interested in!

In my test project, I include 11 houses that are returned from my pseudo home-search API. Custom Vision Service was able to guess correctly 64% of the time with images of houses that were not used to train the AI. That’s pretty awesome given that only 44 images were used to train the model, with only 10 examples of houses that included the window we’re looking for. Besides a few false positives and false negatives, the AI was successful in examples that included a vent-like object on the house that looks like a window, and at least one example where the 3rd-story window was mostly obscured. I am confident that with more iterations and more positive/negative examples that this model can guess with higher accuracy.


Fake realty site showing Custom Vision Service being used to find third-story windows.

With very little effort I was able to create an AI model using Custom Vision Service that can guess with reasonable accuracy, a feature that is otherwise unable to be searched-for by anything other than the human eye. That’s pretty signifiant.

You can view the source of my system project over at GitHub: However, due to cost API limitations, you will have to train your own model and update the API endpoint in order to test. Learn how to get started with your own project by heading over to How to build a classifier with Custom Vision.