Face & object recognition: teach your website to see through images

Recognizing faces and objects seems to be something our brain just knows how to do from birth. Still, the ability isn’t intrinsic to humans alone, since machines have become much smarter these days, and teaching your website or app to identify “what’s there in the picture” can benefit you a lot in terms of automation.

In this article, we’ll cover how you can train your system to recognize various objects using Uploadcare, and how you can actually use it.

Object recognition: with 93% confidence, there’s a ship

Uploadcare’s object recognition lets you know what's in your images. For example, a user uploads an image, and then sees a message that basically says something like this: “With 93% confidence, there’s a ship.” Of course, instead of a ship, there could be any other object, a group of objects, or one or more people.

We should note that object recognition in Uploadcare helps you define a general class of objects, not a specific item. It can detect a car and its type, (for instance, a pickup or a sedan), but it can’t determine the brand or exactly which (whose) car it is. Say, if you want to find Elvis's pink cadillac, you’ll have to use another solution for that. The classes are assigned automatically to all the objects when the feature is enabled. After the file gets uploaded to an Uploadcare account, it’s forwarded to recognition, and then its result is saved.

How can it be used in practice? Well, we’ve encountered the two main use cases.

1. Content moderation

For some websites, it’s important to make sure that the uploaded content is appropriate before it’s published for everyone to see.

For example, let’s say you’re running a dating service and wouldn’t want its users to upload pics of cute animals instead of their real photos. Or you have a marketplace like eBay with a catalog of items, and you want to make sure that a car won’t end up among the bikes. And of course, no one would like it if someone uploaded adult or violent content to their business website, because it can damage the reputation. In all these examples, the results of applying object recognition can be used for implementing automatic moderation.

With a picture identifier, you can automatically moderate ads and images without facesNice haircut by the way, but it probably won’t work if you restrict uploading images without a face in them.

2. Automatically tagging and annotating web images

Another use case is enabling automated annotations for images.

For example, you can configure Uploadcare to automatically create HTML5 alt tags for images. An alt tag is an attribute that provides a text alternative for an image. It’s widely used by search engines when indexing website content and is also important in terms of accessibility of the content for visually impaired users.

Also, you can use this feature to generate custom file names based on recognition outputs, and thus enable categorization.

Face detection: isn’t much different from object recognition

Face detection can determine if there are any faces in a picture. Unlike more sophisticated services that can produce a statement like, “A smiling female with her eyes open,” face detection in Uploadcare does more straightforward, yet still important work. It helps your system understand if there are any points of attraction in the picture.

In terms of the Uploadcare recognition system, faces are just another class of objects: they aren't much different for inanimate objects. So, it doesn’t really matter exactly what’s there — a human face, a carrot, or a plane — since the network can determine that there’s an object. Even if it can’t name it, it's enough information to enable the smart crop feature, for instance, which won’t cut out anything important.

A famous group photoA famous group photo

Image identifier identified ten people in the photoImage identifier identified ten people in the photo

You can play with finding faces in this live demo on this Codepen page.

Face detection is a useful feature when you ask users to add their profile images. For example, when a user uploads an avatar, you can set a frame around the face by default. Plus, it can automatically define if there’s a face at all, and won’t let anyone upload other types of pics. This could be essential for services like AirBnB or CouchSurfing, where credible photographs of potential tenants, housemates, and hosts are crucial for decision making.

How to teach your website to recognize what’s in pictures

To start recognizing images, you need to have an Uploadcare account. Create one if you don’t have one yet. When you’re set, drop us a line at help@uploadcare.com so Uploadcare engineers can enable the feature for you.

Once you’re ready, you can make an API request to get a list of detected objects and confidence levels.

Carrot imageCarrot image

{
  "rekognition_info": {
    "Plant": 0.93765,
    "Flora": 0.93765,
    "Food": 0.93765,
    "Tree": 0.51218,
    "Produce": 0.93765,
    "Carrot": 0.93765,
    "Conifer": 0.51218,
    "Vegetable": 0.93765,
    "Ivy": 0.60237,
    "Vegetation": 0.50727
  }
}

The provided confidence levels are floating-point numbers in the range from 0 to 1. If you prefer percentages, just multiply those numbers by 100. In our case, the result can be interpreted as saying Uploadcare is 93% sure that there are carrots, vegetables, and food in this image.

After an image is uploaded, object recognition automatically analyzes its content and fills in the image info. After the analysis is ready, you can acquire info on the detected objects via the Uploadcare REST API (v0.7).

Step 1. Make a request

In order to get the result of the recognition through the REST API, you need to use the query parameter when requesting a single file or multiple files (the categorization we mentioned earlier).

GET /files/?add_fields=rekognition_info

or

GET /files/:uuid/?add_fields=rekognition_info

Where:

  • the add_field parameter with the respective value directs our API to include the rekognition_info field in a response JSON, which contains object classes and confidence levels.
  • :uuid identifies the unique image for which you are requesting info on detected objects.

Here’s how it goes:

$ curl -H "Accept: application/vnd.uploadcare-v0.6+json" \
       -H "Authorization: Uploadcare.Simple publickey:secretkey" \
       "https://api.uploadcare.com/files/af9b7f02-6cbe-45bd-9bb6-9c928bf8a72e/?add_fields=rekognition_info"

Step 2. Get a response

Once we make a request that is properly authenticated, we get a JSON response:

{
  "datetime_removed": null,
  "datetime_stored": "2018-08-07T05:31:47.326146Z",
  "datetime_uploaded": "2018-08-07T05:31:47.132454Z",
  "image_info": {
    "orientation": null,
    "sequence": false,
    "format": "JPEG",
    "height": 2432,
    "width": 3648,
    "geo_location": null,
    "datetime_original": null,
    "dpi": [
        72,
        72
    ]
  },
  "is_image": true,
  "is_ready": true,
  "mime_type": "image/jpeg",
  "original_file_url": "https://ucarecdn.com/af9b7f02-6cbe-45bd-9bb6-9c928bf8a72e/house-pond.jpg",
  "original_filename": "eberhard-grossgasteiger-767950-unsplash.jpg",
  "size": 2363253,
  "url": "https://api.uploadcare.com/files/af9b7f02-6cbe-45bd-9bb6-9c928bf8a72e/",
  "uuid": "af9b7f02-6cbe-45bd-9bb6-9c928bf8a72e",
  "variations": null,
  "video_info": null,
  "rekognition_info": {
    "Building": 0.63291,
    "House": 0.63291,
    "Housing": 0.63291,
    "Architecture": 0.63008,
    "Outdoors": 0.55667,
    "Cottage": 0.63291,
    "Pond": 0.55667,
    "Castle": 0.63008,
    "Moat": 0.63008,
    "Fort": 0.63008
  }
}

So, we see that there’s likely a building, a house or a cottage in the picture. Let’s check if it’s true:

Picture recognition identified a houseIndeed, it’s a house.

What’s next?

These were just the basics of Uploadcare Object Recognition, a part of a larger Image Intelligence feature set, which allows you to translate pictures into data insights.

Whether you work for a big company or are just starting off with your own project, we’d love to see you beating the competition with a data-driven and automated approach to every uploaded asset. With that in mind, there are three possible scenarios for the next step:

  • Explore the docs section if you’d like to know how the technology works inside out,
  • take a shortcut and reach out to us to ask any questions directly,
  • or jump right in by starting a hands-on exploration of Uploadcare.

Infrastructure for user images, videos & documents