Unsafe content moderation

Uploadcare provides a content moderation feature that detects and identifies unwanted or offensive images right on upload, preventing inappropriate content from being published.

This feature applies only to images and can operate in two modes: automatically via the Dashboard and through the REST API.

Automatically via Dashboard

To automatically scan all uploaded images, enable the Scan images on upload toggle in the Image moderation section of the Dashboard.

You can define what happens to images containing unsafe content by selecting an Action — either automatically deleting flagged files or allowing them to be uploaded (in which case, information about the detected unsafe content will be added to the file’s appdata). See the example below for details on how to check the appdata.

You can fine-tune moderation by selecting content categories to check for. For each category, set a threshold value - a percentage that defines the confidence level above which a file will be automatically deleted.

Using REST API

For a custom solution, integrate Uploadcare’s REST API add-on, which detects and flags inappropriate content for further review.

Unsafe content moderation add-on works asynchronously:

  1. Start a processing job via REST API. Send an input file UUID with the necessary processing operations.
  2. Wait until the processing job status becomes done.
  3. Detected moderation labels will be stored in the JSON response in the appdata section of the processed file.

Example

Execute unsafe content detection on a target UUID:

$curl -X POST \
>     -H 'Content-Type: application/json' \
>     -H 'Accept: application/vnd.uploadcare-v0.7+json' \
>     -H "Authorization: Uploadcare.Simple $YOUR_PUBLIC_KEY:$YOUR_SECRET_KEY" \
>     "https://api.uploadcare.com/addons/aws_rekognition_detect_moderation_labels/execute/" \
>     -d "{\"target\": \"$UUID\"}"

Check out REST API reference to see how to execute an add-on on a target UUID.

To get the job result, you need to enable file.info_updated in the Webhook section of the dashboard.

After completing the processing job, the webhook will be sent to the endpoint you specified in the webhook settings.

File information in response:

1 "initiator": {
2    "type": "addon",
3    "detail": {
4      "addon_name": "aws_rekognition_detect_moderation_labels",
5    }
6  },
7  "hook": {
8    ...
9    "event": "file.info_updated",
10    "is_active": true,
11    "version": "0.7",
12    ...
13  },
14    "uuid": ":UUID",
15    "is_image": true,
16    "is_ready": true,
17    "metadata": {},
18    "appdata": {
19      "aws_rekognition_detect_moderation_labels": {
20        "data": {
21          "ModerationLabels": [
22            {
23              "Confidence": 99.87886810302734,
24              "Name": "Weapons",
25              "ParentName": "Violence"
26            },
27            {
28              "Confidence": 99.87886810302734,
29              "Name": "Violence",
30              "ParentName": ""
31            }
32          ],
33          "ModerationModelVersion": "6.1"
34        },
35        "version": "2016-06-27",
36        "datetime_created": "2023-11-08T09:35:08.907Z",
37        "datetime_updated": "2023-11-08T09:35:08.907Z"
38      },
39    }

If your application does not have a backend or uses a mobile version, you can submit the request yourself to check the execution status.

Use the request ID returned by the add-on execution request described above:

$curl -X GET \
>     -H 'Content-Type: application/json' \
>     -H 'Accept: application/vnd.uploadcare-v0.7+json' \
>     -H "Authorization: Uploadcare.Simple $YOUR_PUBLIC_KEY:$YOUR_SECRET_KEY" \
>     "https://api.uploadcare.com/addons/aws_rekognition_detect_moderation_labels/execute/status/?request_id=$REQUEST_ID"

As a request result, you will receive a UUID of the file when the status is done.

1{
2  "result": {
3    "file_id": "22240276-2f06-41f8-9411-755c8ce926ed"
4  },
5  "status": "done"
6}

Check out the execution status.

Once the status changes to done, appdata will contain the result of the execution.

To get it, run file info request, specifying the include parameter. There are two methods for getting info on detected objects via GET requests for single-file and multi-file cases:

$curl -H "Accept: application/vnd.uploadcare-v0.7+json" \
>     -H "Authorization: Uploadcare.Simple $YOUR_PUBLIC_KEY:$YOUR_SECRET_KEY" \
>     "https://api.uploadcare.com/files/?include=appdata"

or

$curl -H "Accept: application/vnd.uploadcare-v0.7+json" \
>     -H "Authorization: Uploadcare.Simple $YOUR_PUBLIC_KEY:$YOUR_SECRET_KEY" \
>     "https://api.uploadcare.com/files/$UUID/?include=appdata"

You’ll get JSON with appdata:

1"appdata": {
2  "aws_rekognition_detect_moderation_labels": {
3    "data": {
4      "ModerationModelVersion": "6.0",
5      "ModerationLabels": [
6        {
7          "Confidence": 93.41645812988281,
8          "Name": "Weapons",
9          "ParentName": "Violence"
10        }
11      ]
12    },
13    "version": "2016-06-27",
14    "datetime_created": "2023-02-21T11:25:31.259763Z",
15    "datetime_updated": "2023-02-21T11:27:33.359763Z"
16  },
17}

Billing

  • This feature is available on Trial and paid plans.
  • Learn how we charge for this operation.
Built with