Realtime Image Moderation At Scale Using AWS Rekognition

Written by Rachana Sharma, Software Engineer at Powerupcloud.

One of the interesting problems we solved for a customer recently is image moderation at scale. The customer processes close to 2500 user profiles everyday, each user profile consisting an average of 6 different pictures - that makes it processing 15000 user pictures daily.

The customer, who is India's well-known online matrimonial site (for the Western audience reading this, matrimonial sites are roughly equivalent to the dating sites, with blessings from elders and parents) wanted to make sure that the images that are being uploaded are validated for following criteria

  • Irrelevant photos. A photo is irrelevant if no face is detected
  • Indecently dressed, nudity.
  • Group photos are now allowed
  • Photos that are blurred are not allowed
  • Celebrity photographs should be rejected.
  • Match the face from photograph with existing users and check if someone is assuming other's identity

Enter Amazon Rekognition. Ever since Rekognition was introduced at re:invent last year, our team has been waiting for an excuse to play with it and this customer requirement was just the excuse we needed.

The Architecture

Using S3, Lambda, Amazon Rekognition and a small RDS instance to save moderation results and image metadata for application to consume, we quickly came up with a fully functional working setup. The architecture looks like this:

As soon as the images are uploaded to S3, We trigger lambda functions that interact with Rekognition API for various criteria and also a custom Flask API we built to test some edge use cases which are currently not supported by Rekognition. The results are then written to an RDS instance along with Image metadata.

If we were to do the heavy lifting of choosing right machine learning and deep learning libraries, running them to get the desired output it would arguably have taken more time and effort. While Rekognition is pretty neat and I am sure will get better with each release - there were still some edge use cases for which we had to write custom code.

So how we did it?

Detect Faces - for filtering irrelevant photos. As an input, it takes S3 bucket image URI and the response returns the following information for each detected face.

  • Face details: Identifies face in image, this solved problem of irrelevant photos(no face detected) and group image detection( multiple face detection)
  • Gender: With this attribute, we can solve problem of Gender mismatch in user’s profile.
  • Pose: The Pose data describes the rotation of the face detected.( indecent pose)
  • Quality: Describes the brightness and the sharpness of the face. This might be helpful to filter image with certain quality threshold.

Sample request:

Sample Response:

Below are screenshots from this in action from our internal demo site. The dog below, although cute is not human unfortunately.

Here is one of the group photos. Btw, this pic is from our annual outing. Its is rejected because its a group photo.

Searching Face

This operation searches a face collection for face matches. This is done in with storing faces in face collection. Searching face can be used with dataset as collection with celebrity images (Celebrity image filtering) and with the user data of present users ( Duplicate image filtering).

Here is how you create a collection

response = client.create_collection(  
    CollectionId='MyAwesomeCollection'
)
Response  
{
    'StatusCode': 123,
    'CollectionArn': 'string'
}

index_faces detects faces in the input image and adds them to the specified collection.

response = client.index_faces(  
    CollectionId='CollectionName',
    Image={
               'S3Object': {
            'Bucket': 'BucketName',
            'Name': 'ImageName',
        }
    },
    ExternalImageId='AliasImageName',
    DetectionAttributes=[
        'DEFAULT'|'ALL',
    ]
)

Search_faces_by_image - For a given input image, first detects the largest face in the image, and then searches the specified collection for matching faces. The operation compares the features of the input face with faces in the specified collection.

response = client.search_faces_by_image(  
    CollectionId='string',
    Image={
        'Bytes': b'bytes',
        'S3Object': {
            'Bucket': 'string',
            'Name': 'string',
        } },   MaxFaces=123,  FaceMatchThreshold=)

Features (currently) unavailable in Rekoginition

Blur detection and indecently dressed (or pornographic content) are a couple things we had to deliver for this customer but are unavailable in rekognition at the time of this writing. Here is how we dealt with it:

Blur Detection

We used libraries like OpenCV and a custom logic which applies Fast Fourier Transform to the image using the default numpy functions, once this is done the mean value in the transformed image is taken, this is then scaled with respect to the size of the image to compensate for the rippling effect. This value is then used to threshold the image with larger values being indicative of an in focus image while lower values of blurred images. Following is python snippet to convert image in grayScale and then apply transformations.

Nudity Detection The definition of nudity in image is subjective and contextual. There are some open-source algorithms but they are based on skin-tone detection solely. We have used special-made, smut-trained, porn-detecting neural network to get more accurate results. This model uses Caffe which is a deep learning framework made with expression, speed, and modularity in mind. The deep model was first pre trained on ImageNet 1000 class dataset and then with porn and non porn images. Deeper networks, or networks with more filters can improve accuracy.

the network takes in an image and gives output a probability (score between 0-1) which can be used to filter not suitable for work images. Scores < 0.2 indicate that the image is likely to be safe with high probability. Scores > 0.8 indicate that the image is highly probable to be not safe.

Here is an example image:

And a celebrity picture. India know's that Salman Khan is an eternal bachelor, but that doesn't mean that you should use his picture on a matrimony site ;)

Finally, the example of a good and acceptable picture. The model here is our team member, Sarthak :)

Hope you find this useful. Have fun moderating images :)

comments powered by Disqus