By Simon Mackinnon

In 2017, I saw a YOW! conference talk by Chanuki Illushka Seresinhe (Beautiful Environments on Human Well-Being). At the end of her talk, she mentioned that a website, to capture image scenic rating vs. location, didn’t exist in Australia. At the time I had a thought that it’d be cool to build it, but forgot about it for some time. Then, while studying for my AWS Cloud Practitioner exam, I thought I needed a hobby site to apply my learnings to. So, over the last few months, I’ve been spending some of my time building a website for analysing and quantifying where and what is considered scenic in Australia.

The purpose of this was to:

  • practice my AWS and ReactJS skills, and
  •  build something that can be useful in society.

The result of this work can be seen at RateMyScene.net.

The architecture is fully serverless, API-driven and scalable. Below is a description of some of the main functionality:

React front-end, hosted in AWS S3:

I built the front end using create-react-app.I used some templated styling from StyleShout, which made making the site somewhat pretty a little easier. My skills in React are still pretty raw, so I’ve tried to keep is relatively simple, while still keeping it responsive.

Hosting static websites is really easy! Simply create a bucket in S3, upload the website content, set the permissions of the bucket to public, configure to host a static website and Voilà: Serverless (static) Website!

I used CloudFront (CDN) in front of this bucket and then pointed the DNS entry’s alias to the CloudFront Distribution. Leveraging CloudFront also allowed me to use a generated SSL certificate (free as part ofAWS certificate manager)  to get the little locked padlock icon on my site (makes me happy!)

API Simple File-Upload to S3:

Using Lambda pushing files to S3, and exposing this function via an API Gateway endpoint I can push files to an S3 bucket. I limited the size of the uploads to 500KB originally, but made it 5MB after finding 500KB too small. The basis for this function was based on some code provided to me in the MelbourneAWS Programming and Tools Meetup.

Capturing Data in DynamoDB:

I needed to persist data about the image somewhere. This info would be geo-location, filename, rating info, etc. Using a NoSQL DB allows adding extra information going forward, without the overhead of schema changes that go with a relational DB. I’ll explain in more detail about this in a later post.

Integrating Image Recognition via Rekognition

The purpose of the website was to find where is most scenic. However, the rating data can also be to find what is most scenic. To achieve this, I used AWS’s image-recognition service, Rekognition, to get metadata about the images that are uploaded. This information is stored alongside the other image data in the DynamoDB table. It works relatively well for most general images, but can fall over for more ‘unique’ images.

An example of Rekognition can be seen below. The following image,

Results in the following output:

{
"Labels":[
{
"Name":"Cliff",
"Confidence":93.91314697265625
},
{
"Name":"Outdoors",
"Confidence":93.91314697265625
},
{
"Name":"Promontory",
"Confidence":90.42848205566406
},
{
"Name":"Coast",
"Confidence":72.45474243164062
},
{
"Name":"Nature",
"Confidence":72.45474243164062
},
{
"Name":"Ocean",
"Confidence":72.45474243164062
},
{
"Name":"Sea",
"Confidence":72.45474243164062
},
{
"Name":"Water",
"Confidence":72.45474243164062
},
{
"Name":"Rock",
"Confidence":51.833106994628906
},
{
"Name":"Beach",
"Confidence":50.963348388671875
}
]
}

The main aim of this part is to measure the relationship between the image content (what is in the image) vs. how scenic people think it is.

Sending emails on file-upload

AWS Simple Email Service (SES) is really simple to set up and use. The SendEmail function just takes in some text, a subject, sender and recipient, and just works (after the email has been set up).

If you want a html email to be sent, the SendRawEmail would be useful.

Using Google Maps API

This proved to be one of the more difficult parts of the project. The way the API works (registering for developer access, API keys) has been slightly modified recently, so a lot of examples I found on forums, etc.were a bit out of date. Capturing the location of the user’s click as longitude and latitude proved a little difficult.

The API response returns a function that returns this information, so this was a pretty tricky part.

Import the React Google Maps library:

import { withScriptjs,
withGoogleMap, GoogleMap, Marker } from "react-google-maps";

Defining the component:

const MyMapComponent = compose(
googleMapURL: "https://maps.googleapis.com/maps/api/js?key=${YOUR-GOOGLE-MAPS-API-KEY-HERE}&v=3.exp&libraries=geometry,drawing,places",
withProps({
googleMapURL: "https://maps.googleapis.com/maps/api/js?key=${YOUR-GOOGLE-MAPS-API-KEY-HERE}&v=3.exp&libraries=geometry,drawing,places",
    loadingElement: <div style={{ height: `100%` }} />,
    containerElement: <div style={{ height: `27em` }} />,
    mapElement: <div style={{ height: `100%` }} />,
}),
  withScriptjs,
  withGoogleMap
)((props) =>
  <GoogleMap
    defaultZoom={3}
    defaultCenter={{ lat: -26.213, lng: 134.428 }}
    onClick={props.onClick}
  > {props.markers.map(marker => (<Marker {...marker}/>))}
  </GoogleMap>
);

Adding a map component

<MyMapComponent key="map"onMarkerClick={this.handleMarkerClick} onClick={this.handleMapClick}markers={this.state.markers}/>

Handling a users click

  handleMapClick = (props) => {
    this.setState({
      latitude:props.latLng.lat(),
      longatude:props.latLng.lng(),
      markers:[{
        key: "default",
        position:{
          lat:props.latLng.lat(),
          lng:props.latLng.lng(),
        }
      }]
    });
  }

Security

Besides using IAM roles to lock down what can call the APIs, it’s important to make sure any submissions are made by humans and the content is appropriate. For now, the simplest way to exclude bots from submitting images was to use Google’s reCaptcha. This was pretty simple to implement. This is made quite straightforward by the NPM module react-recaptcha (https://www.npmjs.com/package/react-recaptcha).

Some basic moderation of images is required for submitted images too. All new images are reviewed before being allowed to be displayed.After notification is received, I can observe the image easily, and check mark if it should be available to be rated publicly.

Future planned improvements:

Better UX

The submission process is clunky. Errors currently just redirect the user to the home page, so some better error handling, notification of successful uploads and responsive UI

Step Functions

The first planned improvement is to use a step-function for the submission process. This currently has a lambda function that calls multiple other lambdas. To reduce the complexity of this process, and to allow for further functionality to be added, using step functions would be perfect.The basic flow of step functions is defined graphically in the console, e.g.

This makes debugging and organising code execution really straight-forward.

Auto-Moderation

Rekognition also provides the functionality to detecting unsafe content (https://docs.aws.amazon.com/rekognition/latest/dg/moderation.html).Integrating this into the submission process for auto-moderation would be a significant way of reducing manual moderation effort.

The ultimate plan is to feed this approval vs. content data about what images are approved or not into a Machine-Learning algorithm that would classify the approval rating confidence based on the content (i.e. people or animals, art, etc.). This would drastically step up the efficiency of the moderation.

Comments

  1. Nice job, Simon.
    I’ve used Step Function quite a bit … very excellent for chaining Lambda and it does help with debugging.

Leave a Reply

Your email address will not be published. Required fields are marked *