serverless application to find unlabelled photos of you on twitter using machine learning (tensorflow.js).
findme
is a serverless application to find unlabelled photos of you on twitter using machine learning.
Users provide a search query to retrieve tweets from the Twitter API. Face recognition is used to compare all faces found in the search results against the user's twitter profile image. Tweets with matching faces are shown in the client-side web application.
If you want to deploy this project you will need an instance of the Apache OpenWhisk platform, access to a Redis database and credentials for Twitter and Auth0 applications.
Follow the instructions below to deploy this application on IBM Cloud (including dependent services). Auth0 and Twitter developer accounts need to be registered separately.
IBM Cloud Functions is a managed instance of Apache OpenWhisk running in the public cloud. Lite account users have access to 400,000 GB/s of free compute time per month.
IBM Cloud Functions is available in the following regions: us-south
, us-east
, london
and frankfurt
.
Log into the IBM Cloud CLI using the region endpoint chosen to deploy the application.
ibmcloud login -a <REGION_ENDPOINTS>
Install the Cloud Functions CLI plugin.
ibmcloud plugin install cloud-functions
IBM Cloud provides managed Redis instances that charges solely for usage, i.e. no fixed monthly fee.
Provision a new Redis instance through the IBM Cloud web console.
Follow the documentation here to retrieve connection strings for the database.
consumer_key
and consumer_secret
values for the application.app_domain
, client_id
and client_secret
values for the application.Install The Serverless Framework.
npm install serverless
Clone Git repository.
git clone https://github.com/jthomas/findme.git
Install project dependencies.
cd findme && npm install
If you don't want to manually build a custom runtime image, you can use the following pre-existing image: jamesthomas/action-nodejs-v8:tfjs-faceapi
. The serverless.yml
is already configured to use this runtime image. If you need to make changes to the runtime image, follow these steps...
Install Docker and sign up for an account at Docker Hub.
Log into the public Docker Hub registry using the Docker client.
docker login
Build the custom runtime locally.
docker build -t <DOCKERHUB_USERNAME>/tf-js .
Export the local image to Docker Hub.
docker push <DOCKERHUB_USERNAME>/tf-js
Update the serverless.yaml
image property with the new image name.
Create authentication credentials for Redis, Auth0 and Twitter in creds.json
file.
{
"redis": "redis://<REDIS_URL>",
"twitter": {
"consumer_key": "<CONSUMER_KEY>",
"consumer_secret": "<CONSUMER_SECRET>"
},
"auth0": {
"domain": "<USER_NAME>.auth0.com",
"clientId": "<CLIENT_ID>",
"clientSecret": "<CLIENT_SECRET>"
}
}
Run the deploy
command.
serverless deploy
Retrieve API Gateway endpoint (https://<APIGW_URL>/findme
) from deployment logs.
endpoints (api-gw):
GET https://<APIGW_URL>/findme/api/search/{id} --> search_status
POST https://<APIGW_URL>/findme/api/search --> schedule_search
Update CONFIG
value in public/script.js
with API Gateway URL and Auth0 application identifiers.
const CONFIG = {
auth0: {
clientId: '<CLIENT_ID>',
domain: '<USER_ID>.auth0.com'
},
backend: 'https://<APIGW_URL>/findme'
}
Start web server to host static files in public
directory, e.g.
python -m SimpleHTTPServer
Open index.html
on web server. 👍
This application has four serverless functions (two API handlers and two backend services) and a client-side application from a static web page.
Users log into the client-side web page using Auth0 and a valid Twitter account. This provides the backend application with the twitter profile image and API credentials.
When the user invokes a search query, the client-side application invokes the API endpoint for the register_search
function with the query terms and twitter credentials. This function registers a new search job in Redis and fires a search_request
trigger with the query and job id. This job identifier is returned to the client to poll for real-time status updates.
The twitter_search
function is connected to the search_request
trigger and invoked for each event. When this function is invoked, it uses the Twitter Search API to retrieve all tweets for the search terms. If the tweets contains photos, each tweet and photo url is fired as a separate tweet_image
trigger event.
The compare_images
function is connected to the search_request
trigger. When this function is invoked, it downloads the user's twitter profile image along with the tweet image and runs face extraction against both images. If any faces in the tweet image match the face from the user's profile image, tweet ids are written to Redis before exiting.
The client-side web page polls for real-time search results by calling the API endpoint for the search_status
function with the job identifier for the search. If tweet ids are returned from the search results, those tweets are displayed on the web page using the Twitter JS library.
lib/twitter/api.js
file.If you have any issues, comments or want to see new features, please file an issue in the project repository: