ImageMonkey is an attempt to create a free, public open source image dataset.
ImageMonkey is a free, public open source dataset. With all the great machine learning frameworks available it's pretty easy to train pre-trained Machine Learning models with your own image dataset. However, in order to do so you need a lot of images. And that's usually the point where it get's tricky. You either have to create the training images yourself or scrape them together from various datasources. ImageMonkey aims to solve this problem, by providing a platform where users can drop their photos, tag them with a label, and put them into public domain.
There are basically two ways to set up your own ImageMonkey
instance. You can either set up everything by hand, which gives you the flexibility to choose your own linux distribution, monitoring tools and scrips or you could use our Dockerfile
to spin up a new ImageMonkey
instance within just a few minutes.
The docker image is for development only - do NOT use it in production!
The following section contains some notes on how to set up your own instance to host ImageMonkey yourself. This should only give you an idea how you could configure your system. Of course you are totally free in choosing a different linux distribution, tools and scripts. If you are only interested in how to compile ImageMonkey, then you can jump directly to the Build Application section
Info: Some commands are distribution (Debian 10) specific and may not work on your system.
imagemonkey
with adduser imagemonkey
PermitRootLogin
line in /etc/ssh/sshd_config
to PermitRootLogin no
)#!bash
iptables -P INPUT DROP && iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
#!bash
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
#!bash
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables-persistent
to load firewall rules at startupiptables-save > /etc/iptables/rules.v4
iptables -L
/etc/postgresql/9.6/main/postgresql.conf
and set listen_addresses = 'localhost'
service postgresql restart
to apply changes/env/postgres/schema.sql
with psql -f schema.sql
monkey
by executing the following in psql:CREATE USER monkey WITH PASSWORD 'your_password';
\connect imagemonkey
GRANT ALL PRIVILEGES ON DATABASE imagemonkey to monkey;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO monkey;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO monkey;
GRANT USAGE ON SCHEMA blog TO monkey;
test if newly created user works with: psql -d imagemonkey -U monkey -h 127.0.0.1
populate labels with go run populate_labels.go common.go web_secrets.go
add donation image provider with insert into image_provider(name) values('donation');
build temporal_table
extension, as described here: https://github.com/arkhipov/temporal_tables
connect to imagemonkey database and execute CREATE EXTENSION temporal_tables;
connect to imagemonkey database and execute CREATE EXTENSION uuid-ossp;
connect to imagemonkey database and execute CREATE EXTENSION postgis;
apply defaults.sql
apply indexesl.sql
apply sql functions from env/functions
directory
apply sql stored procedures from env/stored_procs
directory
apt-get install redis-server
maxmemory
(e.g: 500mb) and set maxmemory-policy
to allkeys-lru
Windows:
PKG_CONFIG_PATH=/c/Users/Bernhard/Downloads/vips-dev-w64-all-8.6.5/vips-dev-8.6/lib/pkgconfig
/c/Go/bin/go get -u gopkg.in/h2non/bimg.v1
Linux:
apt-get install nginx
apt-get install nginx-extras
apt-get install certbot
certbot certonly
to obtain a certificate for your registered domainconf/nginx/nginx.conf
and replace imagemonkey.io
and api.imagemonkey.io
with your own domain names, copy it to /etc/nginx/nginx.conf
and reload nginx with service nginx reload
Minimal required Go version: v1.11.10
apt-get install git
apt-get install golang
export GOPATH=$HOME/go
export GOBIN=$HOME/bin
go get -d ./...
go install api.go api_secrets.go common.go imagedb.go
go install web.go web_secrets.go common.go imagedb.go
wordlists/en/misc.txt
to /home/imagemonkey/wordlists/en/misc.txt
mkdir -p /home/imagemonkey/donations
mkdir -p /home/imagemonkey/unverified_donations
apt-get install supervisor
imagemonkey
user to supervisor group with adduser imagemonkey supervisor
mkdir -p /var/log/imagemonkey-api
, mkdir -p /var/log/imagemonkey-web
mkdir -p /var/log/imagemonkey-statworker
, mkdir -p /var/log/imagemonkey-bot
, mkdir -p /var/log/imagemonkey-blog-subscription-worker
, mkdir -p /var/log/imagemonkey-data-processor
, mkdir -p /var/log/imagemonkey-labelsdownloader
, mkdir -p /var/log/imagemonkey-trending-labels-worker
conf/supervisor/imagemonkey-api.conf
to /etc/supervisor/conf.d/imagemonkey-api.conf
conf/supervisor/imagemonkey-web.conf
to /etc/supervisor/conf.d/imagemonkey-web.conf
conf/supervisor/imagemonkey-statworker.conf
to /etc/supervisor/conf.d/imagemonkey-statworker.conf
conf/supervisor/imagemonkey-blog-subscription-worker.conf
to /etc/supervisor/conf.d/imagemonkey-blog-subscription-worker.conf
conf/supervisor/imagemonkey-bot.conf
to /etc/supervisor/conf.d/imagemonkey-bot.conf
conf/supervisor/imagemonkey-labels-downloader.conf
to /etc/supervisor/conf.d/imagemonkey-labels-downloader.conf
conf/supervisor/imagemonkey-trending-labels-worker.conf
to /etc/supervisor/conf.d/imagemonkey-trending-labels-worker.conf
EnvironmentFile=/etc/environment
to the service section of the systemctl supervisor config file (see https://stackoverflow.com/questions/47083582/supervisor-not-using-etc-environment)systemctl daemon-reload
and systemctl restart supervisor
supervisorctl reread && supervisorctl update && supervisorctl restart all
on imagemonkey-playground instance
rsync
with apt-get install rsync
backupuser
with adduser backupuser
(use a strong password)backupuser
with su backupuser
and create a new SSH key with ssh-keygen -t ed25519 -a 100
ssh-copy-id -i ~/.ssh/your_generated_id.pub backupuser@imagemonkey-host
backupuser
permissions to write to /home/playground/donations
with: chgrp backupuser /home/playground/donations && chmod g+rwx /home/playground/donations
backupuser
with: crontab -u backupuser -e
and add the following line (runs rsync every 15min):*/15 * * * * rsync -a [email protected]:/home/imagemonkey/donations/ /home/playground/donations/