Node Mongo Express

An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.

---

Expert teams of digital product strategists, developers, and designers.

---

We’re always looking for people who value their work, so come and join us. We are hiring!

Pre-requisites

- yarn
- docker

Features

- Mongo support
- Docker support
- Rate limited APIs
- RBAC middleware using Auth0
- Sharding mongoDB collection support
- Paginated APIs
- Autogenerated APIs from mongoose models
- Built in slack alerting mechanism
- Suport for redis cache
- Support for aggregate caching
- Support for batch jobs in multi-container environment
- Support for circuit breakers
- Autogenerated swagger documentation
- Load testing using k6
- Support for i18n

Running Load tests

• Install k6
• Execute the following command: k6 run __tests__/__load__/script.js

Build and run docker container locally

- docker-compose down
- docker-compose build
- docker-compose up

Shard setup

Run the following script

./setup-shards/scripts/setup/base.sh

Take a look at this to create shards and replica sets.

Seeders

Run the following command to begin seeding

./seeders/seed.sh

How to start

- cd `node-mongo-express`
- yarn
- ./setup-shards/scripts/setup/base.sh
- cp .env.example .env.local
- ./seeders/seed.sh
- yarn start
- open browser to `localhost:9000` (port default to 9000)

API Documentation

Once you've to the server started check out the api documentation at /api-docs

Navigating the code base

• The entry point of the application is the server/index.js
• The server/app.js imports the APIs from server/api/index.js
• All the different APIs in the server/api are registered here
• MongoDB is used to store data & mongoose is used as the ORM
  • mongo
  • models
• The template has support for the following middlewares
  • auth
  • injectRequestId
  • rateLimiter
• The template has inbuilt support for
  • redis
  • circuitBreakers
  • slack alerts
  • docker
  • docker-compose
  • auto generated apis
  • sharding of collections
  • aggregate caching

Philosophy

When using NoSQLs you are optimising for read performance. We're doing this by denormalising data. There are multiple copies of the same data. For example

• Orders contains purchasedProducts which contains Products. Instead of referencing here we embed
• SupplierProducts contains embedded objects for both Suppliers and Products
• StoreProducts contains embedded objects for both Stores and Products

This makes our application write heavy. Every time there is a change to a product we need to make a change to

• SupplierProducts
• StoreProducts
• Products

Orders is not impacted since a change in the product after purchase will not affect the order.

However the application is able to perform extremely fast reads. 2 reasons for better performance is

• shards
• document embedding

NoSQLs are also good for handling large volumes of data. This is supported due to its ability to have shards. In this application we create 4 shards and the data is distributed amongst these shards.

These are the shard keys that we use

• _id
  • Order
• name
  • Products
  • Suppliers
  • Stores
    We got really good distribution across shards(24-26%) per shard after seeding 4 million records. It's possible to get a hot shard due to this but we're yet to see that.
• productId
  • SupplierProducts
  • StoreProducts
    productId is chosen as the shard key since we anticipate that the queries for fetching all suppliers/stores that sell a particular product will be much more than fetching all products of a supplier/store.