Globalstorage Save

Distributed Data Warehouse 🌍

Project README

GlobalStorage

The Concept

This is a distributed DBMS for technological stack Metarhia and it is built with following assumptions:

GS is designed to be built-in DBMS, to work inside Impress Applications Server; it is needed to avoid or minimize interprocess communication to access DB;
GS is compatible with JSTP JavaScript Transfer Protocol, all data slould be stored, stansmitted, handled and placed in RAM in the same format;
All data structures can be reduced to array representation to redice size by removing key names, we can do that with the help of metadata schemas and we can dynamicaly build prototypes from schemas and assign them to arrays, so getters/setters will convert access by name (hash) to assess by position (array);
Maximum memory usage, read-ahead and lazy-write, minimizing data conversion;
Using metadata everywhere, special declarative format for subject domein representation including fields, relations, and indices so we can automatically build a storage scheme in the relational database, memory structures and structure for the database, different the GUI, API server, etc.
The same API for client-side runtime and server-side runtime:
- server-side storage engine;
- client-side storage engine (multiple implementations for different platforms including mobile, desktop and browser);
- sharding for distributed storage of large data amounts, geo-distribution, save backup copies, access load balancing;
- allows user to exchange data in P2P mode;
Syncronization between client and server in realtime (close to realtime) and in lazy mode; so applications can work in online and offline (with locally stored data); having bidirectional data sync and hieratchical versioning like git have;
Global data structures unification for applications working with Metarhia technological stack: GlobalStorage, Impress, JSTP and Console through moderated distributed metadata repositories;
Ability to work with non-unified data structures (custom schemas), specific to certain subject domain;
GlobalStorage provides DAC (data access layer) abstraction, it substitutes ORM but it does not necessarily make maping on the relational model (though RDBMS are also supported);
Data structures have global distributed identification system, so data can be inserted anywhere and will not bring ID conflicts;
Data reliability is provided by distributed storage facilities, so each data structure should have master server and multiple backup and cache servers; using those servers GlobalStorage supports addressing, versioning and branching.

Metamodel Definition Language

Using this domain specific language we will describe subject domain in declarative format. To build GUI, API, business-loguic, data structures dynamically in runtime. For example we can build JavaScript prototype and assign it to positional array to access fields by name, so arrays will work like objects.

Example:

{
  code: { type: 'string', primary: true },
  name: {
    caption: 'City',
    type: 'string',
    size: 32,
    nullable: false,
    index: { unique: false },
    master: { dataset: 'Cities', key: 'name' }
  },
  birth: 'Date',
  city: 'string',
  addresses: {
    type: { array: 'Address' }
  },
  gender: {
    type: 'char',
    lookup: { dictionary: { M: 'Male', F: 'Female' } }
  },
  age: function() {
    var difference = new Date() - this.birth;
    return Math.floor(difference / 31536000000);
  }
}

Data types:

Built-in JavaScript types: string, number, boolean, Date, etc.
Global Storage types: id, uid, tree, ip, etc.
RDBMS data types: char, int, real, text, money, time, date...
Links to other data structures in GlobalStorage
Links to other data structures in Application

JavaScript Query Language

JSQL is a query language for data structures manipulation. JSQL have syntax for: filter, projection, dataset join and set operations. We have a separate repository for examples and specification: metarhia/JSQL. Current Implementation can be found in lib/transformations.js.

Contributors

See github for full contributors list

API

gs(provider, options)

provider: <string> provider name
options: <Object>
- serverSuffix: <Uint64> optional
- serverBitmask: <Uint64> optional
- systemSuffix: <Uint64> optional
- systemBitmas: <Uint64> optional

Create provider

gs.schemaConfig

<Object> metaschema config

class Cursor

Cursor.prototype.constructor(options)

Cursor.prototype.definition(schema, category)

schema: <Metaschema>
category: <string> schema name

Returns: <this>

Attach schema

Cursor.prototype.enableLogging(provider, ctx, args)

Cursor.prototype.copy()

Returns: <Cursor> new instance

Copy references to new dataset

Cursor.prototype.clone()

Returns: <Cursor> new instance

Clone all dataset objects

Cursor.prototype.enroll(jsql)

jsql: <Array> commands array

Returns: <this>

Apply JSQL commands to dataset

Cursor.prototype.empty()

Returns: <this>

Remove all instances from dataset

Cursor.prototype.from(arr)

arr: <Iterable>

Returns: <Cursor> new instance

Synchronous virtualization converts Array to Cursor

Cursor.prototype.map(fn)

Returns: <this>

Lazy map

fn - <Function>, map function

Cursor.prototype.projection(fields)

fields: <string[]>|<Object> projection metadata array of field names or object with structure: { toKey: [ fromKey, functions... ] }

Returns: <this>

Declarative lazy projection

Cursor.prototype.filter(fn)

fn: <Function> filtering function

Returns: <this>

Lazy functional filter

Cursor.prototype.select(query)

query: <Function> filtering expression

Returns: <Cursor> new instance

Declarative lazy filter

Cursor.prototype.distinct()

Returns: <this>

Lazy functional distinct filter

Cursor.prototype.sort(fn)

fn: <Function> comparing function

Returns: <this>

Lazy functional sort

Cursor.prototype.order(fields)

fields: <string>|<string[]>

Returns: <this>

Declarative lazy ascending sort

Cursor.prototype.desc(fields)

fields: <string>|<string[]>

Returns: <this>

Declarative lazy descending sort

Cursor.prototype.count([field])

field: <string> field to use for count, optional

Returns: <this>

Calculate count

Cursor.prototype.sum(field)

field: <string> field to use for sum

Returns: <this>

Calculate sum

Cursor.prototype.avg(field)

field: <string> field to use for avg

Returns: <this>

Calculate avg

Cursor.prototype.max(field)

field: <string> field to use for max

Returns: <this>

Calculate max

Cursor.prototype.min(field)

field: <string> field to use for min

Returns: <this>

Calculate min

Cursor.prototype.col()

Returns: <this>

Convert first column of dataset to Array

Cursor.prototype.row()

Returns: <this>

Return first row from dataset

Cursor.prototype.one()

Returns: <this>

Get single first record from dataset

Cursor.prototype.limit(count)

count: <number>

Returns: <this>

Get first n records from dataset

Cursor.prototype.offset(offset)

offset: <number>

Returns: <this>

Offset into the dataset

Cursor.prototype.union(cursor)

cursor: <Cursor>

Returns: <this>

Calculate union and put results to this Cursor instance

Cursor.prototype.intersection(cursor)

cursor: <Cursor>

Returns: <this>

Calculate intersection and put results to this Cursor instance

Cursor.prototype.difference(cursor)

cursor: <Cursor>

Returns: <this>

Calculate difference and put results to this Cursor instance

Cursor.prototype.complement(cursor)

cursor: <Cursor>

Returns: <this>

Calculate complement and put results to this Cursor instance

Cursor.prototype.selectToMemory(query)

async Cursor.prototype.continue(data)

data: <Array> rows to date

Globalstorage Save

GlobalStorage

The Concept

Metamodel Definition Language

JavaScript Query Language

Contributors

API

gs(provider, options)

gs.schemaConfig

class Cursor

Cursor.prototype.constructor(options)

Cursor.prototype.definition(schema, category)

Cursor.prototype.enableLogging(provider, ctx, args)

Cursor.prototype.copy()

Cursor.prototype.clone()

Cursor.prototype.enroll(jsql)

Cursor.prototype.empty()

Cursor.prototype.from(arr)

Cursor.prototype.map(fn)

Cursor.prototype.projection(fields)

Cursor.prototype.filter(fn)

Cursor.prototype.select(query)

Cursor.prototype.distinct()

Cursor.prototype.sort(fn)

Cursor.prototype.order(fields)

Cursor.prototype.desc(fields)

Cursor.prototype.count([field])

Cursor.prototype.sum(field)

Cursor.prototype.avg(field)

Cursor.prototype.max(field)

Cursor.prototype.min(field)

Cursor.prototype.col()

Cursor.prototype.row()

Cursor.prototype.one()

Cursor.prototype.limit(count)

Cursor.prototype.offset(offset)

Cursor.prototype.union(cursor)

Cursor.prototype.intersection(cursor)

Cursor.prototype.difference(cursor)

Cursor.prototype.complement(cursor)

Cursor.prototype.selectToMemory(query)

async Cursor.prototype.continue(data)

async Cursor.prototype.fetch([permissionChecker])

class StorageProvider

StorageProvider.prototype.constructor(options)

async StorageProvider.prototype.open(options)

async StorageProvider.prototype.close()

async StorageProvider.prototype.setup(options)

StorageProvider.prototype.enableLogging(ctx)

StorageProvider.prototype.error(name, ...ctx)

async StorageProvider.prototype.takeId()

async StorageProvider.prototype.get(id[, permissionChecker])

async StorageProvider.prototype.getDetails(category, id, fieldName[, permissionChecker])

async StorageProvider.prototype.set(obj[, permissionChecker])

async StorageProvider.prototype.create(category, obj[, permissionChecker])

async StorageProvider.prototype.update(category, query, patch[, permissionChecker])

async StorageProvider.prototype.delete(category, query[, permissionChecker])

async StorageProvider.prototype.linkDetails(category, field, fromId, toIds[, permissionChecker])

async StorageProvider.prototype.unlinkDetails(category, field, fromId, toIds[, permissionChecker])

StorageProvider.prototype.select(category, query[, permissionChecker])

async StorageProvider.prototype.execute(category, action, actionArgs[, permissionChecker])

StorageProvider.prototype.log(op, args, ctx, response)

StorageProvider.prototype.getSystemSuffix(id)

StorageProvider.prototype.curSystem(id)

StorageProvider.prototype.getServerSuffix(id)

StorageProvider.prototype.curServer(id)

StorageProvider.prototype.getLocalId(id)

StorageProvider.prototype.parseId(id)

async StorageProvider.prototype.listApplications([filtererByRoles])

async StorageProvider.prototype.listCategories([filtererByPermission])

async StorageProvider.prototype.listActions([filtererByPermission])

StorageProvider.prototype.createJstpApi()

async StorageProvider.prototype.getSchemaSources({ categoryList, actionLists, appList } = {})

async StorageProvider.prototype.getCategoryL10n(langTag, category)

async StorageProvider.prototype.getDomainsL10n(langTag)

async StorageProvider.prototype.getCommonL10n(langTag)

async StorageProvider.prototype.getFormL10n(langTag, category, form)

async StorageProvider.prototype.getActionL10n(langTag, category, action)

class FsProvider extends StorageProvider

FsProvider.prototype.constructor(options // { path } where path is database location)