Littlstar Hivemind Save

For creating distributed jobs using AWS Lambda functions

Project README

Hivemind

For creating distributed jobs using AWS Lambda functions.

Install:

npm install lambda-hivemind

Quick Example:

const Hivemind = require('lambda-hivemind')

const hivemind = new Hivemind({
  func: {
    name: 'buzzbuzz',
    handler: 'buzzbuzz.sting',  // '<name of file>.<exported func name>'
    role: '<arn of IAM role to run function under>'
  },
  accessKeyId: '', // AWS credentials. Optional.
  secretAccessKey: '' // AWS credentials. Optional.
  awsRegion: 'us-east-1',
  data: [
    'bee 1',
    'bee 2',
    'bee 3',
    ...
  ],
  chunkSize: 10  // Size of chunks to split data into
})

// Oh no, we haven't published a function of this name yet!

// List all files you want published as your function
hivemind.publish({
  files: [
    'buzzbuzz.js'
  ]
}, {
  // Any AWS parameters specified under http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#createFunction-property
})

hivemind.on('create', () => {
  // Ok, let's go!
  hivemind.run()
})

hivemind.on('finish', (results) => {
  // Each job's results will stream to here.
  console.log("Woah!")
})

hivemind.on('end', () => {
  // All jobs have finished!
  process.exit(0)
})

hivemind.on('error', (err) => {
  console.log("Aww, man!")
})

Methods

constructor(params)

Parameters:

  • func - Description of function
    • name - Name of the function
    • handler (optional) - Handler of function (If your file is named index and your function is named handler, your handler will be index.handler) (default: index.handler)
    • role - ARN of AWS Role to run function under
    • runtime (optional) - What framework to run function with (default: node6.10)
  • lambda (optional) - Overload of AWS Lambda class
  • data - Array of data to run on
  • chunkSize - How many pieces of data to give to each function
  • accessKeyId (optional) - AWS Access key
  • secretAccessKey (optional) - AWS Secret key
  • awsRegion - AWS region to execute your function in

publish(params, awsParams = {})

Publishes or updates specified code as a Lambda function. This method is not required if the function has been previously published and does not need to updated.

Parameters:

  • zipFile - Relative path to ZIP file to publish as function. This method is required if your script has dependencies. See Code.S3Key below if your ZIP is over 10MB.
  • files - Array of relative paths to files to publish
  • lambdaFunc - JavaScript function to publish as your function. The function must use take 3 parameters (event, context, callback). Data is loaded under event, callback is called when function is done.

AWS Parameters:

run()

Chunks and runs data given in constructor through the function specified in constructor and published in publish()

Notes:

AWS credentials in constructor are only optional if you have the AWS CLI configured or have your credentials as environment variables!

Full example can be found under test/ folder.

Open Source Agenda is not affiliated with "Littlstar Hivemind" Project. README Source: littlstar/hivemind
Stars
250
Open Issues
1
Last Commit
6 years ago
Repository
License
MIT

Open Source Agenda Badge

Open Source Agenda Rating