Just a bunch of useful links
NOTE: Data Engineering / ML / DS / Data Structures is now a separate section.
Just a bunch of useful links. BTW see rust links as well.
Table of Contents generated with DocToc
Two fast ways to get started playing with JDK/Java/Scala:
Good resources for starting out:
Great general Scala knowledge:
Visual Scala Reference - all the Scala collections methods with pictures diagramming their usage
Implicit Design Patterns in Scala - great overview from LiHaoYi on all the different uses of implicits
The Human Side of Scala - great post on styling Scala for readability
Sneaking Scala Through the Back Door - how to promote Scala in an organization
Strategic Scala Style: Principle of Least Power - an excellent read about how to choose the path of less complexity and higher readability in the huge landscape that is Scala programming
SBT - a declarative DSL - an excellent guide to SBT tasks and settings
testOnly a.b.c.TestName -- -z "word1 word2"
- where word1 word2 is a phrase from test description. Use double quotes.Between Zero & Hero - tips and tricks for the intermediate Scala developer
Functors, Applicatives, and Monads in Pictures - great guide to three potentially confusing terms in FP
Better Type Classes - also see one of first links for good intro to type classes
Type classes and generic derivation - How to avoid common boilerplate for type classes and case classes using Shapeless HLists
Type of Types - an unfinished tutorial on the Scala type system
Monads are not Metaphors - a great explanation of monads
Selfless Trait Pattern - allow users to either mix in a Trait or import an Object.
Tagged Types - great post from Scalac blog
Scalacaster - classic data structures in Scala
Reftree - Automatic object tree diagrams for immutable data
Important compiler flags
Recursive Types - signatures like
class Foo[T <: Foo[T]]
, useful for inheritance and proper return types. Tho if you hit this, there are
probably better ways of solving the problem, ie via composition.
Preprocessor - combination of different Scala Types like Phantom Types, Recursive Types, Self Types to make pipeline of computation in typesafe manner
ScalaFix - a tool to rewrite 2.x Scala for the new Dotty compiler
MDoc - type checking for ScalaDoc
ScalaMeta and Macro Annotations - a much more compact way of defining macros
Scalamacros - this is the future for Scala Macros, 2.12 and later, support for Dotty, etc. Coming end of 2017.
Scala Native - compile Scala to LLVM native code! :)
A separate section with notes on Specialization, Boxing, Inlining, Hi-Perf Scala
Java, not Scala
Simple Binary Encoding - supposedly 20-50x faster than Google Protobuf !!
Comparison of Cap'n Proto, SBE, FlatBuffers from the Cap'n Proto people
Sidney - an experimental columnar nested struct serializer, with Parquet-like repetition counts
Boon ByteBuf and the JavaDoc - a very easy to use, auto-growable ByteBuffer replacement, good for efficient IO
Fast-Serialization - a drop in replacement for Java Serialization but much faster
Reed-Solomon Erasure Coding Library from Backblaze. Recover or repair from missing chunks of data; a potential alternative to replication
REPL as a service - would be kick ass if integrated into Spark
Enumeratum - a Scala Enum library, much better than built in Enumeration
Quicklens - modify deeply nested fields in case classes
Ammonite - Scala DSL for easy BASH-like filesystem operations
Better-Files - Great library for easy file operations / NIO utils
IScala - Scala backend for IPython. Looks promising. There is also Scala Notebook but it's more of a research project.
Scaposer - i18n / .po file library
Adding Reflection to Scala Macros - example of using reflection in an annotation macro to add automatic ByteBuffer serialization to case classes :)
Scaldi - A lightweight dependency injection library, with Akka integration
lamma.io - the easiest date generation library
Pimpathon - a set of useful pimp-my-library extensions
Scala-rainbow - super simple terminal color output, easier than Console.XXX
Run Scala scripts with dependencies - ie you don't need a project file
sbt-assembly 0.10.2 supports adding a shell script to your jar to make it executable! No more "java ...." to start your Scala program, and no more ps ax | grep java | grep ....
acyclic - a Compiler plugin to detect cylical dependencies between source files. Eliminate them for faster builds!
Splain - a compiler plugin for better more descriptive error messages!
sbt-view - an SBT plugin to make it easy to view JavaDoc/ScalaDoc of dependencies or your own project
sbt-alldocs - SBT plugin (1.0.0+) to generate docs for ALL dependent jars!
Other useful SBT plugins - sbt-sonatype, sbt-pom-reader, sbt-sound, plugins page
Jitpack - jar packaging for Github repos with no jar publishing (or for non-released versions)
sbt-big-project - a plugin to speed up compilation when there are hundreds of projects
Coursier - a much improved jar dependency fetcher, written in pure Scala. Has SBT plugin, programmatic API
SCoverage - statement coverage tool, much more useful than line-based or branch-based tools. Has SBT plugin. Blog post on why it's an improvement.
sbt-jmh - Plugin for running SBT projects with the JMH microbench profiling tool. Also see jmh-profilers project.
Profiling JVM Applications - a great guide to FlameGraphs and other tools
jmh-visualizer for visualizing JMH result runs
sbt-jol - inspect Scala/Java object memory layout
Airframe Surface - a great small library to determine type and class param info
Clouseau - SBT and runtime plugin to get size of object graphs
Comcast - a tool to inject network latency, and less-severe issues
CharybdeFS - FUSE layer to inject filesystem faults for testing
Blindsight - structured logging library for Scala
Adaptive microbenchmarking of big data - really neat JVM agent which allows turning benchmarking code on and off for better benchmarking
SBT updates - Tool for discovering updated versions of SBT dependencies
Twitter Iago - Perf load test tool based on replaying logs. Compare vs Gatling for example.
Thyme and Parsley - microbenchmarking and profiling tools, seems useful
ScalaStyle - Scala style checker / linter
Towards a Safer Scala - great talk/slides on tools for Scala linting and static analysis
utest - a small micro test framework
lions share - a neat JVM heap and GC analysis tool, with charts and SBT integration.
SBT Alternatives
There are multiple paradigms for concurrency in Scala and it is vital to be familiar with all of them. Java-style low-level shared-mutable-state concurrency won't be covered here much -- we focus on the Scala paradigms that produce safe, easy-to-use concurrency.
Scala has built-in Futures which are good for eager, memoized, asynchronous, one-shot computations. They run in a thread pool and their results are stored in the Future object for multiple consumers. The really nice thing about Futures is that they are strongly typed and composable - you can use map
, flatMap
, filter
etc. to easily chain together asynchronous computation. Many libraries supplement the built-in functionality, and most Scala database libs rely on Future to deliver non-blocking, async I/O.
There are alternatives which offer lazy, non-memoized versions.
Akka is one of the most famous Scala libraries and where Scala Futures came from. It is known for Actors, a paradigm for always-running distributed resilient code popularized by the Erlang language in the 70's. Actors are great for safe stateful and distributed computation, based on a shared-nothing, message passing paradigm.
One of the coolest things built on top of Akka actors, which support remote messaging, is Akka Cluster. There is a separate section for that.
Kamon - great looking Actor monitoring using bytecode weaving? no code change required.
akka-tracing - A distributed tracing Akka extension based on Twitter's Zipkin, which can be used as performance diagnostics and debugging tool. Supports Spray!
DI in Akka - great guide to using MacWire with Akka for DI
Actor Provisioning pattern - if you have a long, failure-prone initialization procedure for an actor, this trait splits out the work, to say another actor and dispatcher
Akka mock scheduler - great for testing!
Akka VisualMailbox - Akka traffic patterns visualized in D3
Reactive Visualization for Akka streams!!
Chymyst - Chymyst is a framework for "chemical reactions" on top of Akka - for distributed, concurrent, functional, declarative computing, much higher level than Akka but built on Akka. Kind of a neat paradigm, check it out.
Ask, Tell, and Per-Request Actors - why one company moved from Ask/Futures to per-request
Dos and Donts deploying Akka in Production - an excellent read, full of advice even for non-Akka JVM apps
Understanding Akka's Quarantine State - another great blog post on how quarantine happens and how to avoid it
Akka Anti-Patterns: Hardware - and many other related good posts on anti-patterns
Akka Typed Protocols - Patrik's blog series on Akka typed - a more functional, type safe actors API :)
What if you want to stream multiple values and use up multiple threads or async I/O? Futures are great only for one-shot or single value. This is where reactive streams comes in - a standard for asynchronous streaming computation.
Here is a great intro to reactive streams covering why, why backpressure, how it compares to other paradigms.
The best Scala API for pure reactive streams I have found is Monix Observables. It is lightweight, designed for performance, and have a high degree of control over concurrency. There is also Akka Streams which is built on actors.
Parallelism in Monix - a great tutorial
Akka Streams Extensions - helpers, connectors with PostGres, and more.
Zoom - reactive programming with ZK, in Scala using ReactiveX
Other possiblities:
Non-Scala:
If you are building a distributed system, you should seriously consider using Akka Cluster.
Intro to Akka Distributed Data - definitely one of the coolest Akka cluster modules, has great potential for distributed system state.
Akka Data Replication - replicated low-latency in memory datastore built using Akka cluster and CRDTs
Akka Management & Discovery - Lightbend's Cluster Extension for auto cluster self discovery using DNS, replaces
Akka Cluster Inventory extension - very useful. All the other blog posts in the series are also excellent reads.
Akka ZK cluster seed - another Akka extension to automatically register seed nodes with ZK
Running an Akka cluster with Docker Containers
New Adaptive Failure Detector for Akka Cluster. Awesome research and hints too about massive clusters.
Other non-Akka (and some non-Scala) distribution libs:
Colossus - an extremely fast, NIO and Akka-based microservice framework. Read their blog post.
Socko and Xitrum - Two very fast web frameworks built on Akka and Netty
Quick Start to Twitter Finagle - though one should really look into Finatra
Airframe - whole bunch of building blocks for Scala apps including server/HTTP/metrics/logging
Scalaj-http - really simple REST client API. Although, the latest Spray-client has been vastly simplified as well.
Sort of orthogonal to small vs big, but more query language related:
Option
Scala-offheap - fast, safe off heap objects
Using Unsafe for C-like memory access speeds - a great guide. Many Unsafe operations turn into Java intrinsics - which translate to direct machine code
FastTuple - a dynamic (runtime-defined) C-style struct library, with support for off-heap storage. Only works for primitives right now :(
ObjectLayout - efficient struct-within-array data structures
jvm-unsafe-utils - @rxin of Spark/Shark fame library for working with Unsafe.
Agrona and blog post - a ByteBuffer wrapper, off-heap, with atomic / thread-safe update operations. Good for building off heap data structures.
Byte-buddy, a Java class generation library
OHC - Java off-heap cache
LWJGL - Potentially useful: very fast off heap memory allocators without limitations of allocateDirect; OpenCL library
jnr-ffi - Java Foreign Function Interface, used by JRuby to provide MUCH simpler interface to C code than JNI. Has native memory allocators and utilities for Struct types.
Breeze, Spire, and Saddle - Scala numeric libraries
Framian - a new data frame implementation from the authors of Spire
Scala DataTable - An immutable, updatable table with heterogenous types of columns. Easily add columns or rows, and have easy Scala collection APIs for iteration.
Squants - The Scala API for Quantities, Units of Measure and Dimensional Analysis
Airframe Metrics - human-readable representations of time, data byte size, etc.
An immutable priority map for Scala
product-collections - useful library for working with collections of tuples. Also, great strongly-typed CSV parser.
SuperFastHash - also see Murmur3
LZ4-Java - very fast compression, but also has version of XXHash - much faster than even Murmur3
SmoothieMap2 - a low-memory implementation of Google SwissTable for the JVM
bloom-filter-scala - and accompanying blog post explaining why it's the fastest bloom filter in the JVM
Moment Sketches - moment-based quantile sketches for summarizing quantile/histogram/latency data
Histogrammar - library for creating and plotting histograms
Relevant: Histogram - the Ultimate Guide of Binning
Asyncpools - Akka-based async connection pool for Slick. Akka 2.2 / Scala 2.10.
Postgresql-Async - Netty-based async drivers for PostgreSQL and MySQL
Relate - a very lightweight, fast Scala wrapper on top of JDBC
GeoTrellis - distributed raster processing on Spark. Also see GeoMesa - distributed vector database + feature filtering
ApertureTiles - system using Spark to generate a tile pyramid for interactive analytical geo exploration
Twofishes - Foursquare's Scala-based coarse forward and reverse geocoder
SFCurve - a Scala spatial curve library from the excellent folks at LocationTech
trails - parser combinators for graph traversal. Supports Tinker/Blueprints/Neo4j APIs.
scala-graph - in-memory graph API based on scala collections. Work in progress.
Phantom - Scala DSL for Cassandra, supports CQL3 collections, CQL generation from data models, async API based on Datastax driver. A bit heavyweight though.
Troy - A lightweight type safe wrapper around CQL/Cassandra client. Focused on CQL type safety.
Athena - Asynchronous Cassandra client built on Akka-IO
CCM - easily build local Cassandra clusters for testing!
Stubbed Cassandra - super useful for testing C* apps
Pithos - an S3-API-compatible object store for Cassandra
Sirius - Akka-based in-memory fast key-value store for JVM objects, with Paxos consistency, persistence/txn logs, HA recovery
Ivory - An immutable, versioned, RDF-triple / fact store for feature extraction / machine learning
Hibari - ordered key-value store using chain replicaton for strong consistency
Storehaus - Twitter's key-value wrapper around Redis, MySql, and other stores. Has a neat merge() functionality for aggregation of values, lists, etc.
ArDB - like Redis, but with spatial indexes, and pluggable storage engines
MapDB - Not a database, but rather a database engine with tunable consistency / ACIDness; support for off-heap memory; fast performance; indexing and other features.
HPaste - a nice Scala client for HBase
OctopusDB paper - interesting idea of using a WAL of RDF triples as the primary storage, with secondary views of row or column orientation
A separate section with notes on Specialization, Boxing, Inlining, Hi-Perf Scala
Norman Maurer's Talk on JVM Performance Tips - definitely worth a read!
JVM Intrinsics/Options list - super useful
jvm-mon - command line tool for JVM monitoring
HotSwapAgent - dynamic reload for standard Java frameworks, needs some plugin
Swiss Java Knife - super handy collection of JVM tools. Try java -jar sjk.jar ttop -p PID -o CPU -n 10
for regular reporting of the top 10 threads by CPU usage!
Hybrid JVM Perf Profiling - using AGTC and perf on Linux
Al's Guide to Cassandra 2.1 Ops - awesome, not just for C* but tools in general
Al Tobey's flags for running JDK8 apps. Note: G1GC! Also no need for MaxPermSize
anymore:
-Xmx8G -Xms8G -Xss256k -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=0
Tuning Spark apps for GC - excellent write-up from Intel
Perils of writing isolating classloaders - Good read, tips on how to write a classloader that can isolate and load different versions of the same classes
Quick dumping your JVM heap using GDB -- too bad it doesn't work on OSX.
Start a JMX agent in running JVM: jcmd <pid> ManagementAgent.start jmxremote.port=26010 jmxremote.ssl=false jmxremote.authenticate=false
HeapAudit - A Java agent for lightweight production heap profiling
Lion's Share - tools for memory analysis, outputs Google Charts compatible output
jHiccup -- "Hiccup" or GC pause analysis tool
Bintray - friendlier alternative to Sonatype OSS / Maven central. Also see bintray-sbt plugin.
Changing JVM flags live - such as enabling GC logging without restarting JVM. Cool!
I love Sublime and use it for everything, even Scala! Going to put my Sublime stuff in a separate page.
Semver - Semantic versioning, how to deal with dev workflows and corner cases -- a must read
Pragmatic RESTful API Design - really good stuff
Blameless Post-Mortems - why they are crucial to good culture
How to Pair with Jr Devs - really good advice. Make them type. Listen and be on the same level.
GitHub Flow - how github.com does continuous deploys, uses pull requests for an automated, process-free development workflow. Some gems include naming branches descriptively and using github.com to browse the work currently in progress by looking at active branches.
On Naming a Database System - pretty awesome.
Awesome public datasets - no doubt some are Socrata sites!
Mermaid - think of it as Markdown for diagrams. Now integrated with remark.js AND GITHUB!!
Asciinema - record your terminal sessions!
Monodraw - diagrams as ASCII. Not free.
Markdeep - Markdown++ with diagrams, add single line at bottom to convert to HTML!
How To Be a Great Developer - a reminder to be empathetic, humble, and make lives around us better. Awesome list.
JQ - JSON processor for the shell. Super useful with RESTful servers.
Hub - CLI for Github :)
Git WebUI - easy to install web UI for visualizing git history, different branches.
Underscore-CLI - a Node-JS based command line JSON parser
MacroPy - Scala-like macros, case classes, pattern matching, parser combos for Python (!!)
Scala 2.11 vs Swift - Apple's new iOS language is often compared to Scala.
Gherkin - a Lisp implemented in bash !!
Futhark - "High-performance purely functional data-parallel array programming on the GPU" - a language for efficient GPU computation
Nimrod - a neat, compile-straight-to-binary, static systems language with beautiful Python-like syntax, union types, generics, macros, first-class functions. What Go should have been.
Pony - A capabilities-based Actor-centric static language, deadlock-free, null-free, data-race-free!
ROC - A statically-typed functional language, with automatic memory management, which aims to be faster than Go and Swift
Bret Victor - A set of excellent essays and talks from a great visual designer
George Mack's Razors - a compilation of razors to make decisions by
becoz it's so darn painful.
sudo chmod -s /usr/sbin/dtrace
(see this Homebrew issue)