DevOps methodology & roadmap for a devops developer in 2019. Interesting books to learn new technologies.
DevOps is a software development methodology that combines software development (Dev) with information technology operations (Ops) participating together in the entire service lifecycle, from design through the development process to production support.
Environment Stabilization
Enforces consistency, increase up-time
Shorter Development Cycle
Manage requirements and code-repository
Increased Release Velocity
Continuous build, push-button deployments
Reduced Defects
Regiment processes, automated testing
Process Metrics
Track both time at each stage, and the errors and exceptions
Step 1: Learn a language
Step 2: Learn about source code management (recommend using Git)
Step 3: Practice what you have learnt
Step 4: Learn Package Manager
Step 5: Learn about Unit Testing
Step 6: Learn about databases (RDBMS & NoSQL)
Step 7: Learn frameworks
Step 8: Learn how to implement caching
Step 9: Creating RESTful APIs
Step 10: Learn about different Authentication/Authorization methods
Step 11: Understand Message Brokers
Step 12: Learn how to use Docker
Step 13: Knowledge of Web Servers
Step 14: Learn how to test service
Step 15: Learn different protocols
Step 16: Learn Microservice, Event Driven, Lambda architectures
Step 17: Learn Big Data technologies/tools
Step 18: Learn algorithms
Reference: https://github.com/TheAlgorithms/Python
Step 19: Learn AI technologies
Step 20: Learn configuration/deployment services
Step 21: Learn how to monitor services
Step 22: Use open source tools
Mock REST APIs: https://github.com/typicode/json-server
UML Designer: https://www.modelio.org
Planning: https://wiki.gnome.org/Apps/Planner
MindMap: MindMaple Lite
UI Wireframe: Pencil (https://pencil.evolus.vn), Balsamiq
SSH clients for Windows: SuperPuTTY
Step 23: Keep exploring
A collection of LARGE Datasets, so it can NOT be Processed by traditional methods…
[Source]: https://topics.amcham.com.tw/wp-content/uploads/2016/03/BigData_2267x1146_white.png
[Source]: Apache Kafka
A flexible and scalable solution is to use a message broker or messaging system. Instead of applications connecting directly to each other, they connect to a message broker or a messaging system. This architecture makes it easy to add producers or consumers to a data pipeline.
Batch Processing
In batch processing, newly arriving data elements are collected into a group. The whole group is then processed at a future time (as a batch, hence the term “batch processing”). Exactly when each group is processed can be determined in a number of ways–for example, it can be based on a scheduled time interval (e.g. every five minutes, process whatever new data has been collected) or on some triggered condition (e.g. process the group as soon as it contains five data elements or as soon as it has more than 1MB of data).
Micro-Batch is frequently used to describe scenarios where batches are small and/or processed at small intervals. Even though processing may happen as often as once every few minutes, data is still processed a batch at a time.
Stream Processing
In stream processing, each new piece of data is processed when it arrives. Unlike batch processing, there is no waiting until the next batch processing interval and data is processed as individual pieces rather than being processed a batch at a time.
Use cases:
Batch Processing vs Streaming Processing
Batch Processing | Stream Processing | |
---|---|---|
Data Scope | Queries or processing over all or most of the data in the dataset | Queries or processing over data within a rolling time window, or on just the most recent data record |
Data Size | Large batches of data | Individual records or micro batches consisting of a few records |
Performance | Latencies in minutes to hours | Requires latency in the order of seconds or milliseconds |
Analyses | Complex analytics | Simple response functions, aggregates, and rolling metrics |
For more details please have a look at my other repo: https://github.com/raycad/stream-processing
Machine learning is creating and using models that are learned from data.
Machine learning referred to as predictive modeling or data mining.
Examples:
Data Science Fields
Best Seller in Computer Neural Networks on Amazon 2018: Hands-On Machine Learning with Scikit-Learn and TensorFlow https://github.com/raycad/devops-roadmap/blob/master/books/ai/ml/hands_on_machine_learning_with_scikit_learn_and_tensorflow.pdf
System Design Primer: https://github.com/raycad/devops-roadmap/blob/master/books/software-architecture/system_design_primer.pdf
Programming Collective Intelligence: https://github.com/raycad/devops-roadmap/blob/master/books/ai/ml/oreilly_programming_collective_intelligence_aug_2007.pdf
Azure Cloud Architecture: https://github.com/raycad/devops-roadmap/blob/master/books/software-architecture/azure_cloud_architecture.pdf
https://github.com/raycad/stream-processing
https://codeburst.io/the-2018-web-developer-roadmap-826b1b806e8d