Awesome Python Testing

Collection of awesome Python resources for testing and generating test data.

Contents

• Assertions
• Behavior-driven Development
• Code Coverage
• Design by Contract
• Fake Data
• Load Testing
• Memory Management
• Mock and Stub
• Mutation Testing
• Object Factories
• Penetration Testing
• Property Based Testing
• Rest API Testing
• Snapshot Tests
• Speed
• Static Checks
• Test Runners
• Testing Frameworks
• Tools
• UI Testing
• Resources

Assertions

• believe - A python package for json/dictionary validation.
• easycheck - A collection of assertion-like functions to be used in code, where assertion themselves should be avoided; easycheck includes also function aliases to be used in unit testing.
• expects - Expects is an expressive and extensible TDD/BDD assertion library for Python.
• expycted - Another Python expect pattern implementation. Simple, intuitive and approachable, with ability to plug in to any testing framework that relies on assertions.
• dirty-equals - A python library that (mis)uses the __eq__ method to make python code (generally unit tests) more declarative and therefore easier to read and write.
• Precisely - Write precise assertions so you only test the behaviour you're really interested in.
• PyHamcrest - A framework for writing matcher objects, allowing you to declaratively define "match" rules.
• pytest_cache_assert - Cache assertion data to simplify regression testing of complex serializable data.
• sure - An idiomatic assertion toolkit with human-friendly failure messages, inspired by RSpec Expectations and should.js.

Behavior-driven Development

• behave - is behavior-driven development, Python style.
• lettuce - Behavior-driven-development tool for python, inspired by Cucumber for Ruby.
• mamba - The definitive testing tool for Python. Born under the banner of BDD.

Code Coverage

• Coverage.py - is a tool for measuring code coverage of Python programs.
• coverage-conditional-plugin - Conditional coverage based on any rules you define.
• diff_cover - Automatically find diff lines that need test coverage.

Design by Contract

• deal - Design by contract for Python with static checker and test generation.
• icontract - Design-by-contract in Python3 with informative violation messages and inheritance.
• pact-python - Python version of Pact. Enables consumer driven contract testing, providing a mock service and DSL for the consumer project, and interaction playback and verification for the service provider project.

Fake Data

• autofaker - designed to minimize the setup/arrange phase of your unit tests by removing the need to manually write code to create anonymous variables as part of a test cases setup/arrange phase.
• faker - A Python package that generates fake data.
• fake2db - Fake database generator.
• genuine-fake - Genuine Fake means an imitation of a (usually) valuable object that is so good that it is, to all intents and purposes, identical.
• mimesis - A Python library that helps you generate fake data.
• radar - Generate random datetime / time.

Load Testing

• Dynamic Workload Model - Code to generate dynamic workload model. Useful for testing autoscaling in cloud or mimicking different load profile for different scenario.
• Grasshopper - A lightweight framework for performing load tests against an environment, primarily against an API. Grasshopper glues Locust, Pytest, some plugins (namely Locust InfluxDBListener ) and some custom code to provide a package that makes authoring load tests simple with very little boilerplate needed.
• Grizzly - is a framework to be able to easily define load scenarios, and is mainly built on-top of Locust and Behave.
• Locust - Scalable user load testing tool written in Python.
• pynonymizer - is a universal tool for translating sensitive production database dumps into anonymized copies.

Memory Management

• filprofiler - A Python memory profiler for data processing and scientific computing applications.
• Guppy 3 - a Python programming environment & heap analysis toolset.
• mem_top - shows top suspects for memory leaks in your Python program.
• Pympler - is a development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.
• tracemalloc - is a debug tool to trace memory blocks allocated by Python.

Mock and Stub

• Aioresponses - is a helper for mock/fake web requests in python aiohttp package.
• Cornell - record & replay mock server.
• doublex - Powerful test doubles framework for Python.
• Flexmock - is a testing library for Python that makes it easy to create mocks, stubs and fakes.
• freezegun - Travel through time by mocking the datetime module.
• httmock - A mocking library for requests for Python 2.6+ and 3.2+.
• httpretty - HTTP request mock tool for Python.
• Kesha - A web service with a user interface for testing http requests and web hooks.
• mock - (Python standard library) A mocking and patching library.
• mocket - A socket mock framework with gevent/asyncio/SSL support.
• Mockintosh - aims to provide usual HTTP mock service functionality with small resource footprint, making it friendly for microservice applications.
• moto - allows you to easily mock out tests based on AWS infrastructure.
• Pretend - is a library to make stubbing with Python easier.
• responses - A utility library for mocking out the requests Python library.
• time-machine - Travel through time in your tests.
• trustme - gives you a fake certificate authority (CA) that you can use to generate fake TLS certs to use in your tests.

Mutation Testing

• Cosmic Ray - makes small changes to your source code, running your test suite for each one.
• Mutatest - Python mutation testing.
• Mutmut - is a mutation testing system for Python, with a strong focus on ease of use.
• MutPy - MutPy is a mutation testing tool for Python 3.x source code
• xmutant.py - Python 3.6 bytecode based mutation analysis framework

Object Factories

• factory_boy - A test fixtures replacement for Python.
• mixer - Another fixtures replacement. Supports Django, Flask, SQLAlchemy, Peewee and etc.
• Model Bakery - offers you a smart way to create fixtures for testing in Django.
• polyfactory - A simple and powerful mock data generation library, based around type hints and supporting dataclasses, typed-dicts, pydantic models, msgspec structs and more.

Penetration Testing

• fsociety - A Modular Penetration Testing Framework.
• python-pentest-tools - Python tools for penetration testers.
• fencer - Fencer is an automated API security testing tool. It's an experimental project to see how much of the API security testing process can be automated.

Property Based Testing

• Atheris - is a coverage-guided Python fuzzing engine. It supports fuzzing of Python code, but also native extensions written for CPython.
• hypofuzz - Adaptive fuzzing of Hypothesis tests.
• Hypothesis - is an advanced Quickcheck style property based testing library.
• hypothesis-auto - An extensions for Hypothesis that provides fully automatic testing for type hinted functions.
• hypothesis-crosshair - Add the power of solver-based symbolic execution to your Hypothesis tests with CrossHair.
• icontract-hypothesis - Instead of writing manually the Hypothesis search strategies for a function, icontract-hypothesis infers them based on the function's precondition.

Rest API Testing

• behave-restful - BDD Framework to Test REST Services and APIs.
• cherrybomb - CLI tool that helps you avoid undefined user behaviour by auditing your API specifications, validating them and running API security tests.
• Dredd - is a language-agnostic command-line tool for validating API description document against backend implementation of the API.
• HttpRunner - is a simple & elegant, yet powerful HTTP(S) testing framework.
• hypothesis-graphql - Hypothesis strategies for GraphQL schemas, queries and data.
• Schemathesis - A tool for automatic property-based testing of web applications built with Open API / Swagger specifications.
• playback - A Python decorator-based framework that lets you "record" and "replay" operations (e.g. API requests, workers consuming jobs from queues).
• RESTler - is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
• Tavern - is a pytest plugin, command-line tool and Python library for automated testing of APIs, with a simple, concise and flexible YAML-based syntax.

Snapshot Tests

• ApprovalTests - work by comparing the test results to a golden master.
• SnapshotTest - is a way to test your APIs without writing actual test cases.
• syrupy - is a zero-dependency pytest snapshot plugin. It enables developers to write tests which assert immutability of computed results.
• VCR.py - Record and replay HTTP interactions on your tests.

Speed

• Partial Testing - Run only the tests that are relevant for your changes.
• Pytest-incremental - analyses your project structure and file modifications between test-runs to modify the order tests are executed and de-select tests.
• pytest-testmon - Selects tests affected by changed files. Continuous test runner when used with pytest-watch.
• Awesome pytest speedup - A checklist of best practices to speed up your pytest suite.

Static Checks

• awesome-flake8-extensions - A curated list of awesome flake8 extensions.
• awesome-python-typing - Collection of awesome Python types, stubs, plugins, and tools to work with them.
• Bandit - is a tool designed to find common security issues in Python code.
• flake8 - is a python tool that glues together pep8, pyflakes, mccabe, and third-party plugins to check the style and quality of python code.
• pyanalyze - A tool for programmatically detecting common mistakes in Python code, such as references to undefined variables and some categories of type mismatches.
• pyflakes - A simple program which checks Python source files for errors.
• Pylint - A Python static code analysis tool which looks for programming errors, helps enforcing a coding standard, sniffs for code smells and offers simple refactoring suggestions.
• Refurb - A tool for refurbishing and modernizing Python codebases.
• ruff - An extremely fast Python linter, written in Rust.
• slotscheck - Find mistakes in your __slots__ definitions.
• Typecheckers - A list of Python type checkers.

Test Runners

• green - A clean, colorful test runner.
• Nox - is a command-line tool that automates testing in multiple Python environments, similar to tox. Unlike tox, Nox uses a standard Python file for configuration.
• tox - Auto builds and tests distributions in multiple Python versions.

Testing Frameworks

• async-asgi-testclient - A framework-agnostic library for testing ASGI web applications.
• awesome-pytest - A curated list of awesome pytest resources.
• awesome-robotframework - A curated list of awesome Robot Framework resources and libraries.
• doctest - (Python standard library) The doctest module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown.
• hammett - Fast python test runner, compatible with a subset of pytest.
• nose2 - The successor to nose, based on unittest2.
• perftester - A lightweight framework for performance testing of Python functions; allows for testing of performance in terms of execution time and memory usage.
• promptimize - a prompt engineering evaluation and testing toolkit. It accelerates and provides structure around prompt engineering at scale with confidence, bringing some of the ideas behind test-driven development (TDD) to engineering prompts.
• pytest - A mature full-featured Python testing tool.
• sundew - Sundew is a testing framework for Python, implementing a new approach to testing. One that combines functional programming concepts and the general best practices for writing tests that we already know, and enforces them in a way that enables some really powerful features that make testing easier, enjoyable, and more effective.
• Robot Framework - A generic test automation framework.
• testbook - A unit testing framework extension for testing code in Jupyter Notebooks.
• unittest - (Python standard library) Unit testing framework.
• vedro - Pragmatic testing framework for Python
• Ward - is a modern test framework for Python with a focus on productivity and readability.
• xdoctest - A rewrite of Python's builtin doctest module (with pytest plugin integration) with AST instead of REGEX.

Tools

• Auger - Project to automatically generate unit tests for Python code.
• CrossHair - An analysis tool for Python that blurs the line between testing and type systems.
• ghostwriter - Writing tests with Hypothesis frees you from the tedium of deciding on and writing out specific inputs to test.
• importlab - A library that automatically infers dependencies for Python files. Importlab's main use case is to work with static analysis tools that process one file at a time, ensuring that a file's dependencies are analysed before it is.
• Klara - Static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system.
• logot - Test whether your code is logging correctly.
• Pifpaf - A suite of fixtures and a command-line tool that allows to start and stop daemons for a quick throw-away usage. This is typically useful when needing these daemons to run integration testing.
• Pynguin - A tool that allows developers to generate unit tests automatically.
• pytest-codegen - Pytest-codgen will statically analyze your code to create pytest function stubs.
• pytestify - Automatically convert unittests to pytest.
• pytest-mock-generator - A pytest fixture wrapper for mock_autogen.
• teyit - A static analyzer and a refactoring tool to rewrite your unittest assertions in the right way.

UI Testing

• Flybirds - is a front-end UI automation test framework based on BDD mode, providing a series of out-of-the-box tools and complete documentation.
• Golem - is a test framework and a complete tool for browser automation. Tests can be written with code in Python, codeless using the web IDE, or both.
• helium - is great for web automation. Helium makes it easier to use.
• Lost Pixel - is an open source visual regression testing tool. Run visual regression tests on your Storybook and Ladle stories and on your application pages.
• LuluTest - is an open source browser automation framework using Python and Selenium.
• PyAutoGUI - is a cross-platform GUI automation Python module for human beings.
• pytest-ui-automatic - Playwright Python tool practice pytest pytest-bdd screen-play page-object allure cucumber-report.
• selene - User-oriented Web UI browser tests in Python (Selenide port).
• Selenium - Python bindings for Selenium WebDriver.
• SeleniumBase - is an all-in-one Python framework for automated browser testing. Tests are run with "pytest", and use WebDriver APIs for web-page interaction.
• sixpack - A language-agnostic A/B Testing framework.
• splinter - Open source tool for testing web applications.
• squape - is a package that extends Squish API providing convenience tools for everyday automated test cases development.

Resources

Articles

• Anna-Lena Popkes: "Mocking in Python" - Mocking might seem confusing in the beginning but once you understand the basics it can be very helpful.
• async test patterns for Pytest - Learn some handy async examples and patterns for testing in Pytest.
• Getting Started With Property-Based Testing in Python With Hypothesis and Pytest
• How not to footgun yourself when writing tests - a showcase of flaky tests
• Stargirl Flowers: “My Python testing style guide” - An attempt to catalog some practices around testing Python projects. It's not meant to be treated as dogma.
• Test & Code: Python Testing - Test & Code is a weekly podcast hosted by Brian Okken. The show covers a wide array of topics including software engineering, development, testing, Python programming, and many related topics.
• Testing your Python Code with Hypothesis - A look at how Hypothesis can help you discover errors in your code.
• Patterns of flakey Python tests - This post details several patterns that cause flakey Python tests. Being aware of these common causes can help when investigating your own flakey tests.
• Unit testing Python code in Jupyter notebooks - This article covers several options for unit testing Python code in a Jupyter notebook.
• 30 best practices for software development and testing - These software engineering rules and testing best practices might help save you time and headaches.

Books

• Architecture Patterns with Python (O'Reilly)
• Crafting Test-Driven Software with Python (Packt)
• pytest Quick Start Guide (Packt)
• Python Testing with pytest (Pragmatic Bookshelf)
• Python Testing with Selenium (Apress)
• Python Unit Test Automation (O'Reilly)
• Testing In Python (Independently published)
• Testing Python (Wiley)
• Test-Driven Development with Python (O'Reilly)

Videos

• PyCascades 2023 - Talk - Brian Okken: "Sharing is Caring - Sharing pytest Fixtures"
• PyCon US 2023 - Talk - Dan Craig: Testing Spacecraft with Pytest
• PyCon US 2023 - Talk - Dave Aronson: Kill All Mutants! (Intro to Mutation Testing)
• PyCon US 2023 - Talk - Shai Geva: 10 Ways To Shoot Yourself In The Foot With Tests
• PyCon US 2023 - Tutorial - Zac Hatfield-Dodds, Ryan Soklaski: Introduction to Property-Based Testing

Related

• Awesome Python - A curated list of awesome Python frameworks, libraries, software and resources.
• Python test automation - A comprehensive curated list of python test automation frameworks, tools, libraries and software to help software engineers easily bootstrap test automation on python.
• commit-check - Check commit message formatting, branch naming, commit author, email, and more.