GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing python.libhunt.com — Python Testing Crawler Alternatives - Python Testing | LibHunt

Python Testing Crawler

A crawler for automated functional testing of a web application

github.com Source Code Docs Changelog

Suggest Changes

Popularity

1.8

Growing

Activity

0.0

Stable

Stars 75

Watchers 3

Forks 5

Last Commit over 3 years ago

Description

Python Testing Crawler is a crawler for automated functional testing of a web application

Crawling a server-side-rendered web application is a low cost way to get low quality test coverage of your JavaScript-light web application.

If you have only partial test coverage of your routes, but still want to protect against silly mistakes, then this is for you.

Features:

* Selectively spider pages and resources, or just request them * Submit forms, and control what values to send * Ignore links by source using CSS selectors * Fail fast or collect many errors * Configurable using straightforward rules

Works with the test clients for Flask (inc Flask-WebTest), Django and WebTest.

Programming language: Python

License: Mozilla Public License 2.0

Tags: Django Flask Web Crawling Testing Web Utilities Dynamic Content

Python Testing Crawler alternatives and similar packages

Based on the "Testing" category.
Alternatively, view Python Testing Crawler alternatives based on common mentions on social networks and blogs.

Selenium

9.8 9.9 L2 Python Testing Crawler VS Selenium

A browser automation framework and ecosystem.
locust

9.5 9.4 L3 Python Testing Crawler VS locust

Write scalable load tests in plain Python 🚗💨

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

Promo www.saashub.com

pytest

9.3 9.8 L4 Python Testing Crawler VS pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
faker

9.3 9.3 L4 Python Testing Crawler VS faker

Faker is a Python package that generates fake data for you.
Robot Framework

9.2 9.6 L4 Python Testing Crawler VS Robot Framework

Generic automation framework for acceptance testing and RPA
PyAutoGUI

9.0 0.0 L4 Python Testing Crawler VS PyAutoGUI

A cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.
Moto

8.8 9.7 Python Testing Crawler VS Moto

A library that allows you to easily mock out tests based on AWS infrastructure.
hypothesis

8.5 9.8 L3 Python Testing Crawler VS hypothesis

The property-based testing library for Python
FuckIt.py

7.7 0.0 L4 Python Testing Crawler VS FuckIt.py

The Python error steamroller.
Mimesis

7.6 7.1 L4 Python Testing Crawler VS Mimesis

Mimesis is a Python library for generating fake but realistic data in multiple languages and locales.
responses

7.5 7.3 L4 Python Testing Crawler VS responses

A utility for mocking out the Python Requests library.
tox

7.5 9.6 Python Testing Crawler VS tox

Command line driven CI frontend and development task automation tool.
freezegun

7.4 6.9 L4 Python Testing Crawler VS freezegun

Let your Python tests travel through time
Behave

7.4 7.8 Python Testing Crawler VS Behave

BDD, Python style.
factory_boy

7.2 4.3 L4 Python Testing Crawler VS factory_boy

A test fixtures replacement for Python
Schemathesis

7.0 9.9 Python Testing Crawler VS Schemathesis

Catch API bugs before your users do
VCR.py

6.9 7.2 L4 Python Testing Crawler VS VCR.py

Automatically mock your HTTP interactions to simplify and speed up testing
splinter

6.9 2.3 L5 Python Testing Crawler VS splinter

splinter - python test framework for web applications
httpretty

6.2 0.0 L4 Python Testing Crawler VS httpretty

Intercept HTTP requests at the Python socket level. Fakes the whole socket module
fake2db

6.1 0.0 L2 Python Testing Crawler VS fake2db

create custom test databases that are populated with fake data
sixpack

5.7 0.0 L4 Python Testing Crawler VS sixpack

Sixpack is a language-agnostic a/b-testing framework
Selenium Wire

5.7 0.0 Python Testing Crawler VS Selenium Wire

DISCONTINUED. Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
nose

5.7 0.0 L3 Python Testing Crawler VS nose

nose is nicer testing for python
PyRestTest

5.5 0.0 L3 Python Testing Crawler VS PyRestTest

Python Rest Testing
coverage

5.3 - Python Testing Crawler VS coverage

Code coverage measurement.
model_mommy

4.8 0.0 L5 Python Testing Crawler VS model_mommy

DISCONTINUED. Creating random fixtures for testing in Django.
time-machine

4.5 8.8 Python Testing Crawler VS time-machine

Travel through time in your tests.
nose2

4.5 7.5 L5 Python Testing Crawler VS nose2

The successor to nose, based on unittest2
mixer

4.4 0.0 L5 Python Testing Crawler VS mixer

Mixer -- Is a fixtures replacement. Supported Django, Flask, SqlAlchemy and custom python objects.
syrupy

4.4 9.3 Python Testing Crawler VS syrupy

:pancakes: The most popular pytest snapshot plugin
green

4.3 6.6 L2 Python Testing Crawler VS green

Green is a clean, colorful, fast python test runner.
inline-snapshot

4.0 8.2 Python Testing Crawler VS inline-snapshot

inline-snapshot boosts efficiency when writing tests by generating code with the expected values and simplifies snapshot tests with pytest.
mock

3.9 7.8 L2 Python Testing Crawler VS mock

The Python mock library
betamax

3.9 5.8 Python Testing Crawler VS betamax

A VCR imitation designed only for python-requests.
mamba

3.8 4.4 L5 Python Testing Crawler VS mamba

The definitive testing tool for Python. Born under the banner of Behavior Driven Development (BDD).
fakeredis

3.7 9.5 Python Testing Crawler VS fakeredis

In-memory fake Redis for Python tests: a drop-in replacement for redis-py and valkey-py, with no server to run. Emulates Redis, Valkey, DragonflyDB, KeyDB and KiviDB.
httmock

3.6 0.0 L4 Python Testing Crawler VS httmock

A mocking library for requests
Mocket

3.2 6.2 Python Testing Crawler VS Mocket

a socket mock framework - for all kinds of socket animals, web-clients included
sentry-telegram

3.1 7.0 L5 Python Testing Crawler VS sentry-telegram

Plugin for Sentry which allows sending notification via Telegram messenger.
Cornell

2.4 4.2 Python Testing Crawler VS Cornell

Cornell - record & replay mock server
Slash

2.4 0.0 Python Testing Crawler VS Slash

The Slash testing infrastructure
picka

2.0 0.0 L3 Python Testing Crawler VS picka

pip install picka - Picka is a python based data generation and randomization module which aims to increase coverage by increasing the amount of tests you _dont_ have to write by hand.
python-libfaketime

2.0 5.1 Python Testing Crawler VS python-libfaketime

A fast time mocking alternative to freezegun that wraps libfaketime.
callee

1.9 0.0 L5 Python Testing Crawler VS callee

Argument matchers for unittest.mock
FauxFactory

1.9 7.6 L5 Python Testing Crawler VS FauxFactory

Generates random data for your tests.
aiounittest

1.8 3.2 Python Testing Crawler VS aiounittest

Test python asyncio-based code with ease.
Vedro

1.6 4.4 Python Testing Crawler VS Vedro

Pragmatic Testing Framework
Mock Generator

1.4 0.0 Python Testing Crawler VS Mock Generator

A tool to auto generate the basic mocks and asserts for faster unit testing
RedExpect

1.1 1.8 Python Testing Crawler VS RedExpect

Automate SSH in python easily!
pymox

1.1 9.2 Python Testing Crawler VS pymox

Pymox - Python mocking on steroids

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.

Do you think we are missing an alternative of Python Testing Crawler or a related project?

Add another 'Testing' Package

Popular Comparisons

README

Python Testing Crawler :snake: :stethoscope: :spider:

A crawler for automated functional testing of a web application

Crawling a server-side-rendered web application is a low cost way to get low quality test coverage of your JavaScript-light web application.

If you have only partial test coverage of your routes, but still want to protect against silly mistakes, then this is for you.

Features:

Selectively spider pages and resources, or just request them
Submit forms, and control what values to send
Ignore links by source using CSS selectors
Fail fast or collect many errors
Configurable using straightforward rules

Works with the test clients for Flask (inc Flask-WebTest), Django and WebTest.

Why should I use this?

Here's an example: Flaskr, the Flask tutorial application has 166 lines of test code to achieve 100% test coverage.

Using Python Testing Crawler in a similar way to the Usage example below, we can hit 73% with very little effort. Disclaimer: Of course! It's not the same quality or utility of testing! But it is better than no tests, a complement to hand-written unit or functional tests and a useful stopgap.

Installation

$ pip install python-testing-crawler

Usage

Create a crawler using your framework's existing test client, tell it where to start and what rules to obey, then set it off:

from python_testing_crawler import Crawler
from python_testing_crawler import Rule, Request, Ignore, Allow

def test_crawl_all():
    client = ## ... existing testing client
    ## ... any setup ...
    crawler = Crawler(
        client=my_testing_client,
        initial_paths=['/'],
        rules=[
            Rule("a", '/.*', "GET", Request()),
        ]
    )
    crawler.crawl()

This will crawl all anchor links to relative addresses beginning "/". Any exceptions encountered will be collected and presented at the end of the crawl. For more power see the Rules section below.

If you need to authorise the client's session, e.g. login, then you should that before creating the Crawler.

It is also a good idea to create enough data, via fixtures or otherwise, to expose enough endpoints.

How do I setup a test client?

It depends on your framework:

Crawler Options

Param	Description
`initial_paths`	list of paths/URLs to start from
`rules`	list of Rules to control the crawler; see below
`path_attrs`	list of attribute names to extract paths/URLs from; defaults to "href" -- include "src" if you want to check e.g. `<link>`, `<script>` or even `<img>`
`ignore_css_selectors`	any elements matching this list of CSS selectors will be ignored when extracting links
`ignore_form_fields`	list of form input names to ignore when determining the identity/uniqueness of a form. Include CSRF token field names here.
`max_requests`	Crawler will raise an exception if this limit is exceeded
`capture_exceptions`	upon encountering an exception, keep going and fail at the end of the crawl instead of during (default `True`)
`output_summary`	print summary statistics and any captured exceptions and tracebacks at the end of the crawl (default `True`)
`should_process_handlers`	list of "should process" handlers; see Handlers section
`check_response_handlers`	list of "check response" handlers; see Handlers section

Rules

The crawler has to be told what URLs to follow, what forms to post and what to ignore, using Rules.

Rules are made of four parameters:

Rule(<source element regex>, <target URL/path regex>, <HTTP method>, <action to take>)

These are matched against every HTML element that the crawler encounters, with the last matching rule winning.

Actions must be one of the following objects:

Request(only=False, params=None) -- follow a link or submit a form
- only=True will retrieve a page/resource but not spider its links.
- the dict params allows you to specify overrides for a form's default values
Ignore() -- do nothing / skip
Allow(status_codes) -- allow a HTTP status in the supplied list, i.e. do not consider it an error.

Example Rules

Follow all local/relative links

HYPERLINKS_ONLY_RULE_SET = [
    Rule('a', '/.*', 'GET', Request()),
    Rule('area', '/.*', 'GET', Request()),
]

Request but do not spider all links

REQUEST_ONLY_EXTERNAL_RULE_SET = [
    Rule('a', '.*', 'GET', Request(only=True)),
    Rule('area', '.*', 'GET', Request(only=True)),
]

This is useful for finding broken links. You can also check <link> tags from the <head> if you include the following rule plus set the Crawler's path_attrs to ("HREF", "SRC").

Rule('link', '.*', 'GET', Request())

Submit forms with GET or POST

SUBMIT_GET_FORMS_RULE_SET = [
    Rule('form', '.*', 'GET', Request())
]

SUBMIT_POST_FORMS_RULE_SET = [
    Rule('form', '.*', 'POST', Request())
]

Forms are submitted with their default values, unless overridden using Request(params={...}) for a specific form target or excluded using (globally) using the ignore_form_fields parameter to Crawler (necessary for e.g. CSRF token fields).

Allow some routes to fail

PERMISSIVE_RULE_SET = [
    Rule('.*', '.*', 'GET', Allow([*range(400, 600)])),
    Rule('.*', '.*', 'POST', Allow([*range(400, 600)]))
]

If any HTTP error (400-599) is encountered for any request, allow it; do not error.

Crawl Graph

The crawler builds up a graph of your web application. It can be interrogated via crawler.graph when the crawl is finished.

See [the graph module](python_testing_crawler/graph.py) for the defintion of Node objects.

Handlers

Two hooks points are provided. These operate on Node objects (see above).

Whether to process a Node

Using should_process_handlers, you can register functions that take a Node and return a bool of whether the Crawler should "process" -- follow a link or submit a form -- or not.

Whether a response is acceptable

Using check_response_handlers, you can register functions that take a Node and response object (specific to your test client) and return a bool of whether the response should constitute an error.

If your function returns True, the Crawler with throw an exception.

Examples

There are currently Flask and Django examples in [the tests](tests/).

See https://github.com/python-testing-crawler/flaskr for an example of integrating into an existing application, using Flaskr, the Flask tutorial application.

*Note that all licence references and agreements mentioned in the Python Testing Crawler README section above are relevant to that project's source code only.

Do not miss the trending, packages, news and articles with our weekly report.

Awesome Python is part of the LibHunt network. Terms. Privacy Policy.

(CC)

BY-SA

Spin The Wheel Of Names for a cryptographically secure random name picker.
Schedule a when to meet poll.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.