All Projects → malexer → pytest-spark

malexer / pytest-spark

Licence: MIT license
pytest plugin to run the tests with support of pyspark

Programming Languages

python
139335 projects - #7 most used programming language
Dockerfile
14818 projects
shell
77523 projects

Projects that are alternatives of or similar to pytest-spark

unittest expander
A library that provides flexible and easy-to-use tools to parameterize Python unit tests, especially those based on unittest.TestCase.
Stars: ✭ 12 (-81.54%)
Mutual labels:  unittest, unit-test
Testify
A unit testing framework written in bash for bash scripts
Stars: ✭ 45 (-30.77%)
Mutual labels:  unittest, unit-test
tsioc
AOP, Ioc container, Boot framework, unit testing framework , activities workflow framework.
Stars: ✭ 15 (-76.92%)
Mutual labels:  unittest, unit-test
Nose2
The successor to nose, based on unittest2
Stars: ✭ 665 (+923.08%)
Mutual labels:  unittest, unit-test
Pytest Spec
Library pytest-spec is a pytest plugin to display test execution output like a SPECIFICATION.
Stars: ✭ 65 (+0%)
Mutual labels:  pytest, unittest
Clarity
A declaritive test framework for Terraform
Stars: ✭ 88 (+35.38%)
Mutual labels:  unittest, unit-test
Nunit cshaprp cheatsheet
Example implementations of each attribute available in Nunit2 unit Testing Framework using C# .NET.
Stars: ✭ 14 (-78.46%)
Mutual labels:  unittest, unit-test
VCore
VCore is a Swift collection containing objects, functions, and extensions that I use for my projects
Stars: ✭ 32 (-50.77%)
Mutual labels:  unittest, unit-test
Unittest Xml Reporting
unittest-based test runner with Ant/JUnit like XML reporting.
Stars: ✭ 255 (+292.31%)
Mutual labels:  pytest, unittest
Kotlinmvparchitecture
Clean MVP Architecture with Dagger2 + Retrofit2 + Mockito + Fresco + EasiestGenericRecyclerAdapter using Kotlin. Added Unit Tests(Kotlin Tests)!
Stars: ✭ 143 (+120%)
Mutual labels:  unittest, unit-test
FineCodeCoverage
Visualize unit test code coverage easily for free in Visual Studio Community Edition (and other editions too)
Stars: ✭ 391 (+501.54%)
Mutual labels:  unittest, unit-test
TestIt
Generate unit testing boilerplate from kotlin files.
Stars: ✭ 32 (-50.77%)
Mutual labels:  unittest, unit-test
PizzaDelivery
Pizza Delivery is an open source sample app with robust and professional features. This is a fully realized built entirely in Swift.
Stars: ✭ 20 (-69.23%)
Mutual labels:  unittest
mutatest
Python mutation testing: test your tests! Safely run mutation trials without source code modifications and see what will get past your test suite.
Stars: ✭ 50 (-23.08%)
Mutual labels:  unit-test
pytest-pipeline
Pytest plugin for functional testing of data analysis pipelines
Stars: ✭ 19 (-70.77%)
Mutual labels:  pytest
pytest-snail
Plugin for adding a marker to slow running tests. 🐌
Stars: ✭ 15 (-76.92%)
Mutual labels:  pytest
HTMLTestRunner cn
HTMLTestRunner 汉化版,同时支持python 2和3,增加截图展示功能,失败重试
Stars: ✭ 191 (+193.85%)
Mutual labels:  unittest
randunit
Bring unittest to your Android/JVM applications with tiny effort.
Stars: ✭ 33 (-49.23%)
Mutual labels:  unittest
torchjs
Test framework to light up the world.
Stars: ✭ 40 (-38.46%)
Mutual labels:  unit-test
python-package-template
Easy to use template for great PyPi packages
Stars: ✭ 19 (-70.77%)
Mutual labels:  pytest

pytest-spark

https://travis-ci.org/malexer/pytest-spark.svg?branch=master

pytest plugin to run the tests with support of pyspark (Apache Spark).

This plugin will allow to specify SPARK_HOME directory in pytest.ini and thus to make "pyspark" importable in your tests which are executed by pytest.

You can also define "spark_options" in pytest.ini to customize pyspark, including "spark.jars.packages" option which allows to load external libraries (e.g. "com.databricks:spark-xml").

pytest-spark provides session scope fixtures spark_context and spark_session which can be used in your tests.

Note: no need to define SPARK_HOME if you've installed pyspark using pip (e.g. pip install pyspark) - it should be already importable. In this case just don't define SPARK_HOME neither in pytest (pytest.ini / --spark_home) nor as environment variable.

Install

$ pip install pytest-spark

Usage

Set Spark location

To run tests with required spark_home location you need to define it by using one of the following methods:

  1. Specify command line option "--spark_home":

    $ pytest --spark_home=/opt/spark
    
  2. Add "spark_home" value to pytest.ini in your project directory:

    [pytest]
    spark_home = /opt/spark
    
  3. Set the "SPARK_HOME" environment variable.

pytest-spark will try to import pyspark from provided location.

Note

"spark_home" will be read in the specified order. i.e. you can override pytest.ini value by command line option.

Customize spark_options

Just define "spark_options" in your pytest.ini, e.g.:

[pytest]
spark_home = /opt/spark
spark_options =
    spark.app.name: my-pytest-spark-tests
    spark.executor.instances: 1
    spark.jars.packages: com.databricks:spark-xml_2.12:0.5.0

Using the spark_context fixture

Use fixture spark_context in your tests as a regular pyspark fixture. SparkContext instance will be created once and reused for the whole test session.

Example:

def test_my_case(spark_context):
    test_rdd = spark_context.parallelize([1, 2, 3, 4])
    # ...

Using the spark_session fixture (Spark 2.0 and above)

Use fixture spark_session in your tests as a regular pyspark fixture. A SparkSession instance with Hive support enabled will be created once and reused for the whole test session.

Example:

def test_spark_session_dataframe(spark_session):
    test_df = spark_session.createDataFrame([[1,3],[2,4]], "a: int, b: int")
    # ...

Overriding default parameters of the spark_session fixture

By default spark_session will be loaded with the following configurations :

Example:

{
    'spark.app.name': 'pytest-spark',
    'spark.default.parallelism': 1,
    'spark.dynamicAllocation.enabled': 'false',
    'spark.executor.cores': 1,
    'spark.executor.instances': 1,
    'spark.io.compression.codec': 'lz4',
    'spark.rdd.compress': 'false',
    'spark.sql.shuffle.partitions': 1,
    'spark.shuffle.compress': 'false',
    'spark.sql.catalogImplementation': 'hive',
}

You can override some of these parameters in your pytest.ini. For example, removing Hive Support for the spark session :

Example:

[pytest]
spark_home = /opt/spark
spark_options =
    spark.sql.catalogImplementation: in-memory

Development

Tests

Run tests locally:

$ docker-compose up --build
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].