Testing Spark Tasks
It is possible to test Spark tasks using Pytest.
In the following example, we test the word_count_inline
task to ensure it is running.
The assert_run_task
, the function makes sure that the task has been completed and all outputs exist:
import pytest
from dbnd.testing.helpers_pytest import assert_run_task
from dbnd_test_scenarios.spark.spark_tasks_inline import word_count_inline
class TestSparkTasksLocally(object):
@pytest.fixture(autouse=True)
def spark_conn(self, spark_conn_str):
self.spark_conn_str = spark_conn_str
def test_spark_inline(self):
assert_run_task(word_count_inline.t(text=__file__))
Updated over 1 year ago