integration: Adds support for external source providers

Updates the harness to read and handle a "source" provider in
providers.yaml, falling back to the existing test export provider
(similar to how the "destination" provider is being handled).

Moves provider prerequisites check to the harness.

Skips tests dependent on the test export provider.

tox.ini now accepts CORIOLIS_SOURCE_PROVIDER_PACKAGE and CORIOLIS_DESTINATION_PROVIDER_PACKAGE
dependencies for integration tests, instead of CORIOLIS_PROVIDER_PACKAGE.
It will install them, as they are required to run the integration tests.

Updates README.md and providers.yaml.sample with details about the
source provider.

coriolis/tests/integration/README.md

@@ -23,7 +23,8 @@ The test harness (`harness.py`) performs a one-time setup per process:
 6. Serves the REST API via cheroot on a random local port, with Keystone
    auth replaced by a no-op middleware that injects a fixed admin context.
 7. Registers the built-in `test_provider` as both the export and import
-   provider.
+   provider, unless external providers are configured via
+   `CORIOLIS_PROVIDERS_YAML` (see below).
 
 Teardown (registered with `atexit`) stops all services, removes the Docker
 container, removes the working directory, and detaches any leftover loop
@@ -94,39 +95,64 @@ sudo tox -e integration -- --no-discover coriolis.tests.integration.transfers.te
 > `sudo` is required because `tox` itself must run as root so that the
 > test process inherits root privileges.
 
-## Using an external destination provider
+## Using an external source / destination provider
 
 By default, the harness uses the built-in Docker test provider for both source
-and destination. To run the integration suite against a real destination
-provider, install the provider package via `CORIOLIS_PROVIDER_PACKAGE` and
-supply provider configuration via `CORIOLIS_PROVIDERS_YAML`.
+and destination. To run the integration suite against a real source and / or
+destination provider, install the provider package(s) via
+`CORIOLIS_SOURCE_PROVIDER_PACKAGE` / `CORIOLIS_DESTINATION_PROVIDER_PACKAGE`
+and supply provider configuration via `CORIOLIS_PROVIDERS_YAML`. The `source`
+and `destination` sections in that file are independent; either can be left
+pointing at the built-in test provider.
 
 ### What the harness does with `providers.yaml`
 
-1. Registers the destination provider class with `oslo.config`.
-2. Creates a destination endpoint with `destination.connection_info`.
-3. Uses `destination.environment` as `destination_environment` and
+1. Registers the source and destination provider classes with `oslo.config`.
+2. Creates a source endpoint with `source.connection_info`, and a destination
+   endpoint with `destination.connection_info`.
+3. For an external source provider, `source.instance_name` names a
+   pre-existing VM to migrate; the harness does not create or delete it (this
+   is unlike the destination side, where resources are created and torn down
+   per test). Merges `source.environment` into each transfer's
+   `source_environment`.
+4. Uses `destination.environment` as `destination_environment` and
    `destination.storage_mappings` as `storage_mappings` for each transfer.
 
 ### Running
 
-Set `CORIOLIS_PROVIDER_PACKAGE` to a local path or any pip-compatible specifier
-(`git+file://`, `git+https://`, etc.); tox installs it into the virtualenv
-before running the tests. Leave it unset to use only the built-in test provider.
+Set `CORIOLIS_SOURCE_PROVIDER_PACKAGE` and / or
+`CORIOLIS_DESTINATION_PROVIDER_PACKAGE` to a local path or pip-compatible
+specifier (`git+file://`, `git+https://`, etc.) for the corresponding
+provider package; tox installs them into the virtualenv before running the
+tests. If unset, the built-in test provider is used for that side.
 
 ```bash
-sudo -E CORIOLIS_PROVIDER_PACKAGE=/path/to/provider \
+# Single external provider (e.g.: destination only).
+sudo -E CORIOLIS_DESTINATION_PROVIDER_PACKAGE=/path/to/provider \
+  CORIOLIS_PROVIDERS_YAML=./providers.yaml tox -e integration
+
+# External providers from different packages.
+sudo -E CORIOLIS_SOURCE_PROVIDER_PACKAGE=/path/to/provider-a \
+  CORIOLIS_DESTINATION_PROVIDER_PACKAGE=/path/to/provider-b \
   CORIOLIS_PROVIDERS_YAML=./providers.yaml tox -e integration
 ```
 
 Supply `CORIOLIS_CONFIG_FILE` when provider-specific configurations are required:
 
 ```bash
-sudo -E CORIOLIS_PROVIDER_PACKAGE=/path/to/provider \
+sudo -E CORIOLIS_DESTINATION_PROVIDER_PACKAGE=/path/to/provider \
   CORIOLIS_CONFIG_FILE=./provider.conf \
   CORIOLIS_PROVIDERS_YAML=./providers.yaml tox -e integration
 ```
 
+Additional shared libraries (required by some providers) may be passed to tox:
+
+```bash
+sudo -E LD_LIBRARY_PATH=/path/to/native/libs \
+  CORIOLIS_SOURCE_PROVIDER_PACKAGE=/path/to/provider \
+  CORIOLIS_PROVIDERS_YAML=./providers.yaml tox -e integration
+```
+
 ## Test modules
 
 ### No block devices (extend `CoriolisIntegrationTestBase`)

coriolis/tests/integration/base.py

@@ -63,6 +63,7 @@ class CoriolisIntegrationTestBase(test_base.CoriolisBaseTestCase):
         cls._exp_provider = cls._harness.exp_provider
         cls._exp_platform = cls._harness.exp_provider_platform
         cls._exp_conn_info = cls._harness.exp_conn_info
+        cls._exp_env_options = cls._harness.exp_env_options
 
         cls._imp_provider = cls._harness.imp_provider
         cls._imp_platform = cls._harness.imp_provider_platform
@@ -263,8 +264,6 @@ class ReplicaIntegrationTestBase(CoriolisIntegrationTestBase):
 
     @classmethod
     def setUpClass(cls):
-        harness._IntegrationHarness.get().imp_provider.check_prerequisites()
-
         super().setUpClass()
 
         cls._src_endpoint = cls._create_endpoint(
@@ -307,6 +306,7 @@ class ReplicaIntegrationTestBase(CoriolisIntegrationTestBase):
             destination_minion_pool_id=self._pool_id,
             source_environment={
                 **extra_source_env,
+                **self._exp_env_options,
                 **self._EXTRA_SOURCE_ENVIRONMENT,
             },
         )

coriolis/tests/integration/deployments/test_luks_osmorphing.py

@@ -53,7 +53,10 @@ class _LUKSOSMorphingMixin:
     @classmethod
     def setUpClass(cls):
         harness = integration_harness._IntegrationHarness.get()
-        if not harness.uses_core_test_import_provider():
+        if not (
+            harness.uses_core_test_export_provider()
+            and harness.uses_core_test_import_provider()
+        ):
             raise unittest.SkipTest("OS morphing tests require local disk access")
         super().setUpClass()
 

coriolis/tests/integration/deployments/test_osmorphing.py

@@ -27,7 +27,10 @@ class OsMorphingDeploymentTestBase(integration_base.ReplicaIntegrationTestBase):
     @classmethod
     def setUpClass(cls):
         harness = integration_harness._IntegrationHarness.get()
-        if not harness.uses_core_test_import_provider():
+        if not (
+            harness.uses_core_test_export_provider()
+            and harness.uses_core_test_import_provider()
+        ):
             raise unittest.SkipTest("OS morphing tests require local disk access")
         super().setUpClass()
 

coriolis/tests/integration/harness.py

@@ -58,6 +58,7 @@ from coriolis.taskflow import runner as taskflow_runner
 from coriolis.tasks import factory as task_runners_factory
 from coriolis.tests.integration import provider_test_base
 from coriolis.tests.integration import utils as test_utils
+from coriolis.tests.integration.test_provider import exp as test_provider_exp
 from coriolis.tests.integration.test_provider import imp as test_provider_imp
 from coriolis.transfer_cron.rpc import server as transfer_cron_rpc_server
 from coriolis.worker.rpc import server as worker_rpc_server
@@ -95,6 +96,13 @@ def _load_providers_config():
         with open(_PROVIDERS_YAML) as f:
             providers_config = yaml.safe_load(f) or {}
 
+    src_config = providers_config.get("source", {})
+    src_provider_path = src_config.get("provider") or _TEST_EXPORT_PROVIDER
+    src_provider_cls = _get_provider(src_provider_path)
+
+    if not issubclass(src_provider_cls, provider_test_base.BaseTestExportProvider):
+        raise TypeError("%s must subclass BaseTestExportProvider" % src_provider_path)
+
     dest_config = providers_config.get("destination", {})
     dest_provider_path = dest_config.get("provider") or _TEST_IMPORT_PROVIDER
     dest_provider_cls = _get_provider(dest_provider_path)
@@ -103,6 +111,13 @@ def _load_providers_config():
         raise TypeError("%s must subclass BaseTestImportProvider" % dest_provider_path)
 
     return {
+        "source": {
+            "provider": src_provider_path,
+            "provider_cls": src_provider_cls,
+            "connection_info": src_config.get("connection_info"),
+            "environment": src_config.get("environment") or {},
+            "instance_name": src_config.get("instance_name"),
+        },
         "destination": {
             "provider": dest_provider_path,
             "provider_cls": dest_provider_cls,
@@ -343,8 +358,9 @@ class _IntegrationHarness:
         cfg.CONF.set_override('messaging_transport_url', 'fake://')
 
         providers_config = _load_providers_config()
+        exp_provider = providers_config["source"]["provider"]
         imp_provider = providers_config["destination"]["provider"]
-        cfg.CONF.set_override('providers', [_TEST_EXPORT_PROVIDER, imp_provider])
+        cfg.CONF.set_override('providers', [exp_provider, imp_provider])
         db_url = (
             'mysql+pymysql://%(user)s:%(password)s@localhost:13306/%(database)s'
         ) % {
@@ -368,18 +384,22 @@ class _IntegrationHarness:
         policy_module.reset()
 
         # Init exporter.
-        self.exp_provider_class = _get_provider(_TEST_EXPORT_PROVIDER)
+        exp_provider_cls = providers_config["source"]["provider_cls"]
+        self.exp_provider_class = exp_provider_cls
         self.exp_provider_platform = self.exp_provider_class.platform
         self.exp_provider = providers_factory.get_provider(
             self.exp_provider_platform,
             constants.PROVIDER_TYPE_TRANSFER_EXPORT,
             event_handler=mock.MagicMock(),
         )
-        self.exp_conn_info = {
+        conn_info = providers_config["source"]["connection_info"]
+        self.exp_conn_info = conn_info or {
             "pkey_path": self.ssh_key_path,
             "role": "source",
         }
-        self.exp_provider.initialize(self.exp_conn_info)
+        self.exp_provider.initialize(self.exp_conn_info, providers_config["source"])
+        self.exp_provider.check_prerequisites()
+        self.exp_env_options = providers_config["source"]["environment"]
 
         # Init importer.
         imp_provider_cls = providers_config["destination"]["provider_cls"]
@@ -396,6 +416,7 @@ class _IntegrationHarness:
             "role": "destination",
         }
         self.imp_provider.initialize(self.imp_conn_info)
+        self.imp_provider.check_prerequisites()
         self.imp_env_options = providers_config["destination"]["environment"]
         self.imp_storage_mappings = providers_config["destination"]["storage_mappings"]
         self.imp_minion_pool_environment = providers_config["destination"][
@@ -610,3 +631,10 @@ class _IntegrationHarness:
             self.imp_provider,
             test_provider_imp.TestImportProvider,
         )
+
+    def uses_core_test_export_provider(self):
+        """Returns True when the test export provider is being used."""
+        return isinstance(
+            self.exp_provider,
+            test_provider_exp.TestExportProvider,
+        )

coriolis/tests/integration/provider_test_base.py

@@ -14,20 +14,49 @@ used for testing-only purposes:
 """
 
 import abc
+import unittest
 
 from oslo_log import log as logging
 
+from coriolis import context as coriolis_context
+
 LOG = logging.getLogger(__name__)
 
 
 class BaseTestExportProvider(abc.ABC):
-    def initialize(self, connection_info: dict):
+    """Base test provider to be used by export providers.
+
+    The base implementation relies on a pre-existing VM referenced by
+    ``source.instance_name`` in providers.yaml, which will then be used in the
+    integration tests.
+
+    Providers that need to track / clean up leaked resources around this default
+    behavior (e.g.: export-side snapshots) should override ``initialize()`` /
+    ``teardown()``.
+
+    Must be mixed in ahead of the concrete ExportProvider class, and its
+    own ``__init__`` called explicitly, e.g.:
+
+        class FooTestExportProvider(BaseTestExportProvider, FooExportProvider):
+            def __init__(self, event_handler=None):
+                FooExportProvider.__init__(self, event_handler)
+                BaseTestExportProvider.__init__(self)
+    """
+
+    def __init__(self):
+        self._ctxt = coriolis_context.get_admin_context()
+        self._connection_info = None
+        self._instance_name = None
+
+    def initialize(self, connection_info: dict, source_config: dict):
         """One-time initialization, before any tests run.
 
         Can be used to list the current resources on the source provider,
         which can then be used to check if any test resources leaked and
         clean them.
         """
+        self._connection_info = connection_info
+        self._instance_name = source_config.get("instance_name")
 
     def teardown(self, connection_info: dict):
         """One-time teardown called at atexit.
@@ -37,6 +66,22 @@ class BaseTestExportProvider(abc.ABC):
 
     def check_prerequisites(self):
         """Raise ``unittest.SkipTest`` if required infrastructure is absent."""
+        provider_name = type(self).__name__
+
+        if not self._connection_info:
+            raise unittest.SkipTest(
+                "%s requires 'source.connection_info' in providers.yaml" % provider_name
+            )
+
+        if not self._instance_name:
+            raise unittest.SkipTest(
+                "%s requires 'source.instance_name' in providers.yaml" % provider_name
+            )
+
+        try:
+            self.validate_connection(self._ctxt, self._connection_info)
+        except Exception as ex:
+            raise unittest.SkipTest() from ex
 
     def get_test_instance(self, size_mb: int):
         """Return ``(instance_name, source_environment)`` to use for one test.
@@ -49,7 +94,7 @@ class BaseTestExportProvider(abc.ABC):
         advertise something about the instance they just created (e.g.: the
         core test provider's ``instance_block_devices``).
         """
-        raise NotImplementedError
+        return self._instance_name, {}
 
     def delete_test_instance(self, instance_name: str):
         """Release resources allocated by ``get_test_instance()``, if any."""
@@ -64,7 +109,7 @@ class BaseTestExportProvider(abc.ABC):
 
 
 class BaseTestImportProvider(abc.ABC):
-    def initialize(self, connection_info: dict):
+    def initialize(self, connection_info: dict, source_config: dict):
         """One-time initialization, before any tests run.
 
         Can be used to list the current resources on the target provider,

coriolis/tests/integration/providers.yaml.sample

@@ -1,19 +1,31 @@
-# Sample providers.yaml - built-in Docker test provider as destination.
+# Sample providers.yaml - built-in Docker test provider as source / destination.
 #
 # This sample uses the provider that ships with the test suite itself and
 # requires no external packages or credentials. It is the same provider the
 # harness uses when CORIOLIS_PROVIDERS_YAML is not set, so it is mainly useful
 # as a reference for the file format.
 #
-# To use a real destination provider, copy this file, adjust the ``provider``
-# dotted path to point at your provider's Import Provider, and fill in
-# ``connection_info``, ``environment``, and ``storage_mappings`` as required by
-# that provider.
+# To use a real source and / or destination provider, copy this file, adjust
+# the ``provider`` dotted path(s) to point at your provider's Export /
+# Import Provider, and fill in ``connection_info``, ``environment``, and
+# ``storage_mappings`` as required by that provider. The "source" and
+# "destination" sections are independent; either one can be left pointing at
+# the built-in test provider while the other uses an external provider.
 #
 # Run with:
 #   sudo -E CORIOLIS_PROVIDER_PACKAGE=/path/to/provider \
 #     CORIOLIS_PROVIDERS_YAML=./providers.yaml.sample tox -e integration
 
+source:
+  # Dotted path to an Export Provider to test. Must also implement BaseTestExportProvider
+  provider: "coriolis.tests.integration.test_provider.exp.TestExportProvider"
+
+  # connection_info is passed to the source endpoint.
+  connection_info: null
+
+  # source_environment options merged into each transfer's source_environment.
+  environment: {}
+
 destination:
   # Dotted path to an Import Provider to test. Must also implement BaseTestImportProvider
   provider: "coriolis.tests.integration.test_provider.imp.TestImportProvider"

coriolis/tests/integration/test_endpoints.py

@@ -17,6 +17,7 @@ Exercises endpoint-related operations via the Coriolis REST API:
 
 import unittest
 
+from coriolis import context as coriolis_context
 from coriolis.providers import base as provider_base
 from coriolis.tests.integration import base
 
@@ -86,7 +87,11 @@ class EndpointCapabilitiesTest(base.CoriolisIntegrationTestBase):
     def test_list_source_options(self):
         options = self._client.endpoint_source_options.list(self._src_endpoint.id)
         self.assertIsInstance(options, list)
-        self.assertTrue(len(options) > 0, "Expected at least one source option")
+
+        expected = self._exp_provider.get_source_environment_options(
+            coriolis_context.get_admin_context(), self._exp_conn_info
+        )
+        self.assertEqual(len(expected), len(options))
 
     def test_list_destination_options(self):
         options = self._client.endpoint_destination_options.list(self._dst_endpoint.id)

coriolis/tests/integration/test_provider/exp.py

@@ -82,7 +82,7 @@ class TestExportProvider(
 
     # BaseTestExportProvider - test only
 
-    def initialize(self, connection_info: dict):
+    def initialize(self, connection_info: dict, source_config: dict):
         self._initial_containers = test_utils.list_containers(_CONTAINER_PREFIXES)
 
     def teardown(self, connection_info: dict):

coriolis/tests/integration/transfers/test_transfer.py

@@ -15,6 +15,7 @@ import socketserver
 import tempfile
 import threading
 import time
+import unittest
 import uuid
 import zlib
 from unittest import mock
@@ -26,6 +27,7 @@ from coriolis.db import api as db_api
 from coriolis.providers import backup_writers
 from coriolis.providers import replicator as replicator_module
 from coriolis.tests.integration import base
+from coriolis.tests.integration import harness as integration_harness
 from coriolis.tests.integration import utils as test_utils
 
 CONF = cfg.CONF
@@ -72,6 +74,12 @@ class _ReplicaTransferTestsMixin:
 
         The content is verified only if the test import provider is being used.
         """
+        if not self._harness.uses_core_test_export_provider():
+            self.skipTest(
+                "Incremental transfer verification requires direct access "
+                "to the source device"
+            )
+
         # First run: full transfer
         self._execute_and_wait(self._transfer.id)
 
@@ -233,6 +241,15 @@ class ClusteredTransferIntegrationTest(base.ReplicaIntegrationTestBase):
     them (same disk id in both instances' export_info).
     """
 
+    @classmethod
+    def setUpClass(cls):
+        h = integration_harness._IntegrationHarness.get()
+        if not h.uses_core_test_export_provider():
+            raise unittest.SkipTest(
+                "Clustered transfers require the core test export provider"
+            )
+        super().setUpClass()
+
     def setUp(self):
         super().setUp()
 
@@ -428,6 +445,15 @@ class ReplicaTransferViaSSHTunnelTest(base.ReplicaIntegrationTestBase):
 
     _EXTRA_SOURCE_ENVIRONMENT = {"use_tunnel": True}
 
+    @classmethod
+    def setUpClass(cls):
+        h = integration_harness._IntegrationHarness.get()
+        if not h.uses_core_test_export_provider():
+            raise unittest.SkipTest(
+                "'use_tunnel' is a core test export provider option"
+            )
+        super().setUpClass()
+
     def test_transfer_via_ssh_tunnel(self):
         tunnel_starts = []
         original_get_ssh_tunnel = replicator_module.Client._get_ssh_tunnel

tox.ini

@@ -50,18 +50,24 @@ commands =
 # Must be run as root: sudo -E tox -e integration
 # Requires losetup support (loop devices).
 #
-# To test with an external provider, set CORIOLIS_PROVIDER_PACKAGE to a local
-# path or pip-compatible specifier (git+file://, git+https://, etc.) and run:
-#   sudo -E CORIOLIS_PROVIDER_PACKAGE=/path/to/provider tox -e integration
+# To test with external providers, set CORIOLIS_SOURCE_PROVIDER_PACKAGE
+# and / or CORIOLIS_DESTINATION_PROVIDER_PACKAGE to a local path or
+# pip-compatible specifier (git+file://, git+https://, etc.) for the
+# provider package to install, and run:
+#   sudo -E CORIOLIS_SOURCE_PROVIDER_PACKAGE=/path/to/provider-a \
+#     CORIOLIS_DESTINATION_PROVIDER_PACKAGE=/path/to/provider-b \
+#     tox -e integration
 setenv =
   {[testenv]setenv}
   PBR_VERSION = 0.0.1
 passenv =
   CORIOLIS_*
+  LD_LIBRARY_PATH
 deps =
   {[testenv]deps}
   git+https://github.com/cloudbase/python-coriolisclient.git
-  {env:CORIOLIS_PROVIDER_PACKAGE:}
+  {env:CORIOLIS_SOURCE_PROVIDER_PACKAGE:}
+  {env:CORIOLIS_DESTINATION_PROVIDER_PACKAGE:}
 commands = stestr run --slowest --concurrency=1 --test-path coriolis/tests/integration/ {posargs}
 
 [testenv:venv]