Merge branch 'master' into middleware

README.rst

@@ -1,7 +1,15 @@
-CloudBridge provides a simple layer of abstraction over different
+CloudBridge provides a consistent layer of abstraction over different
 Infrastructure-as-a-Service cloud providers, reducing or eliminating the need
 to write conditional code for each cloud.
 
+Documentation
+~~~~~~~~~~~~~
+Detailed documentation can be found at http://cloudbridge.cloudve.org.
+
+
+Build Status Tests
+~~~~~~~~~~~~~~~~~~
+
 .. image:: https://coveralls.io/repos/CloudVE/cloudbridge/badge.svg?branch=master&service=github
    :target: https://coveralls.io/github/CloudVE/cloudbridge?branch=master
    :alt: Code Coverage
@@ -14,45 +22,37 @@ to write conditional code for each cloud.
    :target: http://cloudbridge.readthedocs.org/en/latest/?badge=latest
    :alt: Documentation Status
 
-.. image:: https://badge.waffle.io/CloudVE/cloudbridge.png?label=in%20progress&title=In%20Progress 
-   :target: https://waffle.io/CloudVE/cloudbridge?utm_source=badge
-   :alt: 'Waffle.io - Issues in progress'
-
 .. |aws-py27| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/1
               :target: https://travis-ci.org/CloudVE/cloudbridge
 .. |aws-py36| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/5
               :target: https://travis-ci.org/CloudVE/cloudbridge
 
 .. |azure-py27| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/2
-                :target: https://travis-ci.org/CloudVE/cloudbridge/branches
+                :target: https://travis-ci.org/CloudVE/cloudbridge
 .. |azure-py36| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/6
-                :target: https://travis-ci.org/CloudVE/cloudbridge/branches
+                :target: https://travis-ci.org/CloudVE/cloudbridge
 
 .. |gce-py27| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/3
-              :target: https://travis-ci.org/CloudVE/cloudbridge/branches
+              :target: https://travis-ci.org/CloudVE/cloudbridge
 .. |gce-py36| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/7
-              :target: https://travis-ci.org/CloudVE/cloudbridge/branches
+              :target: https://travis-ci.org/CloudVE/cloudbridge
 
 .. |os-py27| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/4
              :target: https://travis-ci.org/CloudVE/cloudbridge
 .. |os-py36| image:: https://travis-matrix-badges.herokuapp.com/repos/CloudVE/cloudbridge/branches/master/8
              :target: https://travis-ci.org/CloudVE/cloudbridge
 
-
-Build Status Tests
-~~~~~~~~~~~~~~~~~~
-
-+--------------------------+--------------+--------------+
-| **Provider/Environment** | py27         | py36         |
-+--------------------------+--------------+--------------+
-| **AWS**                  | |aws-py27|   | |aws-py36|   |
-+--------------------------+--------------+--------------+
-| **OpenStack**            | |os-py27|    | |os-py36|    |
-+--------------------------+--------------+--------------+
-| **Azure**                | |azure-py27| | |azure-py36| |
-+--------------------------+--------------+--------------+
-| **GCE**                  | |gce-py27|   | |gce-py36|   |
-+--------------------------+--------------+--------------+
++---------------------------+----------------+----------------+
+| **Provider/Environment**  | **Python 2.7** | **Python 3.6** |
++---------------------------+----------------+----------------+
+| **Amazon Web Services**   | |aws-py27|     | |aws-py36|     |
++---------------------------+----------------+----------------+
+| **Google Compute Engine** | |gce-py27|     | |gce-py36|     |
++---------------------------+----------------+----------------+
+| **Microsoft Azure**       | |azure-py27|   | |azure-py36|   |
++---------------------------+----------------+----------------+
+| **OpenStack**             | |os-py27|      | |os-py36|      |
++---------------------------+----------------+----------------+
 
 Installation
 ~~~~~~~~~~~~
@@ -93,11 +93,6 @@ presented at the Proceedings of the XSEDE16 Conference on Diversity, Big Data, a
 DOI: http://dx.doi.org/10.1145/2949550.2949648
 
 
-Documentation
-~~~~~~~~~~~~~
-Documentation can be found at https://cloudbridge.readthedocs.org.
-
-
 Quick Reference
 ~~~~~~~~~~~~~~~
 The following object graph shows how to access various provider services, and the resource

cloudbridge/cloud/factory.py

@@ -124,10 +124,10 @@ class CloudProviderFactory(object):
         :param name: Cloud provider name: one of ``aws``, ``openstack``,
         ``azure``.
 
-        :type config: an object with required fields
-        :param config: This can be a Bunch or any other object whose fields can
-                       be accessed using dot notation. See specific provider
-                       implementation for the required fields.
+        :type config: :class:`dict`
+        :param config: A dictionary or an iterable of key/value pairs (as
+                       tuples or other iterables of length two). See specific
+                       provider implementation for the required fields.
 
         :return:  a concrete provider instance
         :rtype: ``object`` of :class:`.CloudProvider`

cloudbridge/cloud/interfaces/provider.py

@@ -18,10 +18,10 @@ class CloudProvider(object):
 
         :type config: :class:`dict`
         :param config: A dictionary object containing provider initialization
-                       values. Alternatively, this can be a Bunch or any other
-                       object whose fields can be accessed as members. See
-                       specific provider implementation for the required
-                       fields.
+                       values. Alternatively, this can be an iterable of
+                       key/value pairs (as tuples or other iterables of length
+                       two). See specific provider implementation for the
+                       required fields.
 
         :rtype: :class:`.CloudProvider`
         :return:  a concrete provider instance

cloudbridge/cloud/providers/gce/helpers.py

@@ -164,18 +164,24 @@ def __if_label_fingerprint_differs(e):
 def change_label(resource, key, value, res_att, request):
     resource.assert_valid_resource_label(value)
     labels = getattr(resource, res_att).get("labels", {})
-    print(labels)
-    labels[key] = value
+    # The returned value from above command yields a unicode dict key, which
+    # cannot be simply cast into a str for py2 so pop the key and re-add it
+    # The casting needs to be done for all labels, as to support both
+    # description and label setting
+    labels[key] = str(value)
+    for k in labels.keys():
+        labels[str(k)] = str(labels.pop(k))
+
     request_body = {
         "labels": labels,
         "labelFingerprint":
-            getattr(resource, res_att).get('labelFingerprint'),
+            str(getattr(resource, res_att).get('labelFingerprint')),
     }
     try:
         request.body = str(request_body)
         request.body_size = len(str(request_body))
         response = request.execute()
-        resource._provider.wait_for_operation(response,
-                                              zone=resource.zone_name)
+        resource._provider.wait_for_operation(
+            response, zone=getattr(resource, 'zone_name', None))
     finally:
         resource.refresh()

cloudbridge/cloud/providers/gce/services.py

@@ -279,7 +279,7 @@ class GCEImageService(BaseImageService):
         self._public_images = None
 
     _PUBLIC_IMAGE_PROJECTS = ['centos-cloud', 'coreos-cloud', 'debian-cloud',
-                              'opensuse-cloud', 'ubuntu-os-cloud']
+                              'opensuse-cloud', 'ubuntu-os-cloud', 'cos-cloud']
 
     def _retrieve_public_images(self):
         if self._public_images is not None:
@@ -379,7 +379,8 @@ class GCEInstanceService(BaseInstanceService):
                     # be used by default which may collide with existing disks.
                     source_value = {
                         'sourceImage': disk.source.id,
-                        'diskName': 'image-disk-{0}'.format(uuid.uuid4())}
+                        'diskName': 'image-disk-{0}'.format(uuid.uuid4()),
+                        'diskSizeGb': disk.size if disk.size else 20}
                 elif isinstance(disk.source, GCEVolume):
                     source_field = 'source'
                     source_value = disk.source.id
@@ -764,30 +765,27 @@ class GCESubnetService(BaseSubnetService):
 
     def list(self, network=None, zone=None, limit=None, marker=None):
         """
-        If the zone is not given, we list all subnetworks, in all regions.
+        If the zone is not given, we list all subnets in the default region.
         """
         filter = None
         if network is not None:
             network = (network if isinstance(network, GCENetwork)
                        else self.provider.networking.networks.get(network))
             filter = 'network eq %s' % network.resource_url
-        region_names = []
         if zone:
-            region_names.append(self._zone_to_region(zone))
+            region_name = self._zone_to_region(zone)
         else:
-            for r in self.provider.compute.regions:
-                region_names.append(r.name)
+            region_name = self.provider.region_name
         subnets = []
-        for region_name in region_names:
-            response = (self.provider
-                            .gce_compute
-                            .subnetworks()
-                            .list(project=self.provider.project_name,
-                                  region=region_name,
-                                  filter=filter)
-                            .execute())
-            for subnet in response.get('items', []):
-                subnets.append(GCESubnet(self.provider, subnet))
+        response = (self.provider
+                        .gce_compute
+                        .subnetworks()
+                        .list(project=self.provider.project_name,
+                              region=region_name,
+                              filter=filter)
+                        .execute())
+        for subnet in response.get('items', []):
+            subnets.append(GCESubnet(self.provider, subnet))
         return ClientPagedResultList(self.provider, subnets,
                                      limit=limit, marker=marker)
 

cloudbridge/cloud/providers/openstack/resources.py

@@ -921,8 +921,8 @@ class OpenStackNetwork(BaseNetwork):
             # pylint:disable=protected-access
             self._network = network._network
         else:
-            # subnet no longer exists
-            self._network.state = NetworkState.UNKNOWN
+            # Network no longer exists
+            self._network = {}
 
     @property
     def gateways(self):
@@ -1064,7 +1064,7 @@ class OpenStackRouter(BaseRouter):
 
     @property
     def id(self):
-        return self._router.get('id', None)
+        return getattr(self._router, 'id', None)
 
     @property
     def name(self):
@@ -1072,7 +1072,7 @@ class OpenStackRouter(BaseRouter):
 
     @property
     def label(self):
-        return self._router.get('name', None)
+        return self._router.name
 
     @label.setter
     def label(self, value):  # pylint:disable=arguments-differ
@@ -1080,65 +1080,63 @@ class OpenStackRouter(BaseRouter):
         Set the router label.
         """
         self.assert_valid_resource_label(value)
-        self._provider.neutron.update_router(
-            self.id, {'router': {'name': value or ""}})
-        self.refresh()
+        self._router = self._provider.os_conn.update_router(self.id, value)
 
     def refresh(self):
-        self._router = self._provider.neutron.show_router(self.id)['router']
+        self._router = self._provider.os_conn.get_router(self.id)
 
     @property
     def state(self):
-        if self._router.get('external_gateway_info'):
+        if self._router.external_gateway_info:
             return RouterState.ATTACHED
         return RouterState.DETACHED
 
     @property
     def network_id(self):
-        if self.state == RouterState.ATTACHED:
-            return self._router.get('external_gateway_info', {}).get(
-                'network_id', None)
+        ports = self._provider.os_conn.list_ports(
+            filters={'device_id': self.id})
+        if ports:
+            return ports[0].network_id
         return None
 
     def delete(self):
-        self._provider.neutron.delete_router(self.id)
+        self._provider.os_conn.delete_router(self.id)
 
     def attach_subnet(self, subnet):
-        router_interface = {'subnet_id': subnet.id}
-        ret = self._provider.neutron.add_interface_router(
-            self.id, router_interface)
+        ret = self._provider.os_conn.add_router_interface(
+            self._router.toDict(), subnet.id)
         if subnet.id in ret.get('subnet_ids', ""):
             return True
         return False
 
     def detach_subnet(self, subnet):
-        router_interface = {'subnet_id': subnet.id}
-        ret = self._provider.neutron.remove_interface_router(
-            self.id, router_interface)
-        if subnet.id in ret.get('subnet_ids', ""):
+        ret = self._provider.os_conn.remove_router_interface(
+            self._router.toDict(), subnet.id)
+        if not ret or subnet.id not in ret.get('subnet_ids', ""):
             return True
         return False
 
     @property
     def subnets(self):
-        # A router and a subnet are linked via a port, so traverse all ports
-        # to find a list of subnets associated with the current router.
+        # A router and a subnet are linked via a port, so traverse ports
+        # associated with the current router to find a list of subnets
+        # associated with it.
         subnets = []
-        for prt in self._provider.neutron.list_ports().get('ports'):
-            if prt.get('device_id') == self.id and \
-               prt.get('device_owner') == 'network:router_interface':
-                for fixed_ip in prt.get('fixed_ips'):
-                    subnets.append(self._provider.networking.subnets.get(
-                        fixed_ip.get('subnet_id')))
+        for port in self._provider.os_conn.list_ports(
+                filters={'device_id': self.id}):
+            for fixed_ip in port.fixed_ips:
+                subnets.append(self._provider.networking.subnets.get(
+                    fixed_ip.get('subnet_id')))
         return subnets
 
     def attach_gateway(self, gateway):
-        self._provider.neutron.add_gateway_router(
-            self.id, {'network_id': gateway.id})
+        self._provider.os_conn.update_router(
+            self.id, ext_gateway_net_id=gateway.id)
 
     def detach_gateway(self, gateway):
-        self._provider.neutron.remove_gateway_router(
-            self.id).get('router', self._router)
+        # TODO: OpenStack SDK Connection object doesn't appear to have a method
+        # for detaching/clearing the external gateway.
+        self._provider.neutron.remove_gateway_router(self.id)
 
 
 class OpenStackInternetGateway(BaseInternetGateway):

cloudbridge/cloud/providers/openstack/services.py

@@ -952,11 +952,11 @@ class OpenStackRouterService(BaseRouterService):
 
     def get(self, router_id):
         log.debug("Getting OpenStack Router with the id: %s", router_id)
-        router = (r for r in self if r.id == router_id)
-        return next(router, None)
+        router = self.provider.os_conn.get_router(router_id)
+        return OpenStackRouter(self.provider, router) if router else None
 
     def list(self, limit=None, marker=None):
-        routers = self.provider.neutron.list_routers().get('routers')
+        routers = self.provider.os_conn.list_routers()
         os_routers = [OpenStackRouter(self.provider, r) for r in routers]
         return ClientPagedResultList(self.provider, os_routers, limit=limit,
                                      marker=marker)
@@ -968,16 +968,8 @@ class OpenStackRouterService(BaseRouterService):
         return ClientPagedResultList(self._provider, list(matches))
 
     def create(self, label, network):
-        """
-        ``network`` is not used by OpenStack.
-
-        However, the API seems to indicate it is a (required) param?!
-        https://developer.openstack.org/api-ref/networking/v2/
-            ?expanded=delete-router-detail,create-router-detail#create-router
-        """
+        """Parameter ``network`` is not used by OpenStack."""
         log.debug("Creating OpenStack Router with the label: %s", label)
         OpenStackRouter.assert_valid_resource_label(label)
-
-        body = {'router': {'name': label}} if label else None
-        router = self.provider.neutron.create_router(body)
-        return OpenStackRouter(self.provider, router.get('router'))
+        router = self.provider.os_conn.create_router(name=label)
+        return OpenStackRouter(self.provider, router)

docs/conf.py

@@ -55,7 +55,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'cloudbridge'
-copyright = u'2017, GVL and Galaxy Projects'
+copyright = u'2019, GVL and Galaxy Projects'
 author = u'GVL and Galaxy Projects'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -63,9 +63,9 @@ author = u'GVL and Galaxy Projects'
 # built documents.
 #
 # The short X.Y version.
-version = '0.1'
+version = '1.0.2'
 # The full version, including alpha/beta/rc tags.
-release = '0.1'
+release = '1.0.2'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/topics/overview.rst

@@ -6,6 +6,7 @@ Introductions to all the key parts of CloudBridge you'll need to know:
    :maxdepth: 1
 
     How to install CloudBridge <install.rst>
+    Procuring access credentials <procuring_credentials.rst>
     Connection and authentication setup <setup.rst>
     Launching instances <launch.rst>
     Networking <networking.rst>

docs/topics/procuring_credentials.rst

@@ -1,24 +1,28 @@
-
 Procuring access credentials
-----------------------------
-To initialize a connection to a cloud and get a provider object, you will
-need to provide the cloud's access credentials to CloudBridge. This page
-will walk you through the process of procuring credentials. For more
-information on providing these credentials to CloudBridge, see
-`Providing Access Credentials <setup.html>`_.
+============================
+
+To initialize a connection to a cloud and get a provider object, you will need
+to provide the cloud's access credentials to CloudBridge. This page will walk
+you through the process of procuring credentials on different providers. For
+more information on using the obtained credentials with CloudBridge, see
+`Connection and Authentication Setup <setup.html>`_ page.
 
-**Microsoft Azure**
+.. _azure-creds:
 
-The page linked below from the Microsoft Documentation was used to create this
+Microsoft Azure
+---------------
+
+The page linked below from the Microsoft documentation was used to create this
 section, and can be followed instead of this CloudBridge-specific documentation
-to procure Azure credentials for other purposes.
-https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal
+to procure Azure credentials
+https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal.
 
-In order to follow the isntructions below, one needs a Microsoft Azure
-account and Subscription, as well as portal access.
-The following section will walk you through the process of creating an
-application, which is required for API access, as well as help you locate
-all required Azure credentials for CloudBridge.
+In order to follow the instructions below, one needs a Microsoft Azure account
+and an active Subscription, as well as web portal access:
+https://portal.azure.com/ The following section will walk you through the
+process of creating an application, which is required for API access, as well
+as help you locate all required Azure credentials for API access via
+CloudBridge.
 
 Azure credentials require a `Subscription ID` which can be procured through
 `Subscriptions` in the Azure portal.
@@ -43,8 +47,8 @@ procured through the Azure Active Directory's `Properties`.
    :alt: Azure Directory 1
 
    The `Azure Active Directory` can be access by searching, or by choosing
-   `Azure Active Directory` in the `All Services` window. `Azure Active
-   Directory` is also a default favorite on the sidebar
+   `Azure Active Directory` in the `All Services` window.
+   `Azure Active Directory` is also a default favorite on the sidebar.
 
 .. figure:: captures/az-dir-2.png
    :alt: Azure Directory 2
@@ -54,16 +58,16 @@ procured through the Azure Active Directory's `Properties`.
 
 In order to access the API, an application needs to be registered and a key
 needs to be created. After creating an application through the
-`App Registrations` window under the `Active Directory`, the `Application
-ID` of the app will correspond to the `Client ID` in CloudBridge, and the
+`App Registrations` window under the `Active Directory`, the `Application ID`
+of the app will correspond to the `Client ID` in CloudBridge, and the
 generated value of its key, will correspond to the `Secret`.
 
 
 .. figure:: captures/az-app-1.png
    :alt: Azure App 1
 
-   `App Registrations` can be access by searching, or through choosing `App
-   Registrations` under `Azure Active Directory`
+   `App Registrations` can be access by searching, or through choosing
+   `App Registrations` under `Azure Active Directory`.
 
 .. figure:: captures/az-app-2.png
    :alt: Azure App 2
@@ -82,26 +86,26 @@ generated value of its key, will correspond to the `Secret`.
    :alt: Azure App 4
 
    After creating the application, one must select it, after which the
-   `Application ID` will map to the `Client ID` in CloudBridge
+   `Application ID` will map to the `Client ID` in CloudBridge.
 
 .. figure:: captures/az-app-5.png
    :alt: Azure App 5
 
    In the application's `Settings` panel, under the `Keys` section, one will
-   be able to create a new `Secret`
+   be able to create a new `Secret`.
 
 .. figure:: captures/az-app-6.png
    :alt: Azure App 6
 
    Any name can be given to the key, and any expiration date, after which
    the `Save` button will generate the `Key` which will correspond to the
-   `Secret` in CloudBridge
+   `Secret` in CloudBridge.
 
 .. figure:: captures/az-app-7.png
    :alt: Azure App 7
 
    The value of the key will correspond to the `Secret` in CloudBridge and
-   needs to be saved at creation-time
+   needs to be saved at creation-time.
 
 
 Finally, in order to have appropriate permissions, you must assign an
@@ -116,32 +120,64 @@ specific roles can also be assigned for more limited access.
    :alt: Azure Roles 1
 
    Subscription-level access will allow the application to access resources
-   from multiple resource groups
+   from multiple resource groups.
 
 .. figure:: captures/az-role-2.png
    :alt: Azure Roles 2
 
    When roles are set at the level of the Resource Group, one must specify
    this Resource Group as part of the credentials, as the application will
-   not have enough permissions to create a Resource Group
+   not have enough permissions to create a Resource Group.
 
 .. figure:: captures/az-role-3.png
    :alt: Azure Roles 3
 
    Adding a role assignment to the application will give it appropriate
-   permissions to manage resources
+   permissions to manage resources.
+
+.. _google-creds:
+
+Google Compute Engine (GCE)
+---------------------------
+
+For Google Compute Engine (GCE), create a service account followed by creating
+and downloading a key. Additional instructions are available at this link
+https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account.
+
+Start off by clicking on the `Create Service Account` button on the
+IAM & admin section of the Google Cloud Console:
+https://console.cloud.google.com/iam-admin/serviceaccounts.
 
+.. figure:: captures/gce-sa-1.png
+   :alt: GCE Service Account 1
 
-**Google**
+Next, we provide a name for the service account and an informative description.
+Note that the supplied name is used to create an email address for the service
+account. Once created, this email cannot be changed.
 
-For Google Compute Engine, create a service account following instructions
-from the link below:
-https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account
+.. figure:: captures/gce-sa-2.png
+   :alt: GCE Service Account 2
 
-Once created, grant the account appropriate permissions for your use through
-roles, and create a key, choosing JSON format, when prompted. These
-credentials can then be used with CloudBridge through the variables shown
-in the sections below.
+We then assign a role to the service account. Depending on what you will be
+doing with the service account, and CloudBridge, you can set granular access
+roles for the service account. The `Editor` role on the project is very broad
+and will allow you to exercise all of the CloudBridge capabilities.
+
+.. figure:: captures/gce-sa-3.png
+   :alt: GCE Service Account 3
+
+After a service account has been created, we need to add a key to it.
+
+.. figure:: captures/gce-sa-4.png
+   :alt: GCE Service Account key 1
+
+Finally, choose the JSON format for the key when prompted. The file that is
+downloaded will be used with CloudBridge through the variables shown
+on the `Connection and Authentication Setup <setup.html>`_ page. Note that you
+may have multiple keys associated with the same service account.
+
+.. figure:: captures/gce-sa-5.png
+   :alt: GCE Service Account key 2
 
 The JSON credentials file will have a similar form to the example shown
 below, and can either be passed through an absolute path to the file, or
@@ -162,3 +198,8 @@ through a variable containing the JSON dictionary itself.
       "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
       "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-name%40my-project.iam.gserviceaccount.com"
     }
+
+
+.. Comments for documentation authors:
+.. When uploading pictures, process them using sips, as follows:
+.. sips -s format png -Z 1500 img-name.png

docs/topics/setup.rst

@@ -1,135 +1,20 @@
 Setup
------
+=====
 To initialize a connection to a cloud and get a provider object, you will
 need to provide the cloud's access credentials to CloudBridge. For more
-details on how to create and find these credentials, see `Procuring Access
-Credentials <procuring_credentials.html>`_. Once available, these may be
-provided in one of following ways:
+details on how to create and find these credentials, see the `Procuring Access
+Credentials <procuring_credentials.html>`_ page. Note that you can selectively
+provide the credentials for any provider you want to use and do not have to
+provide credentials for all the providers. CloudBridge will consume the
+available credentials in one of following ways:
 
-1. Environment variables
-2. A dictionary
-3. Configuration file
-
-
-Providing access credentials through environment variables
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The following environment variables must be set, depending on the provider in use.
-
-
-**Amazon**
-
-+---------------------+
-| Mandatory variables |
-+=====================+
-| AWS_ACCESS_KEY      |
-+---------------------+
-| AWS_SECRET_KEY      |
-+---------------------+
-
-
-**Openstack**
-
-+---------------------+
-| Mandatory variables |
-+=====================+
-| OS_AUTH_URL         |
-+---------------------+
-| OS_USERNAME         |
-+---------------------+
-| OS_PASSWORD         |
-+---------------------+
-| OS_PROJECT_NAME     |
-+---------------------+
-| OS_REGION_NAME      |
-+---------------------+
-
-+------------------------+
-| Optional Variables     |
-+========================+
-| NOVA_SERVICE_NAME      |
-+------------------------+
-| OS_COMPUTE_API_VERSION |
-+------------------------+
-| OS_VOLUME_API_VERSION  |
-+------------------------+
-| OS_STORAGE_URL         |
-+------------------------+
-| OS_AUTH_TOKEN          |
-+------------------------+
-
-
-**Microsoft Azure**
-
-Note that managing resources in Azure requires a Resource Group. If a
-Resource Group is not provided as part of the configuration, cloudbridge will
-attempt to create a Resource Group using the given credentials. This
-operation will happen with the client initialization, and requires a
-"contributor" or "owner" role.
-Similarly, a Storage Account is required when managing some resources, such
-as KeyPairs and Buckets. If a Storage Account name is not provided as part
-of the configuration, cloudbridge will attempt to create the Storage Account
-when initializing the relevant services. This operation similarly requires a
-"contributor" or "owner" role.
-For more information on roles, see: https://docs.microsoft.com/en-us/azure/role-based-access-control/overview
-
-+-----------------------+
-| Mandatory variables   |
-+=======================+
-| AZURE_SUBSCRIPTION_ID |
-+-----------------------+
-| AZURE_CLIENT_ID       |
-+-----------------------+
-| AZURE_SECRET          |
-+-----------------------+
-| AZURE_TENANT          |
-+-----------------------+
-
-+-------------------------------------+
-| Optional Variables                  |
-+=====================================+
-| AZURE_REGION_NAME                   |
-+-------------------------------------+
-| AZURE_RESOURCE_GROUP                |
-+-------------------------------------+
-| AZURE_STORAGE_ACCOUNT               |
-+-------------------------------------+
-| AZURE_VM_DEFAULT_USER_NAME          |
-+-------------------------------------+
-| AZURE_PUBLIC_KEY_STORAGE_TABLE_NAME |
-+-------------------------------------+
-
-
-**Google**
-
-+------------------------+
-| Mandatory variables    |
-+========================+
-| GCE_SERVICE_CREDS_FILE |
-| or                     |
-| GCE_SERVICE_CREDS_DICT |
-+------------------------+
-
-+--------------------+
-| Optional Variables |
-+====================+
-| GCE_PROJECT_NAME   |
-+--------------------+
-| GCE_DEFAULT_ZONE   |
-+--------------------+
-| GCE_REGION_NAME    |
-+--------------------+
-
-Once the environment variables are set, you can create a connection as follows:
-
-.. code-block:: python
-
-    from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList
-
-    provider = CloudProviderFactory().create_provider(ProviderList.OPENSTACK, {})
+1. `Providing access credentials through a dictionary`_
+2. `Providing access credentials through environment variables`_
+3. `Providing access credentials in a CloudBridge config file`_
 
 
 Providing access credentials through a dictionary
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+-------------------------------------------------
 You can initialize a simple config as follows. The key names are the same
 as the environment variables, in lower case. Note that the config dictionary
 will override environment values.
@@ -138,6 +23,7 @@ will override environment values.
 
     from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList
 
+    ## For AWS
     config = {'aws_access_key' : '<your_access_key>',
               'aws_secret_key' : '<your_secret_key>'}
     provider = CloudProviderFactory().create_provider(ProviderList.AWS, config)
@@ -151,58 +37,230 @@ will override environment values.
               'azure_resource_group': '<your resource group>'}
     provider = CloudProviderFactory().create_provider(ProviderList.AZURE, config)
 
+
+    ## For GCE
+    config = {'gce_service_creds_file': '<service_creds_file_name>.json'}
+    # Alternatively, we can supply a dictionary with the credentials values
+    # shown on the access credentials procurement page.
+    config = {'gce_service_creds_dict': credentials_dictionary}
+    provider = CloudProviderFactory().create_provider(ProviderList.GCE, config)
+
+
+    ## For OpenStack
+    config = {'os_username': '<your username>',
+              'os_password': '<your password>',
+              'os_auth_url': '<auth url>,
+              'os_user_domain_name': '<user_domain_name>',
+              'os_project_domain_name': '<project_domain_name>',
+              'os_project_name': '<project_name>')
+    provider = CloudProviderFactory().create_provider(ProviderList.OPENSTACK, config)
+
 Some optional configuration values can only be provided through the config
 dictionary. These are listed below for each provider.
 
-
-**CloudBridge**
+CloudBridge
+~~~~~~~~~~~
 
 +----------------------+------------------------------------------------------------+
-| Variable		       | Description                                                |
+| Variable             | Description                                                |
 +======================+============================================================+
 | default_result_limit | Number of results that a ``.list()`` method should return. |
-|                      | Defaults to 50.                                            |
+|                      | Default is 50.                                             |
 +----------------------+------------------------------------------------------------+
 
-
-**Amazon**
+AWS
+~~~
 
 +---------------------+--------------------------------------------------------------+
-| Variable		      | Description		      	      	      	      	      	     |
+| Variable            | Description                                                  |
 +=====================+==============================================================+
-| aws_session_token   | Session key for your AWS account (if using temporary   	     |
-|                     | credentials).   	      	      	      	      	      	 |
+| aws_session_token   | Session key for your AWS account (if using temporary         |
+|                     | credentials).                                                |
 +---------------------+--------------------------------------------------------------+
-| ec2_is_secure       | True to use an SSL connection. Default is ``True``.   	     |
+| ec2_conn_path	      | Connection path. Default is ``/``.                           |
 +---------------------+--------------------------------------------------------------+
-| ec2_region_name     | Default region name. Defaults to ``us-east-1``.   	       	 |
+| ec2_is_secure       | True to use an SSL connection. Default is ``True``.          |
++---------------------+--------------------------------------------------------------+
+| ec2_port            | EC2 connection port. Does not need to be specified unless    |
+|                     | EC2 service is running on an alternative port.               |
 +---------------------+--------------------------------------------------------------+
 | ec2_region_endpoint | Endpoint to use. Default is ``ec2.us-east-1.amazonaws.com``. |
 +---------------------+--------------------------------------------------------------+
-| ec2_port            | EC2 connection port. Does not need to be specified unless    |
-|                     | EC2 service is running on an alternative port.   	       	 |
+| ec2_region_name     | Default region name. Default is ``us-east-1``.               |
 +---------------------+--------------------------------------------------------------+
-| ec2_conn_path	      | Connection path. Defaults to ``/``.   	       	      	     |
+| ec2_validate_certs  | Whether to use SSL certificate verification. Default is      |
+|                     | ``False``.                                                   |
 +---------------------+--------------------------------------------------------------+
-| ec2_validate_certs  | Whether to use SSL certificate verification. Default is   	 |
-|                     | ``False``.   	       	      	      	      	      	     |
+| s3_conn_path        | Connection path. Default is ``/``.                           |
 +---------------------+--------------------------------------------------------------+
-| s3_is_secure        | True to use an SSL connection. Default is ``True``.   	     |
+| s3_is_secure        | True to use an SSL connection. Default is ``True``.          |
 +---------------------+--------------------------------------------------------------+
 | s3_host             | Host connection endpoint. Default is ``s3.amazonaws.com``.   |
 +---------------------+--------------------------------------------------------------+
 | s3_port             | Host connection port. Does not need to be specified unless   |
-|                     | S3 service is running on an alternative port.   	         |
-+---------------------+--------------------------------------------------------------+
-| s3_conn_path        | Connection path. Defaults to ``/``.   	                     |
+|                     | S3 service is running on an alternative port.                |
 +---------------------+--------------------------------------------------------------+
-| s3_validate_certs   | Whether to use SSL certificate verification. Default is   	 |
-|                     | ``False``.   	                                             |
+| s3_validate_certs   | Whether to use SSL certificate verification. Default is      |
+|                     | ``False``.                                                   |
 +---------------------+--------------------------------------------------------------+
 
+Azure
+~~~~~
+
++-------------------------------------+----------------------------------------------------------+
+| Variable                            | Description                                              |
++=====================================+==========================================================+
+| azure_access_token                  | To sign requests to APIs protected by Azure.             |
++-------------------------------------+----------------------------------------------------------+
+| azure_public_key_storage_table_name | Storage table name where the key pairs are stored.       |
+|                                     | Default is ``cbcerts``.                                  |
++-------------------------------------+----------------------------------------------------------+
+| azure_region_name                   | Default region to use for the current                    |
+|                                     | session. Default is ``eastus``.                          |
++-------------------------------------+----------------------------------------------------------+
+| azure_resource_group                | Azure resource group to use. Default is ``cloudbridge``. |
++-------------------------------------+----------------------------------------------------------+
+| azure_storage_account               | Azure storage account to use. Note that this value must  |
+|                                     | be unique across Azure and all data in a given session   |
+|                                     | is stored within the supplied storage account. Default   |
+|                                     | ``storacc`` + first 6 chars of subscription id + first 6 |
+|                                     | chars of the supplied resource group.                    |
++-------------------------------------+----------------------------------------------------------+
+| azure_vm_default_username           | System user name for which supplied key pair will be     |
+|                                     | placed.                                                  |
++-------------------------------------+----------------------------------------------------------+
+
+GCE
+~~~
+
++-------------------------+----------------------------------------------------------+
+| Variable                | Description                                              |
++=========================+==========================================================+
+| gce_default_zone        | Default placement zone to use for the current session.   |
+|                         | Default is ``us-central1-a``.                            |
++-------------------------+----------------------------------------------------------+
+| gce_region_name         | Default region to use for the current session. Default   |
+|                         | is ``us-central1``.                                      |
++-------------------------+----------------------------------------------------------+
+| gce_vm_default_username | System user name for which supplied key pair will be     |
+|                         | placed.                                                  |
++-------------------------+----------------------------------------------------------+
+
+
+Providing access credentials through environment variables
+----------------------------------------------------------
+The following environment variables must be set, depending on the provider in
+use. For the meaning of the variables and default values, see the descriptions
+above.
+
+AWS
+~~~
+
++---------------------+------------+
+| Variable            | Required?  |
++=====================+============+
+| AWS_ACCESS_KEY      | ✔          |
++---------------------+------------+
+| AWS_SECRET_KEY      | ✔          |
++---------------------+------------+
 
-Providing access credentials in a file
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Azure
+~~~~~
+
+Note that managing resources in Azure requires a Resource Group. If a
+Resource Group is not provided as part of the configuration, CloudBridge will
+attempt to create a Resource Group using the given credentials. This
+operation will happen with the client initialization, and requires a
+"contributor" or "owner" role.
+
+Similarly, a Storage Account is required when managing some resources, such
+as key pairs and buckets. If a Storage Account name is not provided as part
+of the configuration, CloudBridge will attempt to create the Storage Account
+when initializing the relevant services. This operation similarly requires a
+"contributor" or "owner" role.
+
+For more information on roles, see
+https://docs.microsoft.com/en-us/azure/role-based-access-control/overview.
+
++-------------------------------------+-----------+
+| Variable                            | Required? |
++=====================================+===========+
+| AZURE_CLIENT_ID                     | ✔         |
++-------------------------------------+-----------+
+| AZURE_SECRET                        | ✔         |
++-------------------------------------+-----------+
+| AZURE_SUBSCRIPTION_ID               | ✔         |
++-------------------------------------+-----------+
+| AZURE_TENANT                        | ✔         |
++-------------------------------------+-----------+
+| AZURE_PUBLIC_KEY_STORAGE_TABLE_NAME |           |
++-------------------------------------+-----------+
+| AZURE_REGION_NAME                   |           |
++-------------------------------------+-----------+
+| AZURE_RESOURCE_GROUP                |           |
++-------------------------------------+-----------+
+| AZURE_STORAGE_ACCOUNT               |           |
++-------------------------------------+-----------+
+| AZURE_VM_DEFAULT_USER_NAME          |           |
++-------------------------------------+-----------+
+
+GCE
+~~~
+
++------------------------+-----------+
+| Variable               | Required? |
++========================+===========+
+| GCE_SERVICE_CREDS_DICT | ✔         |
+| or                     |           |
+| GCE_SERVICE_CREDS_FILE |           |
++------------------------+-----------+
+| GCE_DEFAULT_ZONE       |           |
++------------------------+-----------+
+| GCE_PROJECT_NAME       |           |
++------------------------+-----------+
+| GCE_REGION_NAME        |           |
++------------------------+-----------+
+
+OpenStack
+~~~~~~~~~
+
++------------------------+-----------+
+| Variable               | Required? |
++========================+===========+
+| OS_AUTH_URL            | ✔         |
++------------------------+-----------+
+| OS_USERNAME            | ✔         |
++------------------------+-----------+
+| OS_PASSWORD            | ✔         |
++------------------------+-----------+
+| OS_PROJECT_NAME        | ✔         |
++------------------------+-----------+
+| OS_REGION_NAME         | ✔         |
++------------------------+-----------+
+| NOVA_SERVICE_NAME      |           |
++------------------------+-----------+
+| OS_AUTH_TOKEN          |           |
++------------------------+-----------+
+| OS_COMPUTE_API_VERSION |           |
++------------------------+-----------+
+| OS_VOLUME_API_VERSION  |           |
++------------------------+-----------+
+| OS_STORAGE_URL         |           |
++------------------------+-----------+
+
+Once the environment variables are set, you can create a connection as follows,
+replacing ``ProviderList.AWS`` with the desired provider (AZURE, GCE, or
+OPENSTACK):
+
+.. code-block:: python
+
+    from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList
+
+    provider = CloudProviderFactory().create_provider(ProviderList.AWS, {})
+
+
+Providing access credentials in a CloudBridge config file
+---------------------------------------------------------
 CloudBridge can also read credentials from a file on your local file system.
 The file should be placed in one of two locations: ``/etc/cloudbridge.ini`` or
 ``~/.cloudbridge``. Each set of credentials should be delineated with the
@@ -214,6 +272,20 @@ OpenStack clouds).
 
 .. code-block:: bash
 
+    [aws]
+    aws_access_key: access key
+    aws_secret_key: secret key
+
+    [azure]
+    azure_subscription_id: subscription id
+    azure_tenant: tenant
+    azure_client_id: client id
+    azure_secret: secret
+    azure_resource_group: resource group
+
+    [gce]
+    gce_service_creds_file: absolute path to credentials file
+
     [openstack]
     os_username: username
     os_password: password
@@ -222,22 +294,27 @@ OpenStack clouds).
     os_project_domain_name: project domain name
     os_project_name: project name
 
-    [aws]
-    aws_access_key: access key
-    aws_secret_key: secret key
+Once the file is created, you can create a connection as follows, replacing
+``ProviderList.AWS`` with the desired provider (AZURE, GCE, or OPENSTACK):
+
+.. code-block:: python
+
+    from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList
+
+    provider = CloudProviderFactory().create_provider(ProviderList.AWS, {})
 
 
-Other configuration variables
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+General configuration variables
+-------------------------------
 In addition to the provider specific configuration variables above, there are
 some general configuration environment variables that apply to CloudBridge as
-a whole
+a whole.
 
 +-----------------------------+------------------------------------------------------+
 | Variable                    | Description                                          |
 +=============================+======================================================+
 | CB_DEBUG                    | Setting ``CB_DEBUG=True`` will cause detailed        |
-|                             | debugoutput to be printed for each provider          |
+|                             | debug output to be printed for each provider         |
 |                             | (including HTTP traces).                             |
 +-----------------------------+------------------------------------------------------+
 | CB_USE_MOCK_PROVIDERS       | Setting this to ``True`` will cause the CloudBridge  |

setup.py

@@ -19,7 +19,6 @@ with open(os.path.join('cloudbridge', '__init__.py')) as f:
             break
 
 REQS_BASE = [
-    'bunch>=1.0.1',
     'six>=1.11',
     'tenacity>=4.12.0,<=5.0',
     'cachetools>=2.1.0',