1. Context

代码：rally-openstack/rally_openstack/contexts/cleanup/user.py

@validation.add(name="check_cleanup_resources", admin_required=False)
# NOTE(amaretskiy): Set maximum order to run this last
@context.configure(name="cleanup", platform="openstack", order=sys.maxsize - 1,
                   hidden=True)
class UserCleanup(base.CleanupMixin, context.Context):
    """Context class for user resources cleanup."""

    def cleanup(self):
        manager.cleanup(
            names=self.config,
            admin_required=False,
            users=self.context.get("users", []),
            api_versions=self.context["config"].get("api_versions"),
            superclass=scenario.OpenStackScenario,
            task_id=self.get_owner_id()
        )

代码：rally-openstack/rally_openstack/contexts/cleanup/admin.py

@validation.add(name="check_cleanup_resources", admin_required=True)
# NOTE(amaretskiy): Set order to run this just before UserCleanup
@context.configure(name="admin_cleanup", platform="openstack",
                   order=(sys.maxsize - 2), hidden=True)
class AdminCleanup(base.CleanupMixin, context.Context):
    """Context class for admin resources cleanup."""

    def cleanup(self):
        manager.cleanup(
            names=self.config,
            admin_required=True,
            admin=self.context["admin"],
            users=self.context.get("users", []),
            api_versions=self.context["config"].get("api_versions"),
            superclass=scenario.OpenStackScenario,
            task_id=self.get_owner_id())

代码：rally-openstack/rally_openstack/contexts/cleanup/base.py

class CleanupMixin(object):

    CONFIG_SCHEMA = {
        "type": "array",
        "$schema": consts.JSON_SCHEMA,
        "items": {
            "type": "string",
        }
    }

    def setup(self):
        pass

上述这些代码，可以看出：

• context实现了cleanup方法（setup方法也实现了，但什么也没做）
• rally的cleanup分为cleanup和admin_cleanup，区别在于：cleaup只清理admin_required==False的资源，admin_cleanup只清理admin_required==True的资源；cleanup的order更大，因此比admin_cleanup先执行；
• 具体清理是调用cleanup.manager.cleanup

2. Cleaup Manager

2.1 cleaup方法

代码：rally-openstack/rally_openstack/cleanup/manager.py

def cleanup(names=None, admin_required=None, admin=None, users=None,
            api_versions=None, superclass=plugin.Plugin, task_id=None):
    """Generic cleaner.

    This method goes through all plugins. Filter those and left only plugins
    with _service from services or _resource from resources.

    Then goes through all passed users and using cleaners cleans all related
    resources.

    :param names: Use only resource managers that have names in this list.
                  There are in as _service or
                  (%s.%s % (_service, _resource)) from
    :param admin_required: If None -> return all plugins
                           If True -> return only admin plugins
                           If False -> return only non admin plugins
    :param admin: rally.deployment.credential.Credential that corresponds to
                  OpenStack admin.
    :param users: List of OpenStack users that was used during testing.
                  Every user has next structure:
                  {
                    "id": <uuid1>,
                    "tenant_id": <uuid2>,
                    "credential": <rally.deployment.credential.Credential>
                  }
    :param superclass: The plugin superclass to perform cleanup
                       for. E.g., this could be
                       ``rally.task.scenario.Scenario`` to cleanup all
                       Scenario resources.
    :param task_id: The UUID of task
    """
    resource_classes = [cls for cls in discover.itersubclasses(superclass)
                        if issubclass(cls, rutils.RandomNameGeneratorMixin)]
    if not resource_classes and issubclass(superclass,
                                           rutils.RandomNameGeneratorMixin):
        resource_classes.append(superclass)
    for manager in find_resource_managers(names, admin_required):
        LOG.debug("Cleaning up %(service)s %(resource)s objects"
                  % {"service": manager._service,
                     "resource": manager._resource})
        SeekAndDestroy(manager, admin, users,
                       api_versions=api_versions,
                       resource_classes=resource_classes,
                       task_id=task_id).exterminate()

上述代码展示了cleanup的过程：

• 查找OpenStackScenario的子类，该类须是rutils.RandomNameGeneratorMixin的子类，一般这些类定义在rally-openstack/rally_openstack/scenarios目录里，这些类的run方法定义了测试过程
• 查找ResourceManager的子类，这些类定义在rally-openstack/rally_openstack/cleanup/resources.py文件中，这些类用来定义某一类资源的具体删除方法等，查找过程见下面的说明
• 根据查找到的资源控制类，实例化SeekAndDestroy，通过其exterminate方法来清理资源

 2.2 find_resource_managers方法

代码：rally-openstack/rally_openstack/cleanup/manager.py

def find_resource_managers(names=None, admin_required=None):
    """Returns resource managers.

    :param names: List of names in format <service> or <service>.<resource>
                  that is used for filtering resource manager classes
    :param admin_required: None -> returns all ResourceManagers
                           True -> returns only admin ResourceManagers
                           False -> returns only non admin ResourceManagers
    """
    names = set(names or [])

    resource_managers = []
    for manager in discover.itersubclasses(base.ResourceManager):
        if admin_required is not None:
            if admin_required != manager._admin_required:
                continue

        if (manager._service in names
           or "%s.%s" % (manager._service, manager._resource) in names):
            resource_managers.append(manager)

    resource_managers.sort(key=lambda x: x._order)

    found_names = set()
    for mgr in resource_managers:
        found_names.add(mgr._service)
        found_names.add("%s.%s" % (mgr._service, mgr._resource))

    missing = names - found_names
    if missing:
        LOG.warning("Missing resource managers: %s" % ", ".join(missing))

    return resource_managers

上述代码展示了资源控制类的查找过程：

• 查找ResourceManager的子类，这些类定义在rally-openstack/rally_openstack/cleanup/resources.py文件中
• 根据参数admin_required过滤子类，_admin_required属性在资源控制类定义时，通过装饰器赋值，例如

@base.resource("cinder", "volume_types", order=next(_cinder_order),
               admin_required=True, perform_for_admin_only=True)
class CinderVolumeType(base.ResourceManager):
    pass

• 根据上下文配置参数中的资源名称，继续过滤子类，_service和_resource属性在资源控制类定义时，通过装饰器赋值，例如上面的代码，"cinder"为_service属性，"volume_types"为_resource属性
• 按照_order属性对资源进行排序，即order越小越先清理，_order属性的赋值方法同上

 ps：下面是装饰器base.resource的定义：

代码：rally-openstack/rally_openstack/cleanup/base.py

def resource(service, resource, order=0, admin_required=False,
             perform_for_admin_only=False, tenant_resource=False,
             max_attempts=3, timeout=CONF.openstack.resource_deletion_timeout,
             interval=1, threads=CONF.openstack.cleanup_threads):
    """Decorator that overrides resource specification.

    Just put it on top of your resource class and specify arguments that you
    need.

    :param service: It is equal to client name for corresponding service.
                    E.g. "nova", "cinder" or "zaqar"
    :param resource: Client manager name for resource. E.g. in case of
                     nova.servers you should write here "servers"
    :param order: Used to adjust priority of cleanup for different resource
                  types
    :param admin_required: Admin user is required
    :param perform_for_admin_only: Perform cleanup for admin user only
    :param tenant_resource: Perform deletion only 1 time per tenant
    :param max_attempts: Max amount of attempts to delete single resource
    :param timeout: Max duration of deletion in seconds
    :param interval: Resource status pooling interval
    :param threads: Amount of threads (workers) that are deleting resources
                    simultaneously
    """

    def inner(cls):
        # TODO(boris-42): This can be written better I believe =)
        cls._service = service
        cls._resource = resource
        cls._order = order
        cls._admin_required = admin_required
        cls._perform_for_admin_only = perform_for_admin_only
        cls._max_attempts = max_attempts
        cls._timeout = timeout
        cls._interval = interval
        cls._threads = threads
        cls._tenant_resource = tenant_resource

        return cls

    return inner

2.3 SeekAndDestroy

该类定义了如何查找创建的资源及删除的方法，通过生产者/消费者模型进行构建

代码：rally-openstack/rally_openstack/cleanup/manager.py

class SeekAndDestroy(object):

    def __init__(self, manager_cls, admin, users, api_versions=None,
                 resource_classes=None, task_id=None):
        """Resource deletion class.

        This class contains method exterminate() that finds and deletes
        all resources created by Rally.

        :param manager_cls: subclass of base.ResourceManager
        :param admin: admin credential like in context["admin"]
        :param users: users credentials like in context["users"]
        :param api_versions: dict of client API versions
        :param resource_classes: Resource classes to match resource names
                                 against
        :param task_id: The UUID of task to match resource names against
        """
        self.manager_cls = manager_cls
        self.admin = admin
        self.users = users or []
        self.api_versions = api_versions
        self.resource_classes = resource_classes or [
            rutils.RandomNameGeneratorMixin]
        self.task_id = task_id

    def _get_cached_client(self, user):
        """Simplifies initialization and caching OpenStack clients."""
        if not user:
            return None
        # NOTE(astudenov): Credential now supports caching by default
        return user["credential"].clients(api_info=self.api_versions)

    def _delete_single_resource(self, resource):
        """Safe resource deletion with retries and timeouts.

        Send request to delete resource, in case of failures repeat it few
        times. After that pull status of resource until it's deleted.

        Writes in LOG warning with UUID of resource that wasn't deleted

        :param resource: instance of resource manager initiated with resource
                         that should be deleted.
        """

        msg_kw = {
            "uuid": resource.id(),
            "name": resource.name() or "",
            "service": resource._service,
            "resource": resource._resource
        }

        LOG.debug(
            "Deleting %(service)s.%(resource)s object %(name)s (%(uuid)s)"
            % msg_kw)

        try:
            rutils.retry(resource._max_attempts, resource.delete)
        except Exception as e:
            msg = ("Resource deletion failed, max retries exceeded for "
                   "%(service)s.%(resource)s: %(uuid)s.") % msg_kw

            if logging.is_debug():
                LOG.exception(msg)
            else:
                LOG.warning("%(msg)s Reason: %(e)s" % {"msg": msg, "e": e})
        else:
            started = time.time()
            failures_count = 0
            while time.time() - started < resource._timeout:
                try:
                    if resource.is_deleted():
                        return
                except Exception as e:
                    LOG.exception(
                        "Seems like %s.%s.is_deleted(self) method is broken "
                        "It shouldn't raise any exceptions."
                        % (resource.__module__, type(resource).__name__))

                    # NOTE(boris-42): Avoid LOG spamming in case of bad
                    #                 is_deleted() method
                    failures_count += 1
                    if failures_count > resource._max_attempts:
                        break

                finally:
                    rutils.interruptable_sleep(resource._interval)

            LOG.warning("Resource deletion failed, timeout occurred for "
                        "%(service)s.%(resource)s: %(uuid)s." % msg_kw)

    def _publisher(self, queue):
        """Publisher for deletion jobs.

        This method iterates over all users, lists all resources
        (using manager_cls) and puts jobs for deletion.

        Every deletion job contains tuple with two values: user and resource
        uuid that should be deleted.

        In case of tenant based resource, uuids are fetched only from one user
        per tenant.
        """
        def _publish(admin, user, manager):
            try:
                for raw_resource in rutils.retry(3, manager.list):
                    queue.append((admin, user, raw_resource))
            except Exception:
                LOG.exception(
                    "Seems like %s.%s.list(self) method is broken. "
                    "It shouldn't raise any exceptions."
                    % (manager.__module__, type(manager).__name__))

        if self.admin and (not self.users
                           or self.manager_cls._perform_for_admin_only):
            manager = self.manager_cls(
                admin=self._get_cached_client(self.admin))
            _publish(self.admin, None, manager)

        else:
            visited_tenants = set()
            admin_client = self._get_cached_client(self.admin)
            for user in self.users:
                if (self.manager_cls._tenant_resource
                   and user["tenant_id"] in visited_tenants):
                    continue

                visited_tenants.add(user["tenant_id"])
                manager = self.manager_cls(
                    admin=admin_client,
                    user=self._get_cached_client(user),
                    tenant_uuid=user["tenant_id"])
                _publish(self.admin, user, manager)

    def _consumer(self, cache, args):
        """Method that consumes single deletion job."""
        admin, user, raw_resource = args

        manager = self.manager_cls(
            resource=raw_resource,
            admin=self._get_cached_client(admin),
            user=self._get_cached_client(user),
            tenant_uuid=user and user["tenant_id"])

        if (isinstance(manager.name(), base.NoName) or
                rutils.name_matches_object(
                    manager.name(), *self.resource_classes,
                    task_id=self.task_id, exact=False)):
            self._delete_single_resource(manager)

    def exterminate(self):
        """Delete all resources for passed users, admin and resource_mgr."""

        broker.run(self._publisher, self._consumer,
                   consumers_count=self.manager_cls._threads)

方法_publisher定义了如何查找资源：

• 第115行和第128行：实例化某一个具体的资源控制类，比如servers
• 第105，106行：使用资源控制类的list方法，来列出各个用户创建的资源，ps： list方法一般是调用各个组件的list方法
• 将列出的资源存入队列

方法_consumer定义了如何删除资源：

• 第138-142行：实例化某一具体的资源控制类（从队列中取出资源，并赋值）；
• 第144-147行：根据资源的命名规则过滤资源的名称； ps：或者名称是NoName的实例
• 调用资源控制类的delete方法来删除资源，一般是调用各个组件的delete方法，同时会检查是否删除成功

3. ResourceManager

rally cleanup是通过这些ResourceManager类来发现和清理资源的，对于某一类资源需要定义一个ResourceManager来对应，比如 nova.servers，这些类定义在rally-openstack/rally_openstack/cleanup/resources.py文件中

3.1 基类ResourceManager

代码：rally-openstack/rally_openstack/cleanup/base.py

@resource(service=None, resource=None)
class ResourceManager(object):
    """Base class for cleanup plugins for specific resources.

    You should use @resource decorator to specify major configuration of
    resource manager. Usually you should specify: service, resource and order.

    If project python client is very specific, you can override delete(),
    list() and is_deleted() methods to make them fit to your case.
    """

    def __init__(self, resource=None, admin=None, user=None, tenant_uuid=None):
        self.admin = admin
        self.user = user
        self.raw_resource = resource
        self.tenant_uuid = tenant_uuid

    def _manager(self):
        client = self._admin_required and self.admin or self.user
        return getattr(getattr(client, self._service)(), self._resource)

    def id(self):
        """Returns id of resource."""
        return self.raw_resource.id

    def name(self):
        """Returns name of resource."""
        return self.raw_resource.name

    def is_deleted(self):
        """Checks if the resource is deleted.

        Fetch resource by id from service and check it status.
        In case of NotFound or status is DELETED or DELETE_COMPLETE returns
        True, otherwise False.
        """
        try:
            resource = self._manager().get(self.id())
        except Exception as e:
            return getattr(e, "code", getattr(e, "http_status", 400)) == 404

        return utils.get_status(resource) in ("DELETED", "DELETE_COMPLETE")

    def delete(self):
        """Delete resource that corresponds to instance of this class."""
        self._manager().delete(self.id())

    def list(self):
        """List all resources specific for admin or user."""
        return self._manager().list()

资源控制类需要继承base.ResourceManager，另外需要注意以下几点：

• 需要使用装饰器resource来设置属性，比如service，resource，order，admin_required等，这些属性前面已经说明
• 根据具体资源的情况，来重写方法list，delete，is_deleted，list用来查找资源，delete用来删除资源，is_deleted用来判断资源是否成功删除

3.2 资源删除顺序

资源在删除时，需要注意顺序，我们以neutron组件来看一下各个资源的删除

从代码中我们可以看出，neutron服务的删除顺序为：

bgpvpn ---> floatingip ---> port ---> subnet ---> network ---> router ---> security_group ---> quota

另外在删除资源时，需要注意与其他资源的关联情况，当删除subnet，network，router时，需要注意先remove port资源，下面看一下port资源的删除

代码：rally-openstack/rally_openstack/cleanup/resources.py

@base.resource("neutron", "port", order=next(_neutron_order),
               tenant_resource=True)
class NeutronPort(NeutronMixin):
    # NOTE(andreykurilin): port is the kind of resource that can be created
    #   automatically. In this case it doesn't have name field which matches
    #   our resource name templates.
    ROUTER_INTERFACE_OWNERS = ("network:router_interface",
                               "network:router_interface_distributed",
                               "network:ha_router_replicated_interface")

    ROUTER_GATEWAY_OWNER = "network:router_gateway"

    def __init__(self, *args, **kwargs):
        super(NeutronPort, self).__init__(*args, **kwargs)
        self._cache = {}

    def _get_resources(self, resource):
        if resource not in self._cache:
            resources = getattr(self._manager(), "list_%s" % resource)()
            self._cache[resource] = [r for r in resources[resource]
                                     if r["tenant_id"] == self.tenant_uuid]
        return self._cache[resource]

    def list(self):
        ports = self._get_resources("ports")
        for port in ports:
            if not port.get("name"):
                parent_name = None
                if (port["device_owner"] in self.ROUTER_INTERFACE_OWNERS or
                        port["device_owner"] == self.ROUTER_GATEWAY_OWNER):
                    # first case is a port created while adding an interface to
                    #   the subnet
                    # second case is a port created while adding gateway for
                    #   the network
                    port_router = [r for r in self._get_resources("routers")
                                   if r["id"] == port["device_id"]]
                    if port_router:
                        parent_name = port_router[0]["name"]
                if parent_name:
                    port["parent_name"] = parent_name
        return ports

    def name(self):
        return self.raw_resource.get("parent_name",
                                     self.raw_resource.get("name", ""))

    def delete(self):
        device_owner = self.raw_resource["device_owner"]
        if (device_owner in self.ROUTER_INTERFACE_OWNERS or
                device_owner == self.ROUTER_GATEWAY_OWNER):
            if device_owner == self.ROUTER_GATEWAY_OWNER:
                self._manager().remove_gateway_router(
                    self.raw_resource["device_id"])

            self._manager().remove_interface_router(
                self.raw_resource["device_id"], {"port_id": self.id()})
        else:
            from neutronclient.common import exceptions as neutron_exceptions

            try:
                self._manager().delete_port(self.id())
            except neutron_exceptions.PortNotFoundClient:
                # Port can be already auto-deleted, skip silently
                LOG.debug("Port %s was not deleted. Skip silently because "
                          "port can be already auto-deleted." % self.id())

另外，也需要注意各个服务之间的删除顺序，cleanup的删除顺序为：

nova ---> neutron ---> cinder ---> glance ---> keystone