OpenStack: Find Orphaned Images in Glance

Hello All – today’s post is on identifying and removing Glance images that are truly not used anywhere, such as when deleting a project (tenant) without first deleting private images within that project. Check out our script!

For the impatient: Here is the link to the updated script: os-glance-swift-crosscheck-sh.txt. Rename to .sh and run it.

In my lab, I create and delete OpenStack tenants (projects) all the time. And I generally have images I upload to those projects…so it occurred to me to ask: “What happens to my tenant-specific projects when I delete the tenant?”

If you work with OpenStack, you already know the answer: Nothing. The images are just left there. To take up space. Forever. And while you could go into the MySQL database and muck with columns to set ownership GUIDs…I *hate* that approach. Drives me nuts.

Regardless of whether you plan to delete orphaned images and reclaim space or muck with database tables and (in effect) dynamically reassign an orphaned image, you still gotta know if they exist. Enter today’s script, which runs against my backing OpenStack Swift storage nodes and uses that information to correlate against OpenStack Glance. A future article will discuss OpenStack Storage Alternatives and at some point I’ll extend the script to handle the Ceph (rbd devices) in my lab; but I probably will never extend the script to handle Glance “file-backed” storage. You’d have to run the script on the Glance host…and what about Glance High Availability? (Which I will cover in a whole series of articles in the future.)

Let’s take a look at how you would identify orphaned images. The first step is to find all your underlying images so you have a reference point. The following shows this for Swift storage:

[l.abruce@co1 rc_scripts(lvosksclu100-rc-admin)]$ swift --os-tenant-name=service list glance | cut -d'-' -f 1-5 | uniq
10e39389-e1c0-4f8c-b62a-39b3bd466f58
141cdb83-739f-445e-b5a5-5452fd6bffd7
17eb9711-53ec-4076-9b71-cbe86e527a36
1b5693e5-e83d-491e-9754-d6d343e0966a
1d338db3-e74b-4bda-a0e2-eedd294127af
23eae911-eb5e-44e1-a7f5-541730895284
2c108578-079e-445a-9775-e2c0f2c9bdc6
33c6b3b1-77d6-4a62-a6bc-fc7eb52c4cba
3564bda8-d649-4b4a-908e-20fb83e6d774
6440e596-5cfe-4e42-b527-ee252f95991d
65c87bae-d096-4061-8bef-de9943b36934
805e79e5-6b97-41e6-8002-6946b2c03b2f
80afe620-b668-49e8-921b-5c41981854da
ce343a10-3dbd-4035-8fe4-5632c5c31147
e51a46cd-5218-478d-952a-860b3190bfa6
eb836e22-ca74-459c-99ca-5f288382f8e3
f28f5ac8-72c5-4be9-adb3-0736daf343e9
f6ab2e65-37ef-44cd-a52f-6b8abc825703
f91e19b0-ee6c-4c32-9537-5a2ab9bca34d
fb8a85e3-1ea2-415c-86d6-cacd2c99425c
fe75d026-3385-4f0e-9592-a402cae5a034

Now that we have all of our images, we need to correlate each one against Glance. Let’s pick one at random and do this:

[l.abruce@co1 rc_scripts(lvosksclu100-rc-admin)]$ glance image-show fb8a85e3-1ea2-415c-86d6-cacd2c99425c
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | c1ac2a09f7f662d3b191476741216d3d     |
| container_format | bare                                 |
| created_at       | 2014-09-26T17:55:58                  |
| deleted          | False                                |
| disk_format      | qcow2                                |
| id               | fb8a85e3-1ea2-415c-86d6-cacd2c99425c |
| is_public        | True                                 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | 500MbSwift                           |
| owner            | b4ecec89c305404c90c16d79511376a7     |
| protected        | False                                |
| size             | 502480000                            |
| status           | active                               |
| updated_at       | 2014-09-26T17:57:38                  |
+------------------+--------------------------------------+

This tells us that the image is in the Glance repository, that it has an owner (b4ecec89c305404c90c16d79511376a7), and that it is active. That’s a good thing, so let’s write the loop to handle that (given all image IDs in a variable):

l_tmp="/tmp/$$.tmp"
for i in $l_swift_images; do
  # get info
  if glance image-show $i > $l_tmp 2>/dev/null; then
    # extract what we want
    l_image_is_orphan='False'
    l_image_deleted=$(cat $l_tmp | grep -e "| deleted " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
    l_image_is_public=$(cat $l_tmp | grep -e "| is_public " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
    l_image_name=$(cat $l_tmp | grep -e "| name " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
    l_image_owner=$(cat $l_tmp | grep -e "| owner " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
    l_image_protected=$(cat $l_tmp | grep -e "| protected " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
    l_image_size=$(cat $l_tmp | grep -e "| size " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
    l_image_status=$(cat $l_tmp | grep -e "| status " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
  fi

  # show the line
  echo "$l_image_is_orphan|$l_image_tenant ($l_image_owner)|$l_image_name ($i)|$l_image_size|$l_image_status|$l_image_deleted|$l_image_is_public|$l_image_protected"
done
rm -f $l_tmp

So the above is good, but it doesn’t account for missing owners (OpenStack “projects”). Let’s add some logic that will check for all the conditions we can think of:

    # get the tenant
    if [ -n "$l_image_owner" ]; then
      # do this way to prevent error messages
      keystone tenant-get $l_image_owner 2>/dev/null >$l_tmp
      l_rc=$?
      if [ $l_rc -eq 0 ]; then
        # the tenant is valid; get the name
        l_image_tenant=$(cat $l_tmp  | grep -e "| \+name " | cut -d'|' -f 3 | sed -e 's#^ \+##; s#[ \t]*$##' 2>&1)
      else
        # we have an orphan; project not found
        l_image_is_orphan='True'
        l_image_tenant="[n/a] ($l_image_owner)"
      fi
    else
      l_image_tenant="none ([n/a])"
    fi

Now that we have the pieces, we can build the complete script. It is included for you above, it walks all of the Swift images and validates them against Glance. Let’s take a look at my output so we can see how I identified my own orphaned images:

Orphan|Tenant|Name|Size|Status|Deleted|Is_Public|Protected
False|none ([n/a]) ()|DemoTenant (5f9427a1-a978-4db0-8e9a-ca81ae13e213)|1375|active|False|False|False
True|[n/a] (32380da42cdf499caf31c5e5a085107f) (32380da42cdf499caf31c5e5a085107f)|redmine.qcow2 (df26f521-d6a3-40be-af0a-f36675c6971d)|12941000704|active|False|False|False

The above shows two problems: the first line was created without an owner (oh, I have much more to say about that fiasco!) and the second line indicates an image that has an invalid owner (namely, a tenant I deleted). So a couple of quick calls to glance image-delete and I’m all clean again.

Enjoy, and Happy Computing!