These examples are meant to clarify the specification of the Shaft. Examples can never be complete, so the specification is of course leading as the only proper source of constraints.

See also:

Subtree Entries derived from Upstream Sources

Start Empty

Initially, Shaft presents an empty service rooted at "", so any sub-realm can still be added.

An operator can login to a web interface, which presents an HTML / JavaScript wrapper into which backends for Crank and Shaft components can be "plugged in". We assume that (at least) a single Shaft has been plugged into the interface and is being modified in these examples.

As a running example, we'll be following package listings from Linux distributions.

For now, all LDAP queries to its contents are answered with "object not found".

Create a Tree

All output from Shaft shown under one tree. The operator creates one such tree, basically setting the baseDN for all output. Listening Pulley (or Shaft) instances can use this baseDN to query the Shaft.

Usually, there will be some setup that needs to go into the root. This step would setup that aspect.

We will assume that the Shaft is setup with baseDN ou=Package Listers.

The Shaft now answers to queries for

  • ou=Package Listers

Add a few Cranks

The operator now adds two Cracks, to follow the LDAP information published by these sources. The two sources do not overlap.

For the first and second, the operator enters a full LDAP URLs,

  • ldap://packages.redhat.com/o=RedHat,c=US
  • ldap://packages.suse.com/o=SuSe,c=DE

This retrieves objects under specific root, namely o=RedHat,c=US and o=SuSe,c=DE which will be relocated to a local Shaft output node that the administrator sets up, for instance cn=RedHat RHEL and cn=SuSe. The latter are RDNsequences that must fall under our local root, ou=Package Listers.

The second one is added as a number of separate settings:

  • Protocol ldaps
  • Hostname packages.debian.org, port 389
  • BaseDN cn=Wheezy,dc=debian,dc=org remote, becomes cn=Debian Wheezy, ou=Package Listers locally
  • All attributes, although we're mainly interested in pkgName,pkgVersion
  • No filter, although we're mainly interested in (objectClass=LinuxPackage)
  • Scope sub

Note that we assume a shared schema between the various distributions; it is not the intention of this work to rewrite schema's, only their baseDN can be rewritten to come to a tree composed to a local standard.

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=RedHat RHEL, ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian Wheezy, ou=Package Listers

Manually remove an entry for an Upstream Crank or Shaft

Our subscription to RedHat is not completely useful; it turns out to have subtrees for distro=RHEL and distro=Fedora which should appear in our setup as separate cn=RedHat RHEL and cn=RedHat Fedora subtrees under our baseDN.

We will add a subscription to handle that further down. For now, we'll make some room:

  • The operator removes cn=RedHat RHEL from the local baseDN. This removes the subscription to o=RedHat,c=US.

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian Wheezy, ou=Package Listers

Subscribe to a List of Entries offered by an Upstream Crank or Shaft

We are now going to subscribe to a list of distro's that is available on the RedHat LDAP server. Subscriptions are dynamic in nature; when a site adds new entries that fall under our subscription, then it will be added automatically (following the subscription profile) and when an entry is removed it will also disappear without manual intervention.

The implementation of subscriptions is specific Shaft functionality, and not likely found in a standard package such as OpenLDAP (yet). It is implemented as a SyncRepl subscription, but not to entries but rather to some sort of list that can be used to construct new subtrees.

We will query RedHat's server for the (distro=*) entries and turn them into separate subtrees. The operator may add this in URI form,

  • ldap://packages.redhat.com/o=RedHat,c=US??sub?(distro=*)

The local entry follows a pattern, that quotes an attribute and puts it into a cn field:

  • The local baseDN of found entries will be cn=RedHat %(attr.distro)s, ou=Package Listers

The shown quoting notation is taken from Python. In general, names from attributes and the RDNs should be available. When there are multiple, it is possible to run multiple of these substitutions, for example when an object has multiple distro attributes.

As soon as this is committed, Shaft subscribes to this query through SyncRepl, so it may immediately receive entries. It finds the following objects of interest:

  • distro:RedHat RHEL
  • distro:RedHat Fedora

In response to this, it creates nodes under the local baseDN following the rewriting pattern:

  • cn=RedHat RHEL, ou=Package Listers
  • cn=RedHat Fedora, ou=Package Listers

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian, ou=Package Listers
  • cn=RedHat RHEL, ou=Package Listers
  • cn=RedHat Fedora, ou=Package Listers

and in addition it listens in on the following subscription:

  • ldap://packages.redhat.com/o=RedHat,c=US??sub?(distro=*)

Subscribe to a second List of Upstream Crank or Shaft Entries

We now repeat the above operation, subscribing to a generic site for dpkg distributions. This time, the operator makes independent settings of the parameters that were in the LDAP URL in the last example:

  • Protocol ldaps
  • Hostname all.dpkg.org, port 389
  • BaseDN dc=all,dc=dpkg,dc=org remote, becomes cn=%(rdn1.cn)s %(attr.cn)s locally
  • Attributes cn
  • Filter ((objectClass=LinuxDistribution)(cn=*))
  • Scope sub

As soon as this is committed, Shaft subscribes to this query through SyncRepl, so it may immediately receive entries. It finds the following objects of interest:

  • cn:Wheezy in cn=Wheezy,cn=Debian,dc=all,dc=dpkg,dc=org
  • cn:Squeeze in cn=Squeeze,cn=Debian,dc=all,dc=dpkg,dc=org
  • cn:Kylin in cn=Kylin,cn=Ubuntu,dc=all,dc=dpkg,dc=org

In response to this, it automatically creates nodes under the local baseDN following the rewriting pattern:

  • cn=Debian Wheezy, ou=Package Listers
  • cn=Debian Squeeze, ou=Package Listers
  • cn=Ubuntu Kylin, ou=Package Listers

It will continue to do this when new entries pop up; it will remove such automatically generated entries when an old one appears.

But there is a problem! There already is an entry named cn=Debian Wheezy, ou=Package Listers. The newly found entry is added to a queue for this item, and is considered a potential replacement of the current item. The operator is informed over this clash through the monitoring interface. (Shaft should interface with monitoring; it might implement the AgentX protocol to plugin to SNMP infrastructure.) The queue is persistent; perhaps its entries are only visible to the Shaft internally but not to the Pulley and Shaft subscribers.

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian, ou=Package Listers
  • cn=RedHat RHEL, ou=Package Listers
  • cn=RedHat Fedora, ou=Package Listers
  • cn=Debian Wheezy, ou=Package Listers
  • Presented is what was entered first
  • There is a queue holding the newly found entry
  • This queue is only visible and manageable over the operators interface
  • The monitoring interface will report on this clash
  • cn=Debian Squeeze, ou=Package Listers
  • cn=Ubuntu Kylin, ou=Package Listers

and in addition it listens in on the following subscriptions for additions or removals:

  • ldap://packages.redhat.com/o=RedHat,c=US??sub?(distro=*)
  • ldaps://all.dpkg.org/dc=all,dc=dpkg,dc=org?cn?sub?(&(objectClass=LinuxDistribution)(cn=*))

The Operator considers Renaming a Subtree

The monitoring output from the clash for cn=Debian Wheezy, ou=Package Listers is an important hint to the Shaft operator. The clash may be due to overlap in naming but not meaning; in this case, the operator may overrule the naming for one of the entries. Note that this may be done with either the manually added one, or the automatically added one. The renaming choice will disappear only when the respective unit is removed.

The change should be made such that no tracking information is lost; so if a live, visible entry is renamed (the manually entered cn=Debian Wheezy, ou=Package Listers) then it would not lead to downtime of the information for this node, merely its location got changed.

We will not assume that the operator makes this change; the competing trees are in fact offering the same information. Furthermore, the manually added one is authoritative, so the secondary is rightfully behind it in the queue.

An Upstream Crank or Shaft goes down

When an upstream provider goes down, it should not impact the availability of data to Shaft subscribers. Indeed, the "last-known good" state will be served up.

When multiple servers exist to serve the resource, as per the SRV records setup for it, then they are tried as prescribed in SRV RFCs. This is usually handled by the LDAP server.

What will not happen, is that an entry under our local tree is removed automatically. So when more queued alternatives exist, they are still not used (because a matching name does not automatically imply that they are the same distributions).

So, even in downtime, the Shaft continues to serve the same content. The one thing that will happen is that monitoring will make mention of upstream Crank or Shaft servers that are down.

The Operator manually corrects Subscription Preference

The operator may manually override the selections made from a queue. This could be done for the subtree cn=Debian Squeeze, ou=Package Listers which now listens to the manually setup entry, and would then move to the automatic addition.

When making such a change, the old tree is in fact removed before the new one is inserted. Any cached entries would also be cleaned up or replaced with new content. Between the removal of the previous selection and the new one, there is bound to be some disruption in the offered information.

To avoid elongated downtime, some sanity checks on the new preferred entry is made before actually switching over; if it is currently down, the switch will be rejected with a notice to the operator; such a switch would erase all data and have nothing to replace it.

We assume that the operator is presented with queues showing up as a list, initially ordered by their additions, and later they may be overridden. Since the entry for cn=Debian Squeeze, ou=Package Listers is still down, this should release the alarms raised by monitoring systems.

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian, ou=Package Listers
  • cn=RedHat RHEL, ou=Package Listers
  • cn=RedHat Fedora, ou=Package Listers
  • cn=Debian Wheezy, ou=Package Listers
  • Presented is what was in the queue before, the automatic addition
  • There is a queue holding the manually entered item (which is down)
  • This queue is only visible and manageable over the operators interface
  • The monitoring interface will report on this clash
  • The monitoring interface will not report downtime of the queued item
  • cn=Debian Squeeze, ou=Package Listers
  • cn=Ubuntu Kylin, ou=Package Listers

and in addition it listens in on the following subscriptions for additions or removals:

  • ldap://packages.redhat.com/o=RedHat,c=US??sub?(distro=*)
  • ldaps://all.dpkg.org/dc=all,dc=dpkg,dc=org?cn?sub?(&(objectClass=LinuxDistribution)(cn=*))

A new Entry is Dynamically Added by a Subscription to an Upstream Crank or Shaft

After a long, long wait, there's a new Debian distribition! It is named Jessie, and shows up in our subscription to the all.dpkg.org directory service as a new entry. Since we track this subscription with SyncRepl, we receive this addition almost instantaneously, as

  • cn:Jessie in cn=Jessie,cn=Debian,dc=all,dc=dpkg,dc=org

In response to this, Shaft automatically creates a node under the local baseDN following the rewriting pattern cn=%(rdn1.cn)s %(attr.cn)s which leads to

  • cn=Debian Jessie, ou=Package Listers

This new entry is not in conflict with anything, so it is automatically made available without further ado.

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian, ou=Package Listers
  • cn=RedHat RHEL, ou=Package Listers
  • cn=RedHat Fedora, ou=Package Listers
  • cn=Debian Wheezy, ou=Package Listers
  • Presented is what was in the queue before, the automatic addition
  • There is a queue holding the manually entered item (which is down)
  • This queue is only visible and manageable over the operators interface
  • The monitoring interface will report on this clash
  • The monitoring interface will not report downtime of the queued item
  • cn=Debian Squeeze, ou=Package Listers
  • cn=Debian Jessie, ou=Package Listers
  • cn=Ubuntu Kylin, ou=Package Listers

and in addition it listens in on the following subscriptions for additions or removals:

  • ldap://packages.redhat.com/o=RedHat,c=US??sub?(distro=*)
  • ldaps://all.dpkg.org/dc=all,dc=dpkg,dc=org?cn?sub?(&(objectClass=LinuxDistribution)(cn=*))

A conflicting Entry is added by a Subscription

Conflicts can arise from either manual or automatic additions. Let's review the situation that a manual addition causes a conflict (which is resolved through queueing and monitoring notifications, as before). However, if the addition is made manually, then feedback is also provided interactively, with hints that help the operator to edit the queue and perhaps place the new entry first.

Our administrator has manually added an entry from dc=debian,dc=org for the Wheezy distribution, and initially preferred it over (placed it ahead in the queue from) the dc=all,dc=dpkg,dc=org subscription-launched entries.

He now wants to do the same for Jessie, seeing that the original service is alive and kicking. So he adds

  • Protocol ldaps
  • Hostname packages.debian.org, port 389
  • BaseDN cn=Jessie,dc=debian,dc=org remote, becomes cn=Debian Jessie, ou=Package Listers locally
  • All attributes, although we're mainly interested in pkgName,pkgVersion
  • No filter, although we're mainly interested in (objectClass=LinuxPackage)
  • Scope sub

The new node is now added to the ordered location,

  • cn=Debian Jessie, ou=Package Listers

where it conflicts the pre-existing entry from all.dpkg.org. The conflict leads to the necessary monitoring alerts, and the entry is queued behind the existing one. This situation is also fed back to the operator in response to his attempt to manually add the new entry.

The operator reads the warning returned, and follows a link to queue management for this entry. This link enables him to replace the currently active node with the one just added.

Before replacing the active node, the system makes a sanity check of the new connection, to make sure that there will continue to be a "last-known good" state after the switch. Then it makes the switch. (If the sanity check fails, then the administrator may still force the switch, but it will trigger a monitoring condition for not having a last-known good, which is geared towards classification as an emergency.)

The Shaft now answers to queries for

  • ou=Package Listers
  • cn=SuSe, ou=Package Listers
  • cn=Debian, ou=Package Listers
  • cn=RedHat RHEL, ou=Package Listers
  • cn=RedHat Fedora, ou=Package Listers
  • cn=Debian Wheezy, ou=Package Listers
  • Presented is what was in the queue before, the automatic addition
  • There is a queue holding the manually entered item (which is down)
  • This queue is only visible and manageable over the operators interface
  • The monitoring interface will report on this clash
  • The monitoring interface will not report downtime of the queued item
  • cn=Debian Squeeze, ou=Package Listers
  • Presented is the newly added manual entry
  • There is a queue holding the dynamic item from all.dpkg.org
  • This queue is only visible and manageable over the operators interface
  • The monitoring interface will continue to report on this clash
  • The monitoring interface will not report downtime of the queued item
  • cn=Debian Jessie, ou=Package Listers
  • cn=Ubuntu Kylin, ou=Package Listers

and in addition it listens in on the following subscriptions for additions or removals:

  • ldap://packages.redhat.com/o=RedHat,c=US??sub?(distro=*)
  • ldaps://all.dpkg.org/dc=all,dc=dpkg,dc=org?cn?sub?(&(objectClass=LinuxDistribution)(cn=*))

A conflict-winning Subscription is Removed

The all.dpkg.org site is not very keen on older stuff, so after Jessie is added, it removes Wheezy (in practice, there may be a few generations in between, but that hardly changes the story).

The currently active node for cn=Debian Wheezy, ou=Package Listers is the one that was dynamically added; it is now being removed. Without a queue present, this would have no effect but removal of the subtree as published by Shaft, but the situation is different here, due to the existence of a queue.

When the active node from a queue is removed, manually or dynamically, the queue is investigated and the next node is put up to replace it. This may lead to a temporary unavailability of the subtree, but this is kept as short as possible. When the head of the queue is inactive, there will however be a clearly noticeable downtime, which leads to the monitoring alarms that are more severe than for downtime; this time, the additional problem reported is that there is no "last-known good" state, which is distinguishable so it may be treated as an emergency.

Controlling Downstream Access

Please see the generic description.

Monitoring Shaft

There are many monitoring systems, and they all do pretty much the same. We are looking at Zabbix for this purpose, and intend to use it through SNMP.

The idea is to follow each of the subtrees added, and note the following conditions on them:

  • Whether the node has a last-known good state
  • Whether the active node is offline
  • Whether items are queued, a.k.a. whether a conflict exists

The idea is of course to be able to flag down alerts if they are acceptable; for instance, in recognition of a conflict that merely reflects that there are alternatives waiting in line until the currently active entry is withdrawn. (Changes in the number of these items should re-trigger this alert though.)

Monitoring should support temporarily ignoring certain warnings; for instance, a conflict is rarely vital and may be only a transitionary state while an operating is working on the configuration. On the other hand, having no last-known good state is bound to be a serious problem, until it is deliberately flagged down by an operator. Changes in causes of this problem should be distinguishable, so the operator needs to flag it down again.

We will probably need a MIB for this application, since SNMP barely scratches the surface of applications; specifically, there are no RFC's dealing with application-specific queues or lists.

Based on this, it would be useful to employ Zabbix's "Low-level SNMP Discovery" to automatically detect new table entries, and use a template to insert new items that will be probed. This will lead to one further complaint, namely that a previously existing entry has been removed; this complaint will disappear after some time of continued unavailability.

We want to demonstrate doing this (with Zabbix) as part of the delivery of Shaft.