API Reference¶

DiffSync front-end classes and logic.

class diffsync.DiffSync(name=None)¶

Bases: object

Class for storing a group of DiffSyncModel instances and diffing/synchronizing to another DiffSync instance.

__init__(name=None)¶

Generic initialization function.

Subclasses should be careful to call super().__init__() if they override this method.

add(obj: diffsync.DiffSyncModel)¶

Add a DiffSyncModel object to the store.

Parameters: obj (DiffSyncModel) – Object to store
Raises: ObjectAlreadyExists – if an object with the same uid is already present

dict(exclude_defaults: bool = True, **kwargs) → Mapping¶: Represent the DiffSync contents as a dict, as if it were a Pydantic model.

diff_from(source: diffsync.DiffSync, diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>, flags: diffsync.enum.DiffSyncFlags = <DiffSyncFlags.NONE: 0>, callback: Optional[Callable[[str, int, int], None]] = None) → diffsync.diff.Diff ¶

Generate a Diff describing the difference from the other DiffSync to this one.

Parameters

source (DiffSync) – Object to diff against.
diff_class (class) – Diff or subclass thereof to use for diff calculation and storage.
flags (DiffSyncFlags) – Flags influencing the behavior of this diff operation.
callback (function) – Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff proceeds.

diff_to(target: diffsync.DiffSync, diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>, flags: diffsync.enum.DiffSyncFlags = <DiffSyncFlags.NONE: 0>, callback: Optional[Callable[[str, int, int], None]] = None) → diffsync.diff.Diff ¶

Generate a Diff describing the difference from this DiffSync to another one.

Parameters

target (DiffSync) – Object to diff against.
diff_class (class) – Diff or subclass thereof to use for diff calculation and storage.
flags (DiffSyncFlags) – Flags influencing the behavior of this diff operation.
callback (function) – Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff proceeds.

get(obj: Union[str, diffsync.DiffSyncModel, Type[diffsync.DiffSyncModel]], identifier: Union[str, Mapping]) → diffsync.DiffSyncModel ¶

Get one object from the data store based on its unique id.

Parameters

obj – DiffSyncModel class or instance, or modelname string, that defines the type of the object to retrieve
identifier – Unique ID of the object to retrieve, or dict of unique identifier keys/values

Raises

ValueError – if obj is a str and identifier is a dict (can’t convert dict into a uid str without a model class)
ObjectNotFound – if the requested object is not present

get_all(obj: Union[str, diffsync.DiffSyncModel, Type[diffsync.DiffSyncModel]]) → List[diffsync.DiffSyncModel]¶

Get all objects of a given type.

Parameters: obj – DiffSyncModel class or instance, or modelname string, that defines the type of the objects to retrieve
Returns: List of Object
Return type: List[DiffSyncModel]

get_by_uids(uids: List[str], obj: Union[str, diffsync.DiffSyncModel, Type[diffsync.DiffSyncModel]]) → List[diffsync.DiffSyncModel]¶

Get multiple objects from the store by their unique IDs/Keys and type.

Parameters

uids – List of unique id / key identifying object in the database.
obj – DiffSyncModel class or instance, or modelname string, that defines the type of the objects to retrieve

Raises

ObjectNotFound – if any of the requested UIDs are not found in the store

load()¶: Load all desired data from whatever backend data source into this instance.

remove(obj: diffsync.DiffSyncModel, remove_children: bool = False)¶

Remove a DiffSyncModel object from the store.

Parameters

obj (DiffSyncModel) – object to remove
remove_children (bool) – If True, also recursively remove any children of this object

Raises

ObjectNotFound – if the object is not present

str(indent: int = 0) → str ¶: Build a detailed string representation of this DiffSync.

sync_complete(source: diffsync.DiffSync, diff: diffsync.diff.Diff, flags: diffsync.enum.DiffSyncFlags = <DiffSyncFlags.NONE: 0>, logger: Optional[structlog._generic.BoundLogger] = None)¶

Callback triggered after a sync_from operation has completed and updated the model data of this instance.

Note that this callback is only triggered if the sync actually resulted in data changes. If there are no detected changes, this callback will not be called.

The default implementation does nothing, but a subclass could use this, for example, to perform bulk updates to a backend (such as a file) that doesn’t readily support incremental updates to individual records.

Parameters

source – The DiffSync whose data was used to update this instance.
diff – The Diff calculated prior to the sync operation.
flags – Any flags that influenced the sync.
logger – Logging context for the sync.

sync_from(source: diffsync.DiffSync, diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>, flags: diffsync.enum.DiffSyncFlags = <DiffSyncFlags.NONE: 0>, callback: Optional[Callable[[str, int, int], None]] = None)¶

Synchronize data from the given source DiffSync object into the current DiffSync object.

Parameters

source (DiffSync) – object to sync data from into this one
diff_class (class) – Diff or subclass thereof to use to calculate the diffs to use for synchronization
flags (DiffSyncFlags) – Flags influencing the behavior of this sync.
callback (function) – Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff and subsequent sync proceed.

sync_to(target: diffsync.DiffSync, diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>, flags: diffsync.enum.DiffSyncFlags = <DiffSyncFlags.NONE: 0>, callback: Optional[Callable[[str, int, int], None]] = None)¶

Synchronize data from the current DiffSync object into the given target DiffSync object.

Parameters

target (DiffSync) – object to sync data into from this one.
diff_class (class) – Diff or subclass thereof to use to calculate the diffs to use for synchronization
flags (DiffSyncFlags) – Flags influencing the behavior of this sync.
callback (function) – Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff and subsequent sync proceed.

top_level: ClassVar[List[str]] = []¶: List of top-level modelnames to begin from when diffing or synchronizing.

type: ClassVar[Optional[str]] = None¶: Type of the object, will default to the name of the class if not provided.

class diffsync.DiffSyncModel(*, model_flags: diffsync.enum.DiffSyncModelFlags = <DiffSyncModelFlags.NONE: 0>, diffsync: DiffSync = None)¶

Bases: pydantic.main.BaseModel

Base class for all DiffSync object models.

Note that read-only APIs of this class are implemented as get_*() functions rather than as properties; this is intentional as specific model classes may want to use these names (type, keys, attrs, etc.) as model attributes and we want to avoid any ambiguity or collisions.

This class has several underscore-prefixed class variables that subclasses should set as desired; see below.

NOTE: The groupings _identifiers, _attributes, and _children are mutually exclusive; any given field name can: be included in at most one of these three tuples.

class Config¶

Bases: object

Pydantic class configuration.

arbitrary_types_allowed = True¶

add_child(child: diffsync.DiffSyncModel)¶

Add a child reference to an object.

The child object isn’t stored, only its unique id. The name of the target attribute is defined in _children per object type

Raises

ObjectStoreWrongType – if the type is not part of _children
ObjectAlreadyExists – if the unique id is already stored

classmethod create(diffsync: diffsync.DiffSync, ids: Mapping, attrs: Mapping) → Optional[diffsync.DiffSyncModel]¶

Instantiate this class, along with any platform-specific data creation.

Subclasses must call super().create(); they may wish to then override the default status information by calling set_status() to provide more context (such as details of any interactions with underlying systems).

Parameters

diffsync – The master data store for other DiffSyncModel instances that we might need to reference
ids – Dictionary of unique-identifiers needed to create the new object
attrs – Dictionary of additional attributes to set on the new object

Returns

instance of this class, if all data was successfully created. None: if data creation failed in such a way that child objects of this model should not be created.

Return type

DiffSyncModel

Raises

ObjectNotCreated – if an error occurred.

classmethod create_unique_id(**identifiers) → str ¶

Construct a unique identifier for this model class.

Parameters: **identifiers – Dict of identifiers and their values, as in get_identifiers().

delete() → Optional[diffsync.DiffSyncModel]¶

Delete any platform-specific data corresponding to this instance.

Subclasses must call super().delete(); they may wish to then override the default status information by calling set_status() to provide more context (such as details of any interactions with underlying systems).

Returns: this instance, if all data was successfully deleted. None: if data deletion failed in such a way that child objects of this model should not be deleted.
Return type: DiffSyncModel
Raises: ObjectNotDeleted – if an error occurred.

dict(**kwargs) → dict ¶: Convert this DiffSyncModel to a dict, excluding the diffsync field by default as it is not serializable.

diffsync: Optional[diffsync.DiffSync]¶

the DiffSync instance that owns this model instance.

Type: Optional

get_attrs() → Mapping¶

Get all the non-primary-key attributes or parameters for this object.

Similar to Pydantic’s BaseModel.dict() method, with the following key differences: 1. Does not include the fields in _identifiers 2. Only includes fields explicitly listed in _attributes 3. Does not include any additional fields not listed in _attributes

Returns: Dictionary of attributes for this object
Return type: dict

classmethod get_children_mapping() → Mapping[str, str]¶: Get the mapping of types to fieldnames for child models of this model.

get_identifiers() → Mapping¶

Get a dict of all identifiers (primary keys) and their values for this object.

Returns: dictionary containing all primary keys for this device, as defined in _identifiers
Return type: dict

get_shortname() → str ¶

Get the (not guaranteed-unique) shortname of an object, if any.

By default the shortname is built based on all the keys defined in _shortname. If _shortname is not specified, then this function is equivalent to get_unique_id().

Returns: Shortname of this object
Return type: str

get_status() → Tuple[diffsync.enum.DiffSyncStatus, str]¶: Get the status of the last create/update/delete operation on this object, and any associated message.

classmethod get_type() → str ¶

Return the type AKA modelname of the object or the class

Returns: modelname of the class, used in to store all objects
Return type: str

get_unique_id() → str ¶

Get the unique ID of an object.

By default the unique ID is built based on all the primary keys defined in _identifiers.

Returns: Unique ID for this object
Return type: str

json(**kwargs) → str ¶: Convert this DiffSyncModel to a JSON string, excluding the diffsync field by default as it is not serializable.

model_flags: diffsync.enum.DiffSyncModelFlags¶

any non-default behavioral flags for this DiffSyncModel.

Can be set as a class attribute or an instance attribute as needed.

Type: Optional

remove_child(child: diffsync.DiffSyncModel)¶

Remove a child reference from an object.

The name of the storage attribute is defined in _children per object type.

Raises

ObjectStoreWrongType – if the child model type is not part of _children
ObjectNotFound – if the child wasn’t previously present.

set_status(status: diffsync.enum.DiffSyncStatus, message: str = '')¶: Update the status (and optionally status message) of this model in response to a create/update/delete call.

str(include_children: bool = True, indent: int = 0) → str ¶: Build a detailed string representation of this DiffSyncModel and optionally its children.

update(attrs: Mapping) → Optional[diffsync.DiffSyncModel]¶

Update the attributes of this instance, along with any platform-specific data updates.

Subclasses must call super().update(); they may wish to then override the default status information by calling set_status() to provide more context (such as details of any interactions with underlying systems).

Parameters: attrs – Dictionary of attributes to update on the object
Returns: this instance, if all data was successfully updated. None: if data updates failed in such a way that child objects of this model should not be modified.
Return type: DiffSyncModel
Raises: ObjectNotUpdated – if an error occurred.

diffsync.diff¶

Diff and DiffElement classes for DiffSync.

class diffsync.diff.Diff¶

Bases: object

Diff Object, designed to store multiple DiffElement object and organize them in a group.

__init__()¶: Initialize a new, empty Diff object.

add(element: diffsync.diff.DiffElement)¶

Add a new DiffElement to the changeset of this Diff.

Raises: ObjectAlreadyExists – if an element of the same type and same name is already stored.

children¶

DefaultDict for storing DiffElement objects.

self.children[group][unique_id] == DiffElement(…)

complete()¶

Method to call when this Diff has been fully populated with data and is “complete”.

The default implementation does nothing, but a subclass could use this, for example, to save the completed Diff to a file or database record.

dict() → Mapping[str, Mapping[str, Mapping]]¶: Build a dictionary representation of this Diff.

get_children() → Iterator[diffsync.diff.DiffElement]¶

Iterate over all child elements in all groups in self.children.

For each group of children, check if an order method is defined, Otherwise use the default method.

groups()¶: Get the list of all group keys in self.children.

has_diffs() → bool¶

Indicate if at least one of the child elements contains some diff.

Returns: True if at least one child element contains some diff
Return type: bool

classmethod order_children_default(children: Mapping) → Iterator[diffsync.diff.DiffElement]¶

Default method to an Iterator for children.

Since children is already an OrderedDefaultDict, this method is not doing anything special.

str(indent: int = 0)¶: Build a detailed string representation of this Diff and its child DiffElements.

summary() → Mapping[str, int]¶: Build a dict summary of this Diff and its child DiffElements.

class diffsync.diff.DiffElement(obj_type: str, name: str, keys: Mapping, source_name: str = 'source', dest_name: str = 'dest', diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>)¶

Bases: object

DiffElement object, designed to represent a single item/object that may or may not have any diffs.

__init__(obj_type: str, name: str, keys: Mapping, source_name: str = 'source', dest_name: str = 'dest', diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>)¶

Instantiate a DiffElement.

Parameters

obj_type – Name of the object type being described, as in DiffSyncModel.get_type().
name – Human-readable name of the object being described, as in DiffSyncModel.get_shortname(). This name must be unique within the context of the Diff that is the direct parent of this DiffElement.
keys – Primary keys and values uniquely describing this object, as in DiffSyncModel.get_identifiers().
source_name – Name of the source DiffSync object
dest_name – Name of the destination DiffSync object
diff_class – Diff or subclass thereof to use to calculate the diffs to use for synchronization

property action: Optional[str]¶

Action, if any, that should be taken to remediate the diffs described by this element.

Returns: “create”, “update”, “delete”, or None
Return type: str

add_attrs(source: Optional[Mapping] = None, dest: Optional[Mapping] = None)¶: Set additional attributes of a source and/or destination item that may result in diffs.

add_child(element: diffsync.diff.DiffElement)¶

Attach a child object of type DiffElement.

Childs are saved in a Diff object and are organized by type and name.

Parameters: element – DiffElement

dict() → Mapping[str, Mapping[str, Any]]¶: Build a dictionary representation of this DiffElement and its children.

get_attrs_diffs() → Mapping[str, Mapping[str, Any]]¶

Get the dict of actual attribute diffs between source_attrs and dest_attrs.

Returns: of the form {“-“: {key1: <value>, key2: …}, “+”: {key1: <value>, key2: …}}, where the “-“ or “+” dicts may be absent.
Return type: dict

get_attrs_keys() → Iterable[str]¶

Get the list of shared attrs between source and dest, or the attrs of source or dest if only one is present.

If source_attrs is not set, return the keys of dest_attrs
If dest_attrs is not set, return the keys of source_attrs
If both are defined, return the intersection of both keys

get_children() → Iterator[diffsync.diff.DiffElement]¶: Iterate over all child DiffElements of this one.

has_diffs(include_children: bool = True) → bool¶

Check whether this element (or optionally any of its children) has some diffs.

Parameters: include_children – If True, recursively check children for diffs as well.

str(indent: int = 0)¶: Build a detailed string representation of this DiffElement and its children.

summary() → Mapping[str, int]¶: Build a summary of this DiffElement and its children.

diffsync.enum¶

DiffSync enums and flags.

class diffsync.enum.DiffSyncFlags(value)¶

Bases: enum.Flag

Flags that can be passed to a sync_* or diff_* call to affect its behavior.

CONTINUE_ON_FAILURE = 1¶: Continue synchronizing even if failures are encountered when syncing individual models.

LOG_UNCHANGED_RECORDS = 8¶

If this flag is set, a log message will be generated during synchronization for each model, even unchanged ones.

By default, when this flag is unset, only models that have actual changes to synchronize will be logged. This flag is off by default to reduce the default verbosity of DiffSync, but can be enabled when debugging.

NONE = 0¶

SKIP_UNMATCHED_BOTH = 6¶

SKIP_UNMATCHED_DST = 4¶

Ignore objects that only exist in the target/”to” DiffSync when determining diffs and syncing.

If this flag is set, no objects will be deleted from the target/”to” DiffSync.

SKIP_UNMATCHED_SRC = 2¶

Ignore objects that only exist in the source/”from” DiffSync when determining diffs and syncing.

If this flag is set, no new objects will be created in the target/”to” DiffSync.

class diffsync.enum.DiffSyncModelFlags(value)¶

Bases: enum.Flag

Flags that can be set on a DiffSyncModel class or instance to affect its usage.

IGNORE = 1¶

Do not render diffs containing this model; do not make any changes to this model when synchronizing.

Can be used to indicate a model instance that exists but should not be changed by DiffSync.

NONE = 0¶

SKIP_CHILDREN_ON_DELETE = 2¶

When deleting this model, do not recursively delete its children.

Can be used for the case where deletion of a model results in the automatic deletion of all its children.

class diffsync.enum.DiffSyncStatus(value)¶

Bases: enum.Enum

Flag values to set as a DiffSyncModel’s _status when performing a sync; values are logged by DiffSyncSyncer.

ERROR = 'error'¶

FAILURE = 'failure'¶

SUCCESS = 'success'¶

UNKNOWN = 'unknown'¶

diffsync.exceptions¶

Exception classes used in DiffSync.

exception diffsync.exceptions.ObjectAlreadyExists¶

Bases: diffsync.exceptions.ObjectStoreException

Exception raised when trying to store a DiffSyncModel or DiffElement that is already being stored.

exception diffsync.exceptions.ObjectCrudException¶

Bases: Exception

Base class for various failures during CRUD operations.

exception diffsync.exceptions.ObjectNotCreated¶

Bases: diffsync.exceptions.ObjectCrudException

Exception raised if an object Create operation failed.

exception diffsync.exceptions.ObjectNotDeleted¶

Bases: diffsync.exceptions.ObjectCrudException

Exception raised if an object Delete operation failed.

exception diffsync.exceptions.ObjectNotFound¶

Bases: diffsync.exceptions.ObjectStoreException

Exception raised when trying to access a DiffSyncModel that isn’t in storage.

exception diffsync.exceptions.ObjectNotUpdated¶

Bases: diffsync.exceptions.ObjectCrudException

Exception raised if an object Update operation failed.

exception diffsync.exceptions.ObjectStoreException¶

Bases: Exception

Base class for various failures during object storage in local caches.

exception diffsync.exceptions.ObjectStoreWrongType¶

Bases: diffsync.exceptions.ObjectStoreException

Exception raised when trying to store a DiffSyncModel of the wrong type.

diffsync.helpers¶

DiffSync helper classes for calculating and performing diff and sync operations.

class diffsync.helpers.DiffSyncDiffer(src_diffsync: DiffSync, dst_diffsync: DiffSync, flags: diffsync.enum.DiffSyncFlags, diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>, callback: Optional[Callable[[str, int, int], None]] = None)¶

Bases: object

Helper class implementing diff calculation logic for DiffSync.

Independent from Diff and DiffElement as those classes are purely data objects, while this stores some state.

__init__(src_diffsync: DiffSync, dst_diffsync: DiffSync, flags: diffsync.enum.DiffSyncFlags, diff_class: Type[diffsync.diff.Diff] = <class 'diffsync.diff.Diff'>, callback: Optional[Callable[[str, int, int], None]] = None)¶: Create a DiffSyncDiffer for calculating diffs between the provided DiffSync instances.

calculate_diffs() → diffsync.diff.Diff ¶: Calculate diffs between the src and dst DiffSync objects and return the resulting Diff.

diff_child_objects(diff_element: diffsync.diff.DiffElement, src_obj: Optional[DiffSyncModel], dst_obj: Optional[DiffSyncModel])¶

For all children of the given DiffSyncModel pair, diff recursively, adding diffs to the given diff_element.

Helper method to calculate_diffs, usually doesn’t need to be called directly.

These helper methods work in a recursive cycle: diff_object_list -> diff_object_pair -> diff_child_objects -> diff_object_list -> etc.

diff_object_list(src: List[DiffSyncModel], dst: List[DiffSyncModel]) → List[diffsync.diff.DiffElement]¶

Calculate diffs between two lists of like objects.

Helper method to calculate_diffs, usually doesn’t need to be called directly.

These helper methods work in a recursive cycle: diff_object_list -> diff_object_pair -> diff_child_objects -> diff_object_list -> etc.

diff_object_pair(src_obj: Optional[DiffSyncModel], dst_obj: Optional[DiffSyncModel]) → Optional[diffsync.diff.DiffElement]¶

Diff the two provided DiffSyncModel objects and return a DiffElement or None.

Helper method to calculate_diffs, usually doesn’t need to be called directly.

These helper methods work in a recursive cycle: diff_object_list -> diff_object_pair -> diff_child_objects -> diff_object_list -> etc.

incr_models_processed(delta: int = 1)¶: Increment self.models_processed, then call self.callback if present.

static validate_objects_for_diff(object_pairs: Iterable[Tuple[Optional[DiffSyncModel], Optional[DiffSyncModel]]])¶

Check whether all DiffSyncModels in the given dictionary are valid for comparison to one another.

Helper method for diff_object_list.

Raises

TypeError – If any pair of objects in the dict have differing get_type() values.
ValueError – If any pair of objects in the dict have differing get_shortname() or get_identifiers() values.

class diffsync.helpers.DiffSyncSyncer(diff: diffsync.diff.Diff, src_diffsync: DiffSync, dst_diffsync: DiffSync, flags: diffsync.enum.DiffSyncFlags, callback: Optional[Callable[[str, int, int], None]] = None)¶

Bases: object

Helper class implementing data synchronization logic for DiffSync.

Independent from DiffSync and DiffSyncModel as those classes are purely data objects, while this stores some state.

__init__(diff: diffsync.diff.Diff, src_diffsync: DiffSync, dst_diffsync: DiffSync, flags: diffsync.enum.DiffSyncFlags, callback: Optional[Callable[[str, int, int], None]] = None)¶: Create a DiffSyncSyncer instance, ready to call perform_sync() against.

incr_elements_processed(delta: int = 1)¶: Increment self.elements_processed, then call self.callback if present.

log_sync_status(action: Optional[str], status: diffsync.enum.DiffSyncStatus, message: str)¶

Log the current sync status at the appropriate verbosity with appropriate context.

Helper method to sync_diff_element/sync_model.

perform_sync() → bool¶

Perform data synchronization based on the provided diff.

Returns: True if any changes were actually performed, else False.
Return type: bool

sync_diff_element(element: diffsync.diff.DiffElement, parent_model: DiffSyncModel = None) → bool¶

Recursively synchronize the given DiffElement and its children, if any, into the dst_diffsync.

Helper method to perform_sync.

Returns: True if this element or any of its children resulted in actual changes, else False.
Return type: bool

sync_model(model: Optional[DiffSyncModel], ids: Mapping, attrs: Mapping) → Tuple[bool, Optional[DiffSyncModel]]¶

Create/update/delete the current DiffSyncModel with current ids/attrs, and update self.status and self.message.

Helper method to sync_diff_element.

Returns: (changed, model) where model may be None if an error occurred
Return type: tuple

diffsync.logging¶

Helpful APIs for setting up DiffSync logging.

diffsync.logging.enable_console_logging(verbosity=0)¶

Enable formatted logging to console with the specified verbosity.

See https://www.structlog.org/en/stable/development.html as a reference

Parameters: verbosity (int) – 0 for WARNING logs, 1 for INFO logs, 2 for DEBUG logs

diffsync.utils¶

Utility functions for DiffSync library.

class diffsync.utils.OrderedDefaultDict(dict_type)¶

Bases: collections.OrderedDict

A combination of collections.OrderedDict and collections.DefaultDict behavior.

__init__(dict_type)¶: Create a new OrderedDefaultDict.

diffsync.utils.intersection(lst1, lst2) → List¶: Calculate the intersection of two lists, with ordering based on the first list.

diffsync.utils.symmetric_difference(lst1, lst2) → List¶: Calculate the symmetric difference of two lists.