Reference

The following is a full reference to the submodules inside of the dynamo_consistency module.

checkphedex.py

A module that provides functions to check the comparison results to the list of files and deletions in PhEDEx.

author:Daniel Abercrombie <dabercro@mit.edu>

dynamo_consistency.checkphedex.check_for_datasets(site, orphan_list_file)

Checks PhEDEx exhaustively to see if a dataset should exist at a site, according to PhEDEx, but has files marked as orphans according to our check. This is done via the PhEDEx filereplicas API. The number of filereplicas for each dataset is printed to the terminal. Datasets that contain any filereplicas are returned by this function.

Parameters:

• site (str) – The name of the site to check
• orphan_list_file (list) – List of LFNs that are listed as orphans at the site

Returns:

The list of number of files and datasets for each dataset that is supposed to have at least 1 file at the site.

Return type:

list of tuples

dynamo_consistency.checkphedex.get_phedex_tree(site, callback=None, **kwargs)

Get the file list tree from PhEDEx. Uses the InventoryAge configuration to determine when to refresh cache.

Parameters:site (str) – The site to get information from PhEDEx for. Returns:A tree containing file replicas that are supposed to be at the site Return type:dynamo_consistency.datatypes.DirectoryInfo

dynamo_consistency.checkphedex.set_of_deletions(site)

Get a list of datasets with approved deletion requests at a given site that were created within the number of days matching the IgnoreAge configuration parameter. This request is done via the PhEDEx deleterequests API.

Parameters:site (str) – The site that we want the list of deletion requests for. Returns:Datasets that are in deletion requests Return type:set

config.py

Small module to get information from the config.

Warning

Must be used on a machine with xrdfs installed (for locate command).

author:Daniel Abercrombie <dabercro@mit.edu>

dynamo_consistency.config.CONFIG_FILE = 'consistency_config.json'

The string giving the location of the configuration JSON file. Generally, you want to set this value of the module before calling config_dict() to get your configuration.

dynamo_consistency.config.DIRECTORYLIST = None

If this is set to a list of directories, it overrides the DirectoryList set in the configuration file. This prevents the tool from attempting to list directories that are not there.

dynamo_consistency.config.LOADER = <module 'json' from '/usr/lib/python2.7/json/__init__.pyc'>

A module that uses the load function on a file descriptor to return a dictionary. (Examples are the json and yaml modules.) If your CONFIG_FILE is not a JSON file, you’ll want to change this also before calling config_dict().

dynamo_consistency.config.config_dict(make_dir=True)

Parameters:

make_dir (bool) – Create the cache directory if it’s missing

Returns:

the configuration file in a dictionary

Return type:

str

Raises:

• IOError – when it cannot find the configuration file
• KeyError – when the CacheDirectory key is not set in the configuration file

dynamo_consistency.config.get_redirector(site, banned_doors=None)

Get the redirector and xrootd door servers for a given site. An example valid site name is T2_US_MIT.

Parameters:

• site (str) – The site we want to contact
• banned_doors (list) – Give a list of doors to not return. These are usually ones that just timed out.

Returns:

Public hostname of the local redirector and a list of xrootd door servers

Return type:

str, list

dynamo_consistency.config.locate_file(file_name, redirs=None)

Find servers for a file.

Parameters:

• file_name (str) – Name of the file to locate
• redirs (list) – Global redirectors to start from. If blank, gets them from the configuration file.

Returns:

List of hostnames that hold the file

Return type:

list

datatypes.py

Module defines the datatypes that are used for storage and comparison. There is also a powerful create_dirinfo function that takes a filler function or object and uses the multiprocessing module to recursively list directories in parallel.

author:Daniel Abercrombie <dabercro@mit.edu>

exception dynamo_consistency.datatypes.BadPath

An exception for throwing when the path doesn’t make sense for various methods of a DirectoryInfo

class dynamo_consistency.datatypes.DirectoryInfo(name='', directories=None, files=None)

Stores all of the information of the contents of a directory

Parameters:

• name (str) – The name of the directory
• directories (list) – If this is set, the infos in the list are merged into a master DirectoryInfo.
• files (list) – List of tuples containing information about files in the directory.

add_file_list(file_infos)

Add a list of tuples containing file_name, file_size to the node. This is most useful when you get a list of files from some other source and want to easily convert that list into a DirectoryInfo()

Parameters:file_infos (list) – The list of files (full path, size in bytes[, timestamp])

add_files(files)

Set the files for this DirectoryInfo node

Parameters:files (list) – The tuples of file information. Each element consists of file name, size, and mod time. Returns:self for chaining calls Return type:DirectoryInfo

compare(other, path='', check=None)

Does one way comparison with a different tree

Parameters:

• other (DirectoryInfo) – The directory tree to compare this one to
• path (str) – Is the path to get to this location so far
• check (function) – An optional function that double checks a file name. If the checking function returns True for a file name, the file will not be included in the output.

Returns:

Tuple of list of files and directories that are present and not in the other tree and the size of the files that corresponds to

Return type:

list, list, long

count_nodes(empty=False)

Parameters:empty (bool) – If True, only return the number of empty nodes Returns:The total number of nodes in this Directory Info. This corresponds to approximately the number of listing requests required to build the data. Return type:int

display(path='')

Print out the contents of this DirectoryInfo

Parameters:path (str) – The full path to this DirectoryInfo instance

displays(path='')

Get the string to print out the contents of this DirectoryInfo.

Parameters:path (str) – The full path to this DirectoryInfo instance Returns:The display string Return type:str

empty_nodes_list()

This function should be used to get the nodes to delete in the proper order for non-recursive deletion

Returns:The list of empty directories to delete in the order to delete Return type:list

empty_nodes_set()

This function recursively builds the entire list of empty directories that can be deleted

Returns:The set of empty directories to delete Return type:set

get_directory_size()

Report the total size used by this directory and its subdirectories.

Returns:Size of files in directory, in bytes Return type:int

get_file(file_name)

Get the file dictionary based off the name.

Parameters:file_name (str) – The LFN of the file Returns:Dictionary of file information Return type:dict Raises:BadPath – if the file_name does not start with self.name

get_files(min_age=0, path='')

Get the list of files that are older than some age

Parameters:

• min_age (int) – The minimum age, in seconds, of files to list
• path (str) – The path to this file. Used for recursive calls

Returns:

List of full file paths

Return type:

list

get_node(path, make_new=True)

Get the node that corresponds to the path given. If the node does not exist yet, and make_new is True, the node is created.

Parameters:

• path (str) – Path to the desired node from current node. If the path does not exist yet, empty nodes will be created.
• make_new (str) – Bool to create new node if none exists at path or not

Returns:

A node with the proper path, unless make_new is False and the node doesn’t exist

Return type:

DirectoryInfo or None

get_num_files(unlisted=False, place_new=False)

Report the total number of files stored.

Parameters:

• unlisted (bool) – If true, return number of unlisted directories, Otherwise return only successfully listed files
• place_new (bool) – If true, pretend there’s one more file inside any new directory or if files is None. This prevents listing of empty directories to include directories that should not actually be deleted.

Returns:

The number of files in the directory tree structure

Return type:

int

get_unlisted(path='')

Parameters:path (str) – Path to prepend to the name, used in recursive calls Returns:List of directories that were unlisted Return type:list

listdir(*args, **kwargs)

Get the list of directory names within a DirectoryInfo. Adding an argument will display the contents of the next directory. For example, if dir.listdir() returns:

0: data
1: mc

dir.listdir(1) then lists the contents of mc and dir.listdir(1, 0) lists the contents of the first subdirectory in mc.

Parameters:

• args – Is a list of indices to list the subdirectories
• kwargs – Supports ‘printing’ which is set to a bool. Defaults as True.

Returns:

The DirectoryInfo that is being listed

Return type:

DirectoryInfo

remove_node(path_name)

Remove an empty node from the DirectoryInfo

Parameters:

path_name (str) – The path to the node, including the self.name at the beginning

Returns:

self for chaining

Return type:

DirectoryInfo

Raises:

• NotEmpty – if the directory is not empty or self.files is None
• BadPath – if the path_name does not start with the self.name

save(file_name)

Save this DirectoryInfo in a file.

Parameters:file_name (str) – is the location to save the file

setup_hash()

Set the hashes for this DirectoryInfo

dynamo_consistency.datatypes.IGNORE_AGE = 1.0

The maximum age, in days, of files and directories to ignore in this check. This variable should be reset once in a while by deamons that run while an operator might be adjusting the configuration.

exception dynamo_consistency.datatypes.NotEmpty

An exception for throwing when a non-empty directory is deleted from a DirectoryInfo

dynamo_consistency.datatypes.compare(inventory, listing, output_base=None, orphan_check=None, missing_check=None)

Compare two different trees and output the differences into an ASCII file

Parameters:

• inventory (DirectoryInfo) – The tree of files that should be at a site
• listing (DirectoryInfo) – The tree of files that are listed remotely
• output_base (str) – The names of the ASCII files to place the reports are generated from this variable.
• orphan_check (function) – A function that double checks each expected orphan. The function takes as an input, an LFN. If the function returns true, the LFN will not be listed as an orphan.
• missing_check (function) – A function checks each expected missing file The function takes as an input, an LFN. If the function returns true, the LFN will not be listed as missing.

Returns:

The two lists, missing and orphan files

Return type:

tuple

dynamo_consistency.datatypes.create_dirinfo(location, first_dir, filler, object_params=None, callback=None)

Create the directory information

Parameters:

• location (str) – This is the beginning of the path where we will find first_dir. For example, to find the first directory mc, we also have to say where it is. In most cases, using LFNs, location would be /store/ (where mc is inside). This is a path.
• first_dir (str) – The name of the first directory that is inside the path of location. This should not be a path, but the name of the directory to list recursively.
• filler (function or constructor) –

  This is either a function that lists the directory contents given just a path of os.path.join(location, first_dir), or it is a constructor that does the same thing with a member function called list. If filler is an object constructor, the parameters for the object creation must be passed through the parameter object_params. Both listings must return the following tuple:

  • A bool saying whether the listing was successful or not
  • A list of tuples of sub-directories and their mod times
  • A list of tuples files inside, their size, and their mode times

• object_params (list) – This only needs to be set when filler is an object constructor. Each element in the list is a tuple of arguments to pass to the constructor.
• callback (function) – A function that is called every time master thread has finished checking the child threads. This can happen very many times at large sites. The function is called with the main DirectoryTree as its argument

Returns:

A DirectoryInfo object containing everything the directory listings from os.path.join(location, first_dir) with name first_dir.

Return type:

DirectoryInfo

dynamo_consistency.datatypes.get_info(file_name)

Get the DirectoryInfo from a file.

Parameters:file_name (str) – is the location of the saved information Returns:Saved info Return type:DirectoryInfo

getsitecontents.py

Tool to get the files located at a site.

Warning

Must be used on a machine with XRootD python module installed.

author:

Daniel Abercrombie <dabercro@mit.edu>

Max Goncharov <maxi@mit.edu>

class dynamo_consistency.getsitecontents.GFalLister(site, thread_num=None)

An object to list a site through gfal-ls calls

ls_directory(path)

Gets the contents of a path

Parameters:path (str) – The full path, starting with /store/, of the directory to list. Returns:A bool indicating the success, a list of directories, and a list of files. Return type:bool, list, list

class dynamo_consistency.getsitecontents.Lister(thread_num, site)

The protoype of the listing facility

Parameters:

• thread_num (int) – This optional parameter is only used to Create a separate logger for this object
• site (str) – Used for reading the correct configuration

list(path, retries=0)

Return the directory contents at the given path. The list member is expected of every object passed to datatypes.

Parameters:

• path (str) – The full path, starting with /store/, of the directory to list.
• retries (int) – Number of attempts so far

Returns:

A bool indicating the success, a list of directories, and a list of files. The list of directories consists of tuples of (directory name, mod time). The list of files consistents of tuples of (file name, size, mod time). The modification times are in seconds from epoch and the file size is in bytes.

Return type:

bool, list, list

ls_directory(path)

Prototype function that lists the directories

Parameters:path (str) – The full path, starting with /store/, of the directory to list.

reconnect()

A prototype for reconnecting to the remote servers

class dynamo_consistency.getsitecontents.XRootDLister(site, door, thread_num=None)

A class that holds two XRootD connections. If the primary connection fails to list a directory, then a fallback connection is used. This keeps the load of listing from hitting more than half of a site’s doors at a time.

Parameters:

• site (str) – The site that this connection is to.
• door (str) – The URL of the door that will get the most load
• thread_num (int) – This optional parameter is only used to Create a separate logger for this object

ls_directory(**kwargs)

Gets the contents of the previously defined redirector at a given path

Parameters:path (str) – The full path, starting with /store/, of the directory to list. Returns:A bool indicating the success, a list of directories, and a list of files. Return type:bool, list, list

class dynamo_consistency.getsitecontents.XRootDSubShell(site, door, thread_num=None)

Very similar to the XRootDLister, but uses a subshell through pexpect.

ls_directory(**kwargs)

Prototype function that lists the directories

Parameters:path (str) – The full path, starting with /store/, of the directory to list.

dynamo_consistency.getsitecontents.ct_timestamp(line)

Takes a time string from gfal and extracts the time since epoch

Parameters:line (str) – The line from the gfal-ls call including month, day, and year in some format with lots of hypens Returns:Timestamp’s time since epoch Return type:int

dynamo_consistency.getsitecontents.get_site_tree(site, callback=None, **kwargs)

Get the information for a site, from XRootD or a cache.

Parameters:

• site (str) – The site name
• callback (function) – The callback function to pass to datatypes.create_dirinfo()

Returns:

The site directory listing information

Return type:

dynamo_consistency.datatypes.DirectoryInfo

getinventorycontents.py

This module gets the information from the inventory about a site’s contents

author:Daniel Abercrombie <dabercro@mit.edu>

class dynamo_consistency.getinventorycontents.InvLoader

Creates an InventoryManager object, if needed, and stores it globally for the module. It also holds a list of datasets in the deletion queue.

get_inventory()

Make an inventory, if needed, and return it.

Returns:Any InventoryManager in the vicinity. Return type:common.inventory.InventoryManager

dynamo_consistency.getinventorycontents.get_db_listing(site, callback=None, **kwargs)

Get the list of files from dynamo database directly from MySQL.

Parameters:site (str) – The name of the site to load Returns:The file replicas that are supposed to be at a site Return type:dynamo_consistency.datatypes.DirectoryInfo

dynamo_consistency.getinventorycontents.get_site_inventory(site, callback=None, **kwargs)

Loads the contents of a site, based on the dynamo inventory

Parameters:site (str) – The name of the site to load Returns:The file replicas that are supposed to be at a site Return type:dynamo_consistency.datatypes.DirectoryInfo

dynamo_consistency.getinventorycontents.set_of_ignored()

Get the full list of IGNORED datasets from the inventory