ENCP release notes, from v2_20 to v3_1

Encp changes:
=============

With the --threaded option, encp will multi-thread the actual data transfer.
This has two advantages.  First, there is usually (though not always)
an increase in the transfer rate.  Second, this allows for encp to
independently measure the access to/from local disk and the network.

A bug fix was made to limit the number of "mover impostor" error messages.
This was caused by an obscure order of events with a symptom of 15 minutes
delays between retries.

With the --ecrc switch, encp (on reads) will re-read the new local file from
beginning to end and recalculated the CRC again.  This has a possible flaw
in that the re-read may only check what is still in the buffer cache and
not what was written to disk.

Originally there was only CRC_ERROR.  This error will now only be generated
by the mover (as it was previously).  However, encp can now generate two
new errors: CRC_ENCP_ERROR or CRC_ECRC_ERROR.  The first is raised when
encp notices during the transfer that the data has been corrupted.  The second
is when the corruption is noticed during the --ecrc readback check.  Frequent
occurances of CRC_ENCP_ERROR or CRC_ECRC_ERROR have indicated in the past
a hardware problem with the client node.

The "List index out of range" error (aka "Error processing retry of
") has been fixed.  The retries and resubmits work correctly.

Encp --help gives a short description of the switches.

Support for encp to notify the alarm server of problems that require
remote computer administrator intervention.  The alarm server then will
send the e-mail to a designated list of e-mail addresses.  This is useful
for errors like CRC_ENCP_ERROR or CRC_ECRC_ERROR.

For multi-file write transfers, the delayed dismount for the first (n - 1)
files will be at least 3 minutes.  The last one will continue to use the
configured system default.  This helps prevent the tape from being
dismounted prematurely because the system was busy and the encp was unable
to submit the next request (writes).

A bug fix was made that could have led encp to return a success exit status
when there was an error.  This only applied to multi-file read transfers
where the files were spread over multiple volumes.  Encp submits read
requests in per volume groups.  If an error occurred while trying to read the
first volume's files, but the second (and final) volume's files all read
without error the second volumes success was returned.  This last-one-wins
error has been fixed to return the or-ed status of all the volumes' transfers.

With dcache and srm there are multiple paths to the same file.  For example,
/pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz.  Encp on reads
will first attempt the path that the user specified, if that path does
not exist it will then search the other two paths to attempt to deliver
the users thier files.  For writes users must still know where they are
writing.

On machines that have interface lines in their enstore.conf file, this file
is re-read before every transfer instead of only once at the beginning
of the encp.  This will limit the time it takes for an interface to be ignored
the retry interval of the encp (which by default is 15 minutes).  Previously,
encp would read this file only once at the beginning of execution and
it could take hours for all encps started before the change to the enstore.conf
file to complete.

There are now 5 rates measured during a transfer:
1) network - The rate at which the file is sent over the network.  For a single
   threaded transfer this rate is the same (minus context switches) as the disk
   and transfer rates.  Measured by encp.
2) tape (or drive) - The rate at which the file is read or written from/to
   the tape including selective CRC check implemented on written files by
   mover. Measured by mover.
3) disk - The rate at which the file is read/written from/to disk.  For a
   single threaded transfer this rate is the same (minus context switches)
   as the network and transfer rates.  Measured by encp.
4) overall - The rate measured from the moment the mover contacts encp until
   the last piece of metadata is set.  This does include any mount time
   and CRC readback check time in the rate calculation.  Measured by encp.
5) transfer - the network rate in previous versions of encp.  This rate is 
   invariant to single or multi-threaded encp transfers.  On reads, it
   is measured from when the first byte is detected on encp's data socket 
   until the last byte has been flushed to disk.  On writes, it is measured 
   from when the mover is ready to receive data from encp until the last bytes 
   has been written two tape (and the mover responds saying it is done).  
   Measured by encp.

Encp now has a consistent interpretation of what a pnfs directory is with
the enstore pnfs commands.  For most users this was not a problem, however
for users that choose to automount their pnfs file systems there were problems.
For the record, the Enstore developers consider automounting pnfs filesytems
to be problematic.

Encp now records completed transfer information to the Accounting server.

With the --mmap-io switch memory mapped i/o may be used on supported 
file systems.  The switch --direct-io may also be used on Linux and IRIX 
nodes with supporting file systems (very tricky to make this work).  If memory
mapped i/o or direct i/o is not supported by the file system, encp will
automatically attempt to revert to more portable file system access routines.
These options (along with others) can *possibly* help speed up some transfers.
However, in the hands of unknowledgeable users they can create problems for
themselves and their collaborators.  These options were originally
developed for use by dcache and are not considered very usefull to the 
general user.

Various file and directory errors are handled better.  In the specific
cases found, correct error messages are now given.  For example, it was
possible to get ENODATA when the parent directory passed to encp on a write
was really a regular file; the reported error should have been ENOTDIR.

The CRC of the file is now returned in the data-access-layer output.  The
CRC is also being added to the pnfs layer 4 of files written with encp v3_1.

This is the first version of encp to be cut with python 2.2.3.

Since this version is cut with python 2.2.3, a large file bug that affected
large files on Linux systems is fixed.

Misc.:
======

The legal characters that "enstore pnfs" commands will let you set into a
tag are: letters, digits, underscore (_), dash (-) and slash (/).

Although, v2_20 does work with the new cern wrapper, this is the first
encp release since it was supported in the mover.  The only advantage the
cern wrapper has over the cpio_odc wrapper is that the cern wrapper is not
limited to (8GB - 1) byte files.

The "enstore pnfs" commands should now work in a consistent way with
automounted pnfs filesystems.  For the record, the Enstore developers
consider automounting pnfs filesytems to be problematic.

The enstore pnfs --tagchown command will now take uids and gids instead of
just user names and group names.

A new executable called ensync is included in the Encp product.  It is an
enstore aware rsync like program.  It is intended for smaller experiments
and is not optimized for massive numbers of file transfers.

Another executable called ddencp (or disk to disk encp) is provided.  It 
can be used as a debugging tool to investigate potential disk problems
on client nodes.

                      Detailed cvs commit logs


========== Trace.py ====================================================================================

support using remedy_api 

Moved an error message to use sys.stderr.write() instead of a print statement. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Commented out two lines that servered no purpose. If default_alarm_func() ever got called it would crash the calling program. 

========== alarm.py ====================================================================================

remove .fnal.gov from remedy system name 

fill in system etc 

remove hard coded system name 

support using remedy_api 

change time.daylight to a -1 

========== alarm_server.py ====================================================================================

support using remedy_api 

add support for actions in alarm server 

add trace.trace calls 

========== atomic.py ====================================================================================

Fixed a bug if open fails. The raise statement would return an empty OSError under the old implimentation. Now it raises the exception correctly. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Modification to deal with the phantom EEXIST errors creating the lock file. 

========== backup.py ====================================================================================

add --ignore-failed-read for tar 

Fixed a few little things so the code will work with enstore.py. Also, a few modifications so that it has less problems with pychecker. 

modulized ... add do_work() 

volume_clerk_client.py 

no cachen2a and force gzip 

========== callback.py ====================================================================================

Better timeout handling for writes. 

========== db.py ====================================================================================

guard it against os.getcwd() failure 

cleanup 

do not need to chdir to dbHome in backup 

========== delete_at_exit.py ====================================================================================

Moved an error message to useing sys.stderr.write() instead of the print statement. 

Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv. 

========== driver.py ====================================================================================

These files are partially modifed to address lingering cern wrapper bugs. 

========== e_errors.py ====================================================================================

check transfer rates and disconnect from encp if it is slow 

1) Included support for e-mailable alarms. New function is_emailable(). 2) New error CRC_ECRC_ERROR. This is now split from the CRC_ENCP_ERROR error. 

added POSITIONING_ERROR 

Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. 

Reviewed various non-retriable and raise alarm errors. Seperated CRC_ENCP_ERROR from CRC_ERROR. Also, cleaned up the code for readability. 

Added new error when EXfer read()/write() calls hang in the kernel to long. This error, DEVICE_ERROR, will also raise alarm. 

set volume to noaccess and mover in error state if mover is stuck in some intermediate state 

========== encp.py ====================================================================================

Encp will now record the crc in layer 4 of the file. All internal encp checks have been updated to handle this. Also, all internal changes to pnfs.py have been made to handle this extra piece of data. 

Fixed a bug preventing encp from reading files without brands. Also, made a change to remove a pychecker warning on IRIX. 

Handle the non-optional routing communication for "Get" correctly. 

Fixed a problem with encp.py working with get.py correctly. 

bumping version to v3_1 because of encpCut 

encp.py now uses fullpath() from enstore_functions2.py. Fixes include improved usage of the newly rewritten pnfs.is_pnfs_path() function. This allows for better checking for "unix to unix" or "hsm to hsm" at the beginning of the program, but still leaves the actual existance, permission, etc. checks for later. 

bumping version to v3_0 because of encpCut 

Fixed more branding problems. Clamped down on valid brands. Fixed bug that returned default csc even after determining which csc it should use. 

Fixed problems with branding and determining which enstore system has the requested file. 

Fix bug preventing encp from reading files with thier bfid brands containing non-alphanumeric characters (notably B0MS and D0MS with zeros). 

For some reason the 'inode' field was always set to 0 instead of the actual inode. The actual inode is now used. 

Added 'address' to the volume clerk sub-ticket. 

Handle a "Get" related problem when the filesize is not known and using python 2.0. 

Changes for mylint.py to be happy. 

Another change for encp.py to support "Get". This time, the 'mover' field is not removed from the ticket before a ticket is resent to the LM. 

First commit of encp after starting modification for use in the "get" program for SDSS and KTEV. All pre-get functionality should continue to work as it did before. 

Slight modification to the EncpInterface file consitancy check. When reading if the directory two (or more) levels up did not exist, then a cryptic ENODATA error was given to the user. Now it is "No such file or directory". 

Fixed a permission bug. A number of ways could trigger it, but the original was a regular user trying to write into a directory owned by root. In this case the user got an error of ENODATA. All changes were in e_access(). 

Added accounting changes to record the new rates. 

new fields in encp_xfer 

Modified the dcache write error when the file has already been written to reflect that the file already exists. Added the new (and old) rates to the log file message in such a way that does not break the inquisitor (hopefully). 

Volume assert was broken. Is fixed now. 

Fixed a bug where the target (or possibly tag) does not exist, because the directory it was supposed to be in does not exist. The bug was that it should have reported the diretory as being missing but was incorrectly stating the target. 

Included to modifications to error/warning messages. The log message, "unable to register bfid" has been removed. The "Paranoid CRC mismatch" message has been changed to "CRC readback mismatch". 

These changes have the encp wait for the mover to send one byte on the data socket when writing. This gives encp a chance to know when the mover is done mounting and positioning the tape on writes. This gives encp a chance at getting more acurate rates. 

Fixed a bug where the wrong error message was given on writes and the file does not exist. The code was trying to test a non-pnfs file as a pnfs file and was then giving the wrong error message. 

Modified encp.py and pnfs.py to agree on what a pnfs directory is. Previously, encp would consider any directory named /pnfs to be a pnfs directory. The "enstore pnfs" commands were more stringent and required /pnfs to be the very first directory in the path. This current fix is based on the looser encp version. In addition to just checking the pathname, a check for a pnfs specific "hidden" file is also performed. 

Modifed the final message that encp gives (with verbosity) to include all of the rates calculated (overall, transfer, network, tape/drive and disk). 

Fixed some rate calculation and reporting problems/inconsistancies. There are now 5 rates: network, tape, disk, overall and transfer. 

bumping version to v2_20_b because of encpCut 

Cleanup of the last sys.exc_info()/tb resource leak. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Removed a debugging print statement. 

Added some timing diagnostics to the verbose equal to level 9 output. This mostly includes the time it takes to set the file metadata on writes. 

Modified encp to use host_config.update_cached_config() to force encp to use updated routes on machines where/when the enstore.conf file changes. 

Added a check for layer 1 and layer 4 being empty on writes from dcache. Before this check it appears the dcache occasionaly could write the same file twice creating two (nearly) identical files in enstore. Now the second attempt will fail. 

On reads, encp should now try all three paths to a file in pnfs (/pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz). 

Fixed one problem with the CRC data access value. Most changes though, deal with adding the --override-ro-mount option. 

Added the crc value to the data access layer output. 

On a multi-file read, the tape with the last set of files would 'win' in determining the exit status of the entire encp process. This has been fixed. This did not effect single read requests, multifile reads involving a single tape or writes. If the routing socket udp_server object failed to reaquire the same port when reinitializing; the code used to error out. Now on this 'error' it will reaquire an unused port. This may cause some 'Connection timed out' errors from the mover, but that is life. Lastly, some underpinnings for printing the CRC in the data_access_layer have been included. However, this information is not used yet for anything. 

bumping version to v2_20_a because of encpCut 

When encp stopped using interface.py and started using option.py the switches --ephemeral and --file-family were accidentally dropped. 

Encp will no longer raise alarms if non-critical server are not accessible. 

Fixed an error handling bug when generating an error message. The error was a server replying with OK, but not returning everything it should with a quit message. 

Include conditional import for fcntl functionality. The FCNTL module is depricated in 2.2. 

Added accounting server to list of servers to check when encp first starts. 

Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv. 

For multi-file-transfers when writing, the delayed dismount for the first (n - 1) transfers will be at least 3 minutes. The last transfer will continue to use the default or user specified value. 

Cleaned up some of the direct i/o code. 

Fixed a bug regarding --ecrc and the permissions needed for the original file. It should have only required readable not read/writable. 

1) The message at the beginning of the program of the tags would give incorect values if the output specified was the directory of the target and not the target itself. 2) If the alarm or log servers are not running, a warning should appear, but the code should not abort the transfer. Previous to the fix it would abort. 

Fixed a bug that caused the real error reported from the mover to be ignored. Also, fixed an error message spelling error. 

These changes are in advance of encp v3. They are for future files to use them based on the user being an admin or user. 

Fixed a bug that prevented --threaded from working with the new option.py handling of command line parsing. 

Includes changes in the paranoid ECRC message. Supports e-mailable alarms. Switched over to option.py from interface.py for interpreting the command line arguments. 

feeding information to accounting server 

Include support for --ecrc in encp. This option (for reads) means that after the transfer is complete encp will run an ecrc check back over the file. 

Handle mover timeouts better internaly. Set the error to be RESUBMITTING instead of ETIMEDOUT when no mover info is known in handle_retries(). 

Fix 'unique_id' traceback error on retry/resubmit. 

Bug fix. There is the possibility that some elements may not exist do to errors. This fix tests for the elements existance first. 

Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. 

Uses the new CRC_ENCP_ERROR instead of CRC_ERROR. 

Bug fixes: 1) open_control_socket() will now check the control socket after reading the inital ticket for possible socket closed by mover case. 2) handle_retries() bug fixed that could have resulted in "mover imposter" error messages. A copy of a dictionary item was returned rather than a reference to it. 3) Slow rate errors from EXfer (from the read()/write() calls hanging in the kernel to long) are now handled specially/correctly. 

Fixed the os.getcwd() vulnerability. Added the hostname to the cwd output when verbosity is turned on. 

Handles the socket.socket() errors now. Tickets returned by the mover are checked for consitancy (which is being very parinoid). The gids were being incorrectly listed at the beginning of the transfer. 

When an error occurs on reads the output file is truncated. Also, there are now is_read and is_write functions for use in functions used in reads and writes that return true if the transfer(s) is/are reads or writes accordingly. 

Minor fix for a bug that prevented writing to /dev/null under some conditions. 

Small bug fix. Wrong value types were specified into some print_data_access_layer_format() calls. 

Moved where the $ENSTORE_SPECIAL_LIB environmental variable is processed on reads. Code moved from submit_one_request() to create_read_requests(). 

Added --shortcut switch. This is used in conjunction with the --get-cache switch to bypass full file path extrapolation. This avoids linear searches of all pnfs ids for each component in the full file path. This works similar to the --get-bfid switch. If the --shortcut switch is not given then the original --get-cache functionality remains unchanged. This change has no effect on the --put-chache switch. 

Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp. 

use environment variable to specify LM for read requests 

Fixed how the file-family-wrapper was read in at the begining for verbose output. It was always returning as "Unknown" because there was an extra set of () trying to call the integer returned as a function. 

Modified to call verify_read/write_request_consistancy() after receiving callback ticket from mover. 

Included the current working directory in the verbose and log output. 

Modified mover_handshake to return complete tickets. This should fix the "Error processing retry of " errors. Also, modified the code to be more resilant against 'problem' tickets when processing them in handle_retries(). Specificaly, with the 'resubmits' key. 

Encp checks to see if all critical servers are running at the begging of the program. Also, encp now handles the "KeyError: system_inhibit" traceback error. 

Fixed the format of one of the new verbose output lines. 

Added gid/group and uid/user_name info to log and terminal output. Modified the request verify functions incase the database contains incomplete data. On writes, the tag info is included in the verbose output. 

Fixed a bug so the previous version passes my (current) set of regression tests. The bug was that mmap io needs a file opend for read and write. If the file only has read permissions encp will fall back to posix io. 

These files (encp.py, interface.py, EXfer.c) allow for encp to accept which EXfer optimizations to use from the command line. 

Encp now passes more paramaters to EXfer. These allow for encp to optionally select which optimizations: threaded, direct io or memory mapped io; to use. Requires Exfer.c 1.71 to work. 

Reuse the address and port when reinitalizing the routing socket on retry/resubmit. 

Fixed "Linix" vs. "Linux" bug for direct io. Better error handling when direct io no implemented. 

Undue previous commit. 

Put in hack to handle calculating write rates when very large buffer sizes are used. 

Modified encp to use O_DIRECT on xfs on linix. 

========== enstore.py ====================================================================================

for Estart and Estop call enstore start or stop correspondingly 

make rip9 and rip10 not a special case 

made 1.65 current 

made 1.65 current 

Uses new Estart and Estop. Removed the dbs command. 

Uses new start and stop functionality. $ENSTORE_DIR/src/enstore_start.py $ENSTORE_DIR/src/enstore_stop.py $ENSTORE_DIR/src/enstore_restart.py 

add a enstore command support for event relay 

========== enstore_alarm_search_cgi.py ====================================================================================

type bug fix 

support yesterday as search time 

========== enstore_constants.py ====================================================================================

add support for using remedy_api 

add accounting server and drivestat server to servers page 

add drivestat 

Included new constants for the accounting server and monitor server to be started and stopped. Additional constants were created for support of the ability to e-mail specified alarms. 

add support for no debug log 

add no_max_log_file_size value 

use encp_ip when available on encp history page 

========== enstore_files.py ====================================================================================

obtain encp plot info in a more scalable way 

try to fix problem of plotting on cdf, make code more scalable 

add more debugging info 

add more debugging info 

add other links to saag page 

use encp_ip when available on encp history page 

reverse order of output to status only file 

fix getting lm info when lm is timing out 

attempt to fix problem wher lm is alive but then times out from getting additional information 

========== enstore_html.py ====================================================================================

support using remedy_api 

fix a spelling error 

change enstore to mass storage 

add other links to saag page 

remove network link from saag page 

make navigation bar optional 

fix getting lm info when lm is timing out 

attempt to fix problem wher lm is alive but then times out from getting additional information 

========== enstore_log_file_search_cgi.py ====================================================================================

type bug fix 

support yesterday as search time 

========== enstore_make_plot_page.py ====================================================================================

add support for the quota plot page 

========== enstore_saag.py ====================================================================================

add other links to saag page 

========== enstore_up_down.py ====================================================================================

add a test alarm for auto ticket generation 

remove traceback 

server.movers is a list 

do not turn lm yellow if we are watching no movers 

Modified some literal strings to use strings from enstore_constants. 

========== entv.py ====================================================================================

Fixed a problem when a mover belonged to multiple libraries. Phantom spaces for movers were in the display for these duplicate movers. This was fixed by forcing uniques on the list of names. 

Bug fix for previous entry. When the reupdate occurs an exception occured. 

Fixed the problem where entv forgot about the mover outline colors that movers from a library manager share. 

Added some dels to prevent eronious warning messages being printed out to the terminal with python 2.2. 

Removed references to fcntl module since it wasn't being used. 

Fix some resource leaks from "exc, msg, tb = sys.exc_info()" code. Also, uncomment the unsubscribe line for the event relay messages. 

Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv. 

Fixed the repositioning of the window. Also, another attempt to fix the resource leaks. 

Added some missing exceptions that should have been caught. 

Major modifications toword the removal of resource leaks. There were two leaks fixed (a third was from generic_client) that stemmed from cyclic refereneces. Part of this change was to stop having entv exec itself on certain errors. Other fixes include removing the rate, buffer bar and progress bar from the display when the mover moves into IDLE, ERROR, OFFLINE or Unknown state. Also, most (but not all) blanket excepts now only catch specific exceptions. Most remaing blanket excepts catch KeyboardInterupt first and re-raise. 

Modified entv.py to stop creating resource leaks. This included explicitly creating a master frame for the Canvas. Entv reinitializations under any situations no longer exec itself (which doesn't close open files.) 

Modified to process event relay client errors. 

If the .entvrc file was missing entv would through a traceback. It now correctly skips this and uses the defaults. 

Modifications to pass mylint.py. 

Fixed potential race condition if multiple entvs try to write out .entvrc at once. 

Added code to color code groups of clients and movers. 

Improved error handling when reading in .entvrc file. Also, added functionality to remember if animation is on or off in the .entvrc file. 

Forgot to save one last change from previous commit. 

Fixed a bug if csc.get_movers() fails. 

'Canned' version will now correctly loop back to the beginning of the commands file after it reaches the end. 

Bug fix for canned version. It will not try to resubscribe to a non- existant event relay when in 'canned' mode. 

Change to slow down the handle_message() thread when running canned version of entv. 

Added the ability for entv to read in a command file for running in canned mode. Entv now has a --commands-file similar to the --movers-file. 

Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly. 

Entv now uses interface.py for command line prarsing. The switch --dont-show allows the user to hide all movers belonging to a library manager. 

========== enstore_functions.py ====================================================================================

Added new function get_dict_from_config_file(). It is similar to get_from_config_file(), but the new function returns the entire dictionary. 

Added function get_enstore_tmp_dir() to return the location of temporary enstore files. 

========== event_relay_messages.py ====================================================================================

add a enstore command support for event relay 

========== ftt_driver.py ====================================================================================

fixed problem with ftt returning None and mover tracing back when tape has no label. 

fix a potential memory leak 

Fixed a bug when causing incorrect mover behaviour in case of the failure while labeling tapes. 

added a log message 

check for extended vol header 

remove pdb import 

These files are partially modifed to address lingering cern wrapper bugs. 

added new method 

========== inquisitor.py ====================================================================================

add accounting server and drivestat server to servers page 

add more debugging info 

add more debugging info 

moved tmp files to tmp area 

add migrated to VOLUME_STATES 

typo 

check other_saag_links in config file when marking things down 

fix getting lm info when lm is timing out 

attempt to fix problem wher lm is alive but then times out from getting additional information 

init a variable 

check for periodic tasks when system is backed up 

========== inquisitor_plots.py ====================================================================================

reset acc_db after closing db connection 

do not open db unless need to 

remove enstore_date 

fix query string 

merge in acc_temp file 

make mount plot from accounting db info 

obtain encp plot info in a more scalable way 

========== interface.py ====================================================================================

Include support for --ecrc in encp. This option (for reads) means that after the transfer is complete encp will run an ecrc check back over the file. 

Added --shortcut switch. This is used in conjunction with the --get-cache switch to bypass full file path extrapolation. This avoids linear searches of all pnfs ids for each component in the full file path. This works similar to the --get-bfid switch. If the --shortcut switch is not given then the original --get-cache functionality remains unchanged. This change has no effect on the --put-chache switch. 

Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp. 

These files (encp.py, interface.py, EXfer.c) allow for encp to accept which EXfer optimizations to use from the command line. 

========== inventory.py ====================================================================================

fix the wrong referenc 

fix the wrong size 

bytes used does not depend on capacity any more 

fix sys.exec_info() 

change the semantics of blank 

fix a typo 

remove html tag for plain VOLUMES_DEFINED 

remove html tag for plain VOLUMES_DEFINED 

simply alarm message for too many mounts 

change too alarm client 

generate alarm when tape mount exceeds limit 

remove volume_clerk 

add VOLUMES_TOO_MANY_MOUNTS 

fix a typo again 

change format of volumes_defined again 

change format of volumes_defined 

fix a typo again 

fix a typo 

add mounts to volume_defined 

ignore missing key due to live backup 

add number of files to volume_size 

handle last_access more accurately for recycled volumes 

========== libtpshelve.py ====================================================================================

do not assign trace back of exc_info() to local variable 

========== alarm_client.py ====================================================================================

support using remedy_api 

add init of message 

text as text 

allow sending message in alarm client 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

added client name option 

========== monitored_server.py ====================================================================================

add accounting server and drivestat server to servers page 

========== mover.py ====================================================================================

fixed bug in a function call 

added low level diagnostics for FTT_EBLANK 

fixed incorrectly formatted eod_cookie 

mods for multiple LMs 

shuffle libraries in the idle state to implement a fair selection of LM for multiple LMs 

send a correct state in update_lm 

Fixed/modified the code for multiple LMs serving one mover. 

fixed bug causing mover to not error out if too long in ATIVE state reading tape 

fixed update_stats 

fixed update_stats 

fixed update_stats 

fixed update_stats 

put back Michaels changes that were accidentally erased 

trace to fix update_stats 

trace to fix update_stats 

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

This is a patch to fix the previous patch. This one fixes the disk mover. The mover will now send a byte to the encp before it will start reading bytes from the network on writes into enstore. 

add trigger message for disk mover 

fixed problem with ftt returning None and mover tracing back when tape has no label. 

send message on the data socket when tape write starts for encp to calculate network rate 

replace trace with log to track why mover does not error out when read tape has finished and the client does not receive data for a long time 

adjust types of the stats to match with the DB definition 

convert write protection to int 

modified a log message 

added write protection to stats 

added trace 

added trace 

Catch an exception when ftt fails to read stats due to the hardware problem. In fact some data is present and some not. Use low leve diagnostics to identify the problem 

added log messages to figure out why mover gets stuck in the dismount_wait 

1. If stats file path does not exist. Create it. It could have been deleted by tmpwatch. 2. If there is any error rading the volume label set volume to NOACCESS 

make mover to not send alarm "Too long in state..." while reading tape when data is read but network transfer has not completed because of quite slow transfer rates 

attempt to fix avery rare situation when some variables get modified in one thread while they are being looked at in another thread. Also included MOUNT_WAIT state to generate alarms when too long in state 

update LM 10 time less frequently in the OFFLINE state 

if in OFFLINE send error message to LM just once 

set mover offline after forced dismount if eject failed 

typo fixed 

added some log messages 

addded drivestat 

Fixed a bug when causing incorrect mover behaviour in case of the failure while labeling tapes. 

introduction of new state FINISH_WRITING has prevented the mover to go offline immediately after writing a file. Now it is fixed 

fixed (must have) false alarm when writing tape 

added trace to catch a bug 

added trace to catch a bug 

recover from error state if eventually mover gets out of a very long seek 

fixed a 'typo' bug 

removed test timeout 

fix problem with tape being set to naccess and not being considered by library manager 

check transfer rates and disconnect from encp if it is slow 

1. Modifed format of low level diagn. message 2. Change in response to AML2 MC mount failure: go broke 

make last_error a 2 tuple 

change log to trace 

made sending statistics to the central db server configurable, by default do not send 

some more log messages added 

added log message and diagnostics to track tape writing and reading 

wait forever if vc does not respond to set_remaining_bytes in state FINISH_WRITE and generate alarm 

removed test statement 

1. Corrected transition to error state. 2. Made transfer_failed more clear 3. Changed crc error alarm message and error source. 

handle mount error more gracefuly 

fixed accounting calls 

added accounting calls 

close open sockets if connection failed 

Made mover - encp connection timeout and number of retries configurable. 

added error processing for mount_volume, corrected some errors 

1. Configurable restart on error. If restart_on_error key is set in the mover configuration mover will restart. Otherwise not. 2. Fixed (I think) processing of certain FTT errors in transfer_failed. 3. Improved change of the state in transfer_error 4. Modified offline() to make it thread compliant. 

reordered some lines in transfer_failed 

changed watch_syslog 

some fix to start 

some fix to start 

some fix to start 

some fixe to start 

1. Added a low level diagnostics 2. Offline mover if FTT_EBUSY caused a transfer failure 3. do not restart mover if it is already running. This applies to a case when mover process could not complete because it was in the kernel (D state). 

set mover offline if consecutive_failures exceeded its threshold and error_times exceeded its threshold. This is to fix a problem when the mover was unable to mount a tape due to drive busy and caused a lot of tapes to be set to NOACCESS. 

fixed volume family for volume assert 

restart mover if it too long in the ERROR state 

extend NULL in the null mover served path to anything after NULL 

fixed a bug 

always read a block size number of bytes to fix a firmware bug for some tape drives that return not numbre of bytes requested but a many as left to the file mark 

removed sleep 

use drivestat only if needed 

use drivestat only if needed 

set volume to noaccess and mover in error state if mover is stuck in some intermediate state 

added mount counter to vc and corresponding method 

change rate report for read request 

modified Trace.notify messages for entv/ratekeeper. 

modified calculation of the tape transfer rate 

reverse name change for transfer_time and intrroduce drive_transfer_time 

measure time for the tape and overall transfer separately and return tape time as transfer_time for the backward compatibility. Replace old transfer_time with total_tranfser_time 

========== mover_client.py ====================================================================================

changed TO and retry pars. 

========== net_driver.py ====================================================================================

I know it is silly but mover gets float div. exceptions, and I do not understand where from 

========== null_driver.py ====================================================================================

added diagnistics to close method 

added actve_time 

========== null_wrapper.py ====================================================================================

These files are partially modifed to address lingering cern wrapper bugs. 

========== plotter.py ====================================================================================

make mount plot from accounting db info 

========== priority_selector.py ====================================================================================

increase max pri to 1000001 

do not set adm pri unless it is already set or encp_daq is enabled 

========== setpath.py ====================================================================================

Added \n to the end of the error strings printed. 

Fixed the os.getcwd() vulnerability. 

========== verify_db.py ====================================================================================

fix sys.exec_info() 

change the error message 

========== volume_clerk.py ====================================================================================

do not assign trace back of exc_info() to local variable 

add quit() 

add removing 'mounts' to check_record() 

fix wrong filed 'mounts' 

fix sg-count bug -- do not count deleted volumes 

add sum_mounts at addvol 

guard against None volume at update_counts() 

cope with migrated state 

add --migrated 

keep sg count even when quota is not enforced 

build sgdb at start up if it was not there 

new sgpg related changes 

temporarily fix the missing mounts problem 

fix mounts missing key -- backward compatible 

add 'mounts' to newly created vol 

add --show-quota to volume 

add default event processing routine 

fixed a bug 

fixed a bug 

add event relay subscription to changed configuration 

set initail value correctly 

added mount counter to vc and corresponding method 

add check_record() to trim obsolete fields 

clean up extra fields when recycling 

========== pnfs.py ====================================================================================

Encp will now record the crc in layer 4 of the file. All internal encp checks have been updated to handle this. Also, all internal changes to pnfs.py have been made to handle this extra piece of data. 

pnfs.py now uses fullpath() from enstore_functions2.py. 

Fixed is_pnfs_path() to better handle automounting of pnfs directories AND handle as a local directory those with "pnfs" occuring in a path that is not a pnfs file (i.e. /data1/bja/pnfs/cdfen/filesets/GI/GI10/GI1073/GI1073.1). 

Various fixes for "get" to work. Minor changes were made for the encp functionality used by get. 

Modified is_pnfs_path() to OPTIONALY only check if the name is in the /pnfs area. Previously it would always return false when the target did not exist which was resulting in some incorrect/confusing error messages. 

Modified encp.py and pnfs.py to agree on what a pnfs directory is. Previously, encp would consider any directory named /pnfs to be a pnfs directory. The "enstore pnfs" commands were more stringent and required /pnfs to be the very first directory in the path. This current fix is based on the looser encp version. In addition to just checking the pathname, a check for a pnfs specific "hidden" file is also performed. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Removed the --pnfs-state and --enstore-state options. 

Fix for --tagchown to accept uid and gid. It always should have accepted both ids and names but that was not the case. 

Includes fix for --tags and --tag when a tag contains no/bad data. 

Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. 

Two bug fixes. One, relative paths w.r.t. tags are now respected. Two, empty tags will no longer result in IndexErrors being raised. They are now caught and re-raised as IOErrors. 

Fixed the os.getcwd() vulnerability. 

Removed the pstat_decode messages that are put in the log file. Also, did a little code cleanup. 

Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp. 

Added legal charset test when setting a tag. 

Allow the --size option to take a long integer instead of just an integer. 

========== option.py ====================================================================================

--volume for encp was added. (This is for early get tests.) 

remove remedy_ticket option 

add support for remedy tickets 

Minor modifications for newer (test) version of pychecker. 

Added the encp option --override-ro-mount. 

do not assign trace back of exc_info() to local variable 

add --pvols [--just] 

Fixed some --help related items. Long text lines getting printed instead of cut off and wrapped correctly. Added a help string for --usage. Removed the = from the --help info. If the short options list is long, it gets substituted with "[OPTIONS]..." for usage output. 

allow sending message in alarm client 

add --keep-vol 

Quick mod. to get around termios vs. TERMIOS problem. 

Modified option.py to work under python 2.0 and 2.2 without printing a warning message in python 2.2. 

Included new switches for encp. This is the first attempt to move encp to use option.py. Also, the new --all for enstore start/stop. 

added client name option 

Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. 

add --migrated 

new sgpg related changes 

add GVOL 

add support for the quota plot page 

Added the ability for entv to read in a command file for running in canned mode. Entv now has a --commands-file similar to the --movers-file. 

Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly. 

Include options for entv. The option is --dont-show. Note: the cvs log for entv.py accidentally refered to the old interface.py file instead of the newer option.py file. 

add --show-quota to volume 

Added --just option for start/stop. Added long as a type for options. 

add --trim-obsolete 

========== ratekeeper_client.py ====================================================================================

A fix was made to prevent a traceback from occuring when the user executes "enstore ratekeeper". In line with other clients it now performs and implicit --help. 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

========== host_config.py ====================================================================================

Slight fix to test if the specified enstore.conf file exists AND is a regular file. Note: it is still bad to tell encp to use a file as its enstore.conf file that is not a real enstore.conf file. 

Added a function update_cached_config() that rereads the enstore.conf file and updates the global cached information. 

There was a bug were a local variable could be accessed before having been set due to socket error. Said variable is now given a default value. 

Put in a some time delayed waits in case the DNS server is down for restart. 

More straitforward handling of errors raised if popen failes. Attempt to get the FQDN while determining the default ip address. 

========== monitor_client.py ====================================================================================

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Handle a possible exception better. 

add debugging info after a keyerror 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

Fixed the socket.socket() vulernability. 

========== volume_assert.py ====================================================================================

Modified the request timeout time for the volume clerk query. 

One more option vs generic_client cleanup. 

Have the Interface class inherit from generic_client instead of option. 

Include timeouts incase the configuration server does not respond. 

Volume assert was broken. Is fixed now. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. 

========== generic_client.py ====================================================================================

This is another attempt at fixing the generic_client resource leak problems. There is also a modification to change a print statement to a sys.stderr.write() call. 

use timeouts properly 

Second half of previous entry. 

Second attempt at removing the cyclic reference if the generic client is also a configuration client. 

back out 

Removed a resource leak. If the generic_client was also a configuration client, a cyclic reference was created causing the reference leak. 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

These modifications are in support of the new enstore_start and enstore_stop. Mostly having to do with the event relay dealing with 'alive' and 'quit' messages. 

========== log_trans_fail.py ====================================================================================

modificaions needed to process new sys log message 

include low level tape diagn 

========== monitor_server.py ====================================================================================

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Fixed the socket.socket() vulernability. 

========== event_relay.py ====================================================================================

add a enstore command support for event relay 

========== enstore_show_inventory_cgi.py ====================================================================================

add COMPLETE_FILE_LISTING 

add VOLUMES_TOO_MANY_MOUNTS 

change VOLUMES_DEFINED to VOLUMES_DEFINED.html 

========== file_clerk.py ====================================================================================

do not assign trace back of exc_info() to local variable 

add quit() 

add FILE_DB_USE_INDEX to switch on index file if there is none 

========== library_manager_client.py ====================================================================================

Modified these clients to print the full --help output when no options are given. (i.e. "enstore library") It used to print just the shorter usage help output. This is to have all the server clients conform to a single look and feel. 

fixed interface probelm for delete option 

remove enstore_functions by using e_errors.is_ok() 

changed TO and retry pars. 

Fixed the printing of the active asserts heading. Before any work at movers would cause it to print, now just active asserts to so. 

========== enstore_display.py ====================================================================================

Fixed a problem when using python 2_2_3. A syntax error would show up. Also, a color problem was fixed when a mover belongs to multiple libraries. 

Fixed some discrepencies when moving between states. This involved removing some displayed objects when a message to do so was dropped. 

Fixed some resource leaks w.r.t. "exc, msg, tb = sys.exc_info(). Also, modified some of the "master" variable code for python 2.2. Added the states 'HAVE_BOUND', 'FINISH_WRITE' and 'CLEANING' to the list of states to remove connections, rates, progress bars, etc. Fixed the problem of states sometimes being displayed with full text instead of being truncated to fit the space. 

Modified to support three columns of movers. The connection line now weaves through correctly from the third column. 

Added 'Unknown' to list of states to include for automatic client disconnections. 

Fixed the repositioning of the window. Also, another attempt to fix the resource leaks. 

Added some missing exceptions that should have been caught. 

Major modifications toword the removal of resource leaks. There were two leaks fixed (a third was from generic_client) that stemmed from cyclic refereneces. Part of this change was to stop having entv exec itself on certain errors. Other fixes include removing the rate, buffer bar and progress bar from the display when the mover moves into IDLE, ERROR, OFFLINE or Unknown state. Also, most (but not all) blanket excepts now only catch specific exceptions. Most remaing blanket excepts catch KeyboardInterupt first and re-raise. 

Previous commit broke displaying the connection lines in different colors. 

Modified entv.py to stop creating resource leaks. This included explicitly creating a master frame for the Canvas. Entv reinitializations under any situations no longer exec itself (which doesn't close open files.) Also, modified to handle the new mover state FINISH_WRITE correctly. Previously, FINISH_WRITE and the rate could have been displayed on to of each other. One other thing is that the second collumn's last mover fits in the window. Half was being "drawn" becasuse it is lower than the first column. It was this first collumn where the spacial calculations were done, now they take into account that the second collumn is lower. 

Cleaned up the code that locates the tcl/tk libraries. 

Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. 

If the mover is in "ACTIVE", "SEEK", "SETUP", "loaded", "MOUNT_WAIT", or "DISMOUNT_WAIT" to long the timer color will turn red. 

Fixed the os.getcwd() vulnerability. 

Modification to bring it in sync with entv.py. 

Bug fix when the domain name server is not available. A variable would be accessed but because of the lookup failure was never created. 

Hopefully fixed a problem with "host not found" errors. 

Added fit_string(). It will truncate strings to fit into the provided space. It is used on the state and volume names. 

First attempt to truncate strings to fit. 

Now the connection lines match the outline of the clients. Also, the mouse clicking on a connection will invert the color. 

Added code to color code groups of clients and movers. 

Improved error handling when reading in .entvrc file. Also, added functionality to remember if animation is on or off in the .entvrc file. 

Adjust the width of the mover rectangles based on the number of columns. 

Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly. 

Entv now uses interface.py for command line prarsing. The switch --dont-show allows the user to hide all movers belonging to a library manager. 

Reinitalize the display every so often. 

If the width is very small, then the image for the active state is suppressed. I consider the rate information to be more informative in this case. 

Made changes to the positioning calulation of things in the mover section. Fixed relocation bugs of the differenct percentage bars. Fixed the "How's on top?" problem between the media and network bars. Fixed the un-initialization problem of the volume label font size. 

Added an instantaneous rate display. Also added another bar that indicated percent on mover buffer filled. For the progress bar, two progresses are tracked one for network and one for media. 

========== udp_client.py ====================================================================================

Cleaned up some code like the following: exc, msg, tb = sys.exc_info() to look like the following: exc, msg = sys.exc_info()[:2] This should help to avoid resource leaks. 

Made a fix such that it would send all "done_cleanup" messages before closing the socket. Previously it would close the socket after sending just one. 

Modification to return better error message for misconfigured /etc/hosts file. This seems to happen with default 7.3 fermi linux installs. 

========== event_relay_client.py ====================================================================================

protect from socket.socket and bind errors 

These modifications are in support of the new enstore_start and enstore_stop. Mostly having to do with the event relay dealing with 'alive' and 'quit' messages. 

dont do sys.exit 

add a enstore command support for event relay 

========== configuration_server.py ====================================================================================

remove 2 traceback leak problems 

Fixed a bug when attempting to build a log message. The status was not wrapped in (status,) format (since status is already a tuple). 

Fixed a bug preventing a reload of the configuration file. 

Modified the configuration server to use the event relay client to help manage socket.socket() volnerability. 

Fixed problem in dump(). Trace.trace() call was modified and was given to many arguments. Now the correct number exist. 

removed annoying pprint 

========== udp_server.py ====================================================================================

Modified some Trace.trace levels. Also, added a Trace.trace() to the reply_to_caller_using_interface_ip() function. 

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

If the socket() call failed, do not attempt to use it in the fcntl() call. 

Include conditional import for fcntl functionality. The FCNTL module is depricated in 2.2. 

========== ratekeeper.py ====================================================================================

Modified how tracebacks are handled in the mainloop. This stemmed from fixing a sys.exc_info()/tb resource leak fix. Now Trace.handle_error() and rk.serve_forever_error() are called. 

Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. 

Ratekeeper now uses the event_relay_client for handling event_relay related handling of the socket.socket() vulnerability. 

========== configuration_client.py ====================================================================================

removed a print statement 

add debugging for network malaise 

remove redundancy in check 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

dump() didn't catch a possible TCP_EXCEPTION from callback. 

========== media_changer_test.py ====================================================================================

fixed convert_version 

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

========== super_remove.py ====================================================================================

add --keep-vol 

========== enstore_status.py ====================================================================================

ignore alarm lines 

weed out non- ENCP_XFER messages 

do not process debug messages 

print out encp errors too on history page 

use encp_ip when available on encp history page 

set disk_rate 

support new encp log file line format 

========== cpio_odc_wrapper.py ====================================================================================

These files are partially modifed to address lingering cern wrapper bugs. 

========== cern_wrapper.py ====================================================================================

These files are partially modifed to address lingering cern wrapper bugs. 

remove spurious print 

========== file_clerk_client.py ====================================================================================

handle a strange error 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

honor alive timeout and retry from command line 

========== log_server.py ====================================================================================

do not output misc messages to the extra log files 

fix bug in startup of log server 

fix extra files when not asked for 

fix bugs when starting 

add support for no debug log 

skip file if too big at beginning 

fix orig save logfile name 

fix restarting of log file 

optionally close log file at certain size 

========== disk_driver.py ====================================================================================

fixed to comply with new mover reqs 

========== enstore_functions2.py ====================================================================================

Fixed fullpath(). 

Moved the fullpath function from encp.py (and pnfs.py) to this file. 

add support for actions in alarm server 

========== show_volume_cgi.py ====================================================================================

make pychecker happy 

make pychecker happy 

fix minor bugs 

using --gvol instead 

========== checkBackedUpDatabases.py ====================================================================================

better way to determine if main backup is done 

========== makeplot.py ====================================================================================

A full disk could cause this to traceback. Not it errors out gracefully. 

Fixed a bug that caused it to include to many days worth of files in the plot than it is supposed to. 

make plot backgrounds grey 

remove -modulate for new convert 

========== log_client.py ====================================================================================

attempt to fix start/stop from unnecessary instantiation of lots of clients and delays cusing them to waste about a minute before starting doing anything 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

added client name option 

support yesterday as search time 

========== scanfiles.py ====================================================================================

reduce unnecessary stat 

handle files younger than 1 hour 

try back door to gain permission 

exclude .B_* and .A_* 

========== media_changer.py ====================================================================================

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

diagnostics added 

be aware of 9940B media. Added trace 

ignore Enter 

fixed a bug in cleanCycle of stk class 

fixed a bug in cleanCycle of stk class 

added clean method to stk class 

add migrated state 

========== volume_clerk_client.py ====================================================================================

fix a typo 

add --pvols [--just] 

allow L2 suffix for volume labels 

Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". 

honor alive timeout and retry from command line 

tunning timeout and retry 

tunning timeout and retry 

tuning the timeout and retry 

add --migrated 

new sgpg related changes 

fix previous log message which is supposed to be: fixing missin si_time 

volume_clerk_client.py 

add --gvol 

add timeout for self.send 

fix --clear 

add --show-quota to volume 

added mount counter to vc and corresponding method 

add --trim-obsolete 

========== enstore_overall_status.py ====================================================================================

change enstore to mass storage 

========== media_changer_client.py ====================================================================================

Modified these clients to print the full --help output when no options are given. (i.e. "enstore library") It used to print just the shorter usage help output. This is to have all the server clients conform to a single look and feel. 

========== get_total_bytes_counter.py ====================================================================================

add new libraries 

display TB on system pages 

transfer number as bytes not tb 

========== get_all_bytes_counter.py ====================================================================================

display TB on system pages 

output in correct format 

strip off newline 

transfer number as bytes not tb 

change name of bytes file 

make a file with total bytes as an integer for sc2002. 

========== enstore_show_inv_summary_cgi.py ====================================================================================

add COMPLETE_FILE_LISTING 

add VOLUMES_TOO_MANY_MOUNTS 

change VOLUMES_DEFINED to VOLUMES_DEFINED.html 

========== library_manager.py ====================================================================================

do nothing in bad_volume. This is taken care of in mover_error 

changed version comparison 

log errror message 

fixed convert_version 

fixed convert_version 

code changes to address tape ingest issues 

code changes to address tape ingest issues 

do not set volume to noaccess while processing requests. This must be done in the error processing 

attempt to identify unpack non-sequence type error in bad_volume call 

changes in mover_error to process FTT_EBLANK error 

some mods for ktev request porcessing 

allow setting max_suspect_movers and max_suspect_volumes in the configuration file 

modified mover_busy to clean up at movers list for multiple LM per mover 

mods for multiple LMs 

Fixed /modified the code for multiple LMs serving one mover. 

These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. 

fixed bug in delete_mover 

5. Found why the requests may not go to movers even if the file family width allows. At least one case is understood.If the mover fails to mount tape it sends an error message to the library manager and then goes to the idle state. In the error message the mover,volume pair is mover,current_volume. In the idle message this pair is mover,previous_volume. When removing the entry from volumes_at_movers queue LM looks up for the pair mover,volume and it may not get removed because of different combinations described in the mover message. Now this is fixed. 

modified get_work_at_movers to search for pairs: label, mover. Before it was searching only for the label. This was wrong. Consider the following. Mover1 fails on VOL1. It send error rq and lm removes VOL1 from work at movers. Suppose VOL1 was dismounted and is available. Mover2 gets the VOL1 and now it is again in work at movers. But the error message from mover1 comes and removes this work from work at movers. 

trace added 

some minor changes 

some more logging 

replace 33 with 31 in trace 

replaced 13 with 33 in trace. 13 is used in config client and inteferes 

fixed bug 

added some diagnostics 

create path for a lock file if does not exist 

remove double entry 

fix problem with tape being set to naccess and not being considered by library manager 

added retrn value checks for vol_info 

add migrated state 

fix bug in mover_bound_volume causing LM to not update volumes_at_movers for HIPRI requests, which in turn causes a wrong calculation of busy volumes 

do not allow any, hence hipr rq. go if it is not already in the list of active tapes 

fix ambiguous logic in busy_volumes 

changed trace level for assigning write rqs 

LM with new queue manager 

added trace 

added some traces changed get in manage queue 

removed some traces to speedup the server 

added trace next_work_any_volume 

added traces to process_read_request 

some more mods for next_work_this_volume 

some more mods for next_work_this_volume 

modified next_work_this_volume to chech for restricted host access 

========== generic_server.py ====================================================================================

Modified the generic server to avoid a sys.exc_info()/tb resource leak. Another still remains. 

========== enstore_plots.py ====================================================================================

do not write a plot if there is no data 

make mount plot from accounting db info 

make plot backgrounds grey 

remove -modulate for new convert 

translate a / to a - in drive_id 

========== manage_queue.py ====================================================================================

removed the remnants of the old adm_queue, and fixed the self.admin_queue.find call 

increase max pri to 1000001 

removed some heavy diagnistics 

LM with new queue manager 

bakked off changes 

cnaged trace 

modified get and get_next 

added some traces 

added some traces 

added some traces 

added some traces 

added some traces 

added some traces 

added some traces changed get in manage queue 

bakked off the change 

removed some traces to speedup the server 

fixed a typo 

When request qeue contains admin and not admin requests, there was incorrect processing of ordinary requests skipping the 1st one in the queue. Now it is fixed in Request_Queue.get 

========== dispatching_worker.py ====================================================================================

fix the annoying fd is not integer problem 

add debugging information 

If the UDPServer client failed to initialize (specifically to obtain its udp socket) then exit the server. 

Added a comment about a line of code.