ENCP release notes, from v2_20 to v3_1 Encp changes: ============= With the --threaded option, encp will multi-thread the actual data transfer. This has two advantages. First, there is usually (though not always) an increase in the transfer rate. Second, this allows for encp to independently measure the access to/from local disk and the network. A bug fix was made to limit the number of "mover impostor" error messages. This was caused by an obscure order of events with a symptom of 15 minutes delays between retries. With the --ecrc switch, encp (on reads) will re-read the new local file from beginning to end and recalculated the CRC again. This has a possible flaw in that the re-read may only check what is still in the buffer cache and not what was written to disk. Originally there was only CRC_ERROR. This error will now only be generated by the mover (as it was previously). However, encp can now generate two new errors: CRC_ENCP_ERROR or CRC_ECRC_ERROR. The first is raised when encp notices during the transfer that the data has been corrupted. The second is when the corruption is noticed during the --ecrc readback check. Frequent occurances of CRC_ENCP_ERROR or CRC_ECRC_ERROR have indicated in the past a hardware problem with the client node. The "List index out of range" error (aka "Error processing retry of") has been fixed. The retries and resubmits work correctly. Encp --help gives a short description of the switches. Support for encp to notify the alarm server of problems that require remote computer administrator intervention. The alarm server then will send the e-mail to a designated list of e-mail addresses. This is useful for errors like CRC_ENCP_ERROR or CRC_ECRC_ERROR. For multi-file write transfers, the delayed dismount for the first (n - 1) files will be at least 3 minutes. The last one will continue to use the configured system default. This helps prevent the tape from being dismounted prematurely because the system was busy and the encp was unable to submit the next request (writes). A bug fix was made that could have led encp to return a success exit status when there was an error. This only applied to multi-file read transfers where the files were spread over multiple volumes. Encp submits read requests in per volume groups. If an error occurred while trying to read the first volume's files, but the second (and final) volume's files all read without error the second volumes success was returned. This last-one-wins error has been fixed to return the or-ed status of all the volumes' transfers. With dcache and srm there are multiple paths to the same file. For example, /pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz. Encp on reads will first attempt the path that the user specified, if that path does not exist it will then search the other two paths to attempt to deliver the users thier files. For writes users must still know where they are writing. On machines that have interface lines in their enstore.conf file, this file is re-read before every transfer instead of only once at the beginning of the encp. This will limit the time it takes for an interface to be ignored the retry interval of the encp (which by default is 15 minutes). Previously, encp would read this file only once at the beginning of execution and it could take hours for all encps started before the change to the enstore.conf file to complete. There are now 5 rates measured during a transfer: 1) network - The rate at which the file is sent over the network. For a single threaded transfer this rate is the same (minus context switches) as the disk and transfer rates. Measured by encp. 2) tape (or drive) - The rate at which the file is read or written from/to the tape including selective CRC check implemented on written files by mover. Measured by mover. 3) disk - The rate at which the file is read/written from/to disk. For a single threaded transfer this rate is the same (minus context switches) as the network and transfer rates. Measured by encp. 4) overall - The rate measured from the moment the mover contacts encp until the last piece of metadata is set. This does include any mount time and CRC readback check time in the rate calculation. Measured by encp. 5) transfer - the network rate in previous versions of encp. This rate is invariant to single or multi-threaded encp transfers. On reads, it is measured from when the first byte is detected on encp's data socket until the last byte has been flushed to disk. On writes, it is measured from when the mover is ready to receive data from encp until the last bytes has been written two tape (and the mover responds saying it is done). Measured by encp. Encp now has a consistent interpretation of what a pnfs directory is with the enstore pnfs commands. For most users this was not a problem, however for users that choose to automount their pnfs file systems there were problems. For the record, the Enstore developers consider automounting pnfs filesytems to be problematic. Encp now records completed transfer information to the Accounting server. With the --mmap-io switch memory mapped i/o may be used on supported file systems. The switch --direct-io may also be used on Linux and IRIX nodes with supporting file systems (very tricky to make this work). If memory mapped i/o or direct i/o is not supported by the file system, encp will automatically attempt to revert to more portable file system access routines. These options (along with others) can *possibly* help speed up some transfers. However, in the hands of unknowledgeable users they can create problems for themselves and their collaborators. These options were originally developed for use by dcache and are not considered very usefull to the general user. Various file and directory errors are handled better. In the specific cases found, correct error messages are now given. For example, it was possible to get ENODATA when the parent directory passed to encp on a write was really a regular file; the reported error should have been ENOTDIR. The CRC of the file is now returned in the data-access-layer output. The CRC is also being added to the pnfs layer 4 of files written with encp v3_1. This is the first version of encp to be cut with python 2.2.3. Since this version is cut with python 2.2.3, a large file bug that affected large files on Linux systems is fixed. Misc.: ====== The legal characters that "enstore pnfs" commands will let you set into a tag are: letters, digits, underscore (_), dash (-) and slash (/). Although, v2_20 does work with the new cern wrapper, this is the first encp release since it was supported in the mover. The only advantage the cern wrapper has over the cpio_odc wrapper is that the cern wrapper is not limited to (8GB - 1) byte files. The "enstore pnfs" commands should now work in a consistent way with automounted pnfs filesystems. For the record, the Enstore developers consider automounting pnfs filesytems to be problematic. The enstore pnfs --tagchown command will now take uids and gids instead of just user names and group names. A new executable called ensync is included in the Encp product. It is an enstore aware rsync like program. It is intended for smaller experiments and is not optimized for massive numbers of file transfers. Another executable called ddencp (or disk to disk encp) is provided. It can be used as a debugging tool to investigate potential disk problems on client nodes. Detailed cvs commit logs ========== Trace.py ==================================================================================== support using remedy_api Moved an error message to use sys.stderr.write() instead of a print statement. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Commented out two lines that servered no purpose. If default_alarm_func() ever got called it would crash the calling program. ========== alarm.py ==================================================================================== remove .fnal.gov from remedy system name fill in system etc remove hard coded system name support using remedy_api change time.daylight to a -1 ========== alarm_server.py ==================================================================================== support using remedy_api add support for actions in alarm server add trace.trace calls ========== atomic.py ==================================================================================== Fixed a bug if open fails. The raise statement would return an empty OSError under the old implimentation. Now it raises the exception correctly. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Modification to deal with the phantom EEXIST errors creating the lock file. ========== backup.py ==================================================================================== add --ignore-failed-read for tar Fixed a few little things so the code will work with enstore.py. Also, a few modifications so that it has less problems with pychecker. modulized ... add do_work() volume_clerk_client.py no cachen2a and force gzip ========== callback.py ==================================================================================== Better timeout handling for writes. ========== db.py ==================================================================================== guard it against os.getcwd() failure cleanup do not need to chdir to dbHome in backup ========== delete_at_exit.py ==================================================================================== Moved an error message to useing sys.stderr.write() instead of the print statement. Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv. ========== driver.py ==================================================================================== These files are partially modifed to address lingering cern wrapper bugs. ========== e_errors.py ==================================================================================== check transfer rates and disconnect from encp if it is slow 1) Included support for e-mailable alarms. New function is_emailable(). 2) New error CRC_ECRC_ERROR. This is now split from the CRC_ENCP_ERROR error. added POSITIONING_ERROR Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. Reviewed various non-retriable and raise alarm errors. Seperated CRC_ENCP_ERROR from CRC_ERROR. Also, cleaned up the code for readability. Added new error when EXfer read()/write() calls hang in the kernel to long. This error, DEVICE_ERROR, will also raise alarm. set volume to noaccess and mover in error state if mover is stuck in some intermediate state ========== encp.py ==================================================================================== Encp will now record the crc in layer 4 of the file. All internal encp checks have been updated to handle this. Also, all internal changes to pnfs.py have been made to handle this extra piece of data. Fixed a bug preventing encp from reading files without brands. Also, made a change to remove a pychecker warning on IRIX. Handle the non-optional routing communication for "Get" correctly. Fixed a problem with encp.py working with get.py correctly. bumping version to v3_1 because of encpCut encp.py now uses fullpath() from enstore_functions2.py. Fixes include improved usage of the newly rewritten pnfs.is_pnfs_path() function. This allows for better checking for "unix to unix" or "hsm to hsm" at the beginning of the program, but still leaves the actual existance, permission, etc. checks for later. bumping version to v3_0 because of encpCut Fixed more branding problems. Clamped down on valid brands. Fixed bug that returned default csc even after determining which csc it should use. Fixed problems with branding and determining which enstore system has the requested file. Fix bug preventing encp from reading files with thier bfid brands containing non-alphanumeric characters (notably B0MS and D0MS with zeros). For some reason the 'inode' field was always set to 0 instead of the actual inode. The actual inode is now used. Added 'address' to the volume clerk sub-ticket. Handle a "Get" related problem when the filesize is not known and using python 2.0. Changes for mylint.py to be happy. Another change for encp.py to support "Get". This time, the 'mover' field is not removed from the ticket before a ticket is resent to the LM. First commit of encp after starting modification for use in the "get" program for SDSS and KTEV. All pre-get functionality should continue to work as it did before. Slight modification to the EncpInterface file consitancy check. When reading if the directory two (or more) levels up did not exist, then a cryptic ENODATA error was given to the user. Now it is "No such file or directory". Fixed a permission bug. A number of ways could trigger it, but the original was a regular user trying to write into a directory owned by root. In this case the user got an error of ENODATA. All changes were in e_access(). Added accounting changes to record the new rates. new fields in encp_xfer Modified the dcache write error when the file has already been written to reflect that the file already exists. Added the new (and old) rates to the log file message in such a way that does not break the inquisitor (hopefully). Volume assert was broken. Is fixed now. Fixed a bug where the target (or possibly tag) does not exist, because the directory it was supposed to be in does not exist. The bug was that it should have reported the diretory as being missing but was incorrectly stating the target. Included to modifications to error/warning messages. The log message, "unable to register bfid" has been removed. The "Paranoid CRC mismatch" message has been changed to "CRC readback mismatch". These changes have the encp wait for the mover to send one byte on the data socket when writing. This gives encp a chance to know when the mover is done mounting and positioning the tape on writes. This gives encp a chance at getting more acurate rates. Fixed a bug where the wrong error message was given on writes and the file does not exist. The code was trying to test a non-pnfs file as a pnfs file and was then giving the wrong error message. Modified encp.py and pnfs.py to agree on what a pnfs directory is. Previously, encp would consider any directory named /pnfs to be a pnfs directory. The "enstore pnfs" commands were more stringent and required /pnfs to be the very first directory in the path. This current fix is based on the looser encp version. In addition to just checking the pathname, a check for a pnfs specific "hidden" file is also performed. Modifed the final message that encp gives (with verbosity) to include all of the rates calculated (overall, transfer, network, tape/drive and disk). Fixed some rate calculation and reporting problems/inconsistancies. There are now 5 rates: network, tape, disk, overall and transfer. bumping version to v2_20_b because of encpCut Cleanup of the last sys.exc_info()/tb resource leak. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Removed a debugging print statement. Added some timing diagnostics to the verbose equal to level 9 output. This mostly includes the time it takes to set the file metadata on writes. Modified encp to use host_config.update_cached_config() to force encp to use updated routes on machines where/when the enstore.conf file changes. Added a check for layer 1 and layer 4 being empty on writes from dcache. Before this check it appears the dcache occasionaly could write the same file twice creating two (nearly) identical files in enstore. Now the second attempt will fail. On reads, encp should now try all three paths to a file in pnfs (/pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz). Fixed one problem with the CRC data access value. Most changes though, deal with adding the --override-ro-mount option. Added the crc value to the data access layer output. On a multi-file read, the tape with the last set of files would 'win' in determining the exit status of the entire encp process. This has been fixed. This did not effect single read requests, multifile reads involving a single tape or writes. If the routing socket udp_server object failed to reaquire the same port when reinitializing; the code used to error out. Now on this 'error' it will reaquire an unused port. This may cause some 'Connection timed out' errors from the mover, but that is life. Lastly, some underpinnings for printing the CRC in the data_access_layer have been included. However, this information is not used yet for anything. bumping version to v2_20_a because of encpCut When encp stopped using interface.py and started using option.py the switches --ephemeral and --file-family were accidentally dropped. Encp will no longer raise alarms if non-critical server are not accessible. Fixed an error handling bug when generating an error message. The error was a server replying with OK, but not returning everything it should with a quit message. Include conditional import for fcntl functionality. The FCNTL module is depricated in 2.2. Added accounting server to list of servers to check when encp first starts. Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv. For multi-file-transfers when writing, the delayed dismount for the first (n - 1) transfers will be at least 3 minutes. The last transfer will continue to use the default or user specified value. Cleaned up some of the direct i/o code. Fixed a bug regarding --ecrc and the permissions needed for the original file. It should have only required readable not read/writable. 1) The message at the beginning of the program of the tags would give incorect values if the output specified was the directory of the target and not the target itself. 2) If the alarm or log servers are not running, a warning should appear, but the code should not abort the transfer. Previous to the fix it would abort. Fixed a bug that caused the real error reported from the mover to be ignored. Also, fixed an error message spelling error. These changes are in advance of encp v3. They are for future files to use them based on the user being an admin or user. Fixed a bug that prevented --threaded from working with the new option.py handling of command line parsing. Includes changes in the paranoid ECRC message. Supports e-mailable alarms. Switched over to option.py from interface.py for interpreting the command line arguments. feeding information to accounting server Include support for --ecrc in encp. This option (for reads) means that after the transfer is complete encp will run an ecrc check back over the file. Handle mover timeouts better internaly. Set the error to be RESUBMITTING instead of ETIMEDOUT when no mover info is known in handle_retries(). Fix 'unique_id' traceback error on retry/resubmit. Bug fix. There is the possibility that some elements may not exist do to errors. This fix tests for the elements existance first. Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. Uses the new CRC_ENCP_ERROR instead of CRC_ERROR. Bug fixes: 1) open_control_socket() will now check the control socket after reading the inital ticket for possible socket closed by mover case. 2) handle_retries() bug fixed that could have resulted in "mover imposter" error messages. A copy of a dictionary item was returned rather than a reference to it. 3) Slow rate errors from EXfer (from the read()/write() calls hanging in the kernel to long) are now handled specially/correctly. Fixed the os.getcwd() vulnerability. Added the hostname to the cwd output when verbosity is turned on. Handles the socket.socket() errors now. Tickets returned by the mover are checked for consitancy (which is being very parinoid). The gids were being incorrectly listed at the beginning of the transfer. When an error occurs on reads the output file is truncated. Also, there are now is_read and is_write functions for use in functions used in reads and writes that return true if the transfer(s) is/are reads or writes accordingly. Minor fix for a bug that prevented writing to /dev/null under some conditions. Small bug fix. Wrong value types were specified into some print_data_access_layer_format() calls. Moved where the $ENSTORE_SPECIAL_LIB environmental variable is processed on reads. Code moved from submit_one_request() to create_read_requests(). Added --shortcut switch. This is used in conjunction with the --get-cache switch to bypass full file path extrapolation. This avoids linear searches of all pnfs ids for each component in the full file path. This works similar to the --get-bfid switch. If the --shortcut switch is not given then the original --get-cache functionality remains unchanged. This change has no effect on the --put-chache switch. Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp. use environment variable to specify LM for read requests Fixed how the file-family-wrapper was read in at the begining for verbose output. It was always returning as "Unknown" because there was an extra set of () trying to call the integer returned as a function. Modified to call verify_read/write_request_consistancy() after receiving callback ticket from mover. Included the current working directory in the verbose and log output. Modified mover_handshake to return complete tickets. This should fix the "Error processing retry of " errors. Also, modified the code to be more resilant against 'problem' tickets when processing them in handle_retries(). Specificaly, with the 'resubmits' key. Encp checks to see if all critical servers are running at the begging of the program. Also, encp now handles the "KeyError: system_inhibit" traceback error. Fixed the format of one of the new verbose output lines. Added gid/group and uid/user_name info to log and terminal output. Modified the request verify functions incase the database contains incomplete data. On writes, the tag info is included in the verbose output. Fixed a bug so the previous version passes my (current) set of regression tests. The bug was that mmap io needs a file opend for read and write. If the file only has read permissions encp will fall back to posix io. These files (encp.py, interface.py, EXfer.c) allow for encp to accept which EXfer optimizations to use from the command line. Encp now passes more paramaters to EXfer. These allow for encp to optionally select which optimizations: threaded, direct io or memory mapped io; to use. Requires Exfer.c 1.71 to work. Reuse the address and port when reinitalizing the routing socket on retry/resubmit. Fixed "Linix" vs. "Linux" bug for direct io. Better error handling when direct io no implemented. Undue previous commit. Put in hack to handle calculating write rates when very large buffer sizes are used. Modified encp to use O_DIRECT on xfs on linix. ========== enstore.py ==================================================================================== for Estart and Estop call enstore start or stop correspondingly make rip9 and rip10 not a special case made 1.65 current made 1.65 current Uses new Estart and Estop. Removed the dbs command. Uses new start and stop functionality. $ENSTORE_DIR/src/enstore_start.py $ENSTORE_DIR/src/enstore_stop.py $ENSTORE_DIR/src/enstore_restart.py add a enstore command support for event relay ========== enstore_alarm_search_cgi.py ==================================================================================== type bug fix support yesterday as search time ========== enstore_constants.py ==================================================================================== add support for using remedy_api add accounting server and drivestat server to servers page add drivestat Included new constants for the accounting server and monitor server to be started and stopped. Additional constants were created for support of the ability to e-mail specified alarms. add support for no debug log add no_max_log_file_size value use encp_ip when available on encp history page ========== enstore_files.py ==================================================================================== obtain encp plot info in a more scalable way try to fix problem of plotting on cdf, make code more scalable add more debugging info add more debugging info add other links to saag page use encp_ip when available on encp history page reverse order of output to status only file fix getting lm info when lm is timing out attempt to fix problem wher lm is alive but then times out from getting additional information ========== enstore_html.py ==================================================================================== support using remedy_api fix a spelling error change enstore to mass storage add other links to saag page remove network link from saag page make navigation bar optional fix getting lm info when lm is timing out attempt to fix problem wher lm is alive but then times out from getting additional information ========== enstore_log_file_search_cgi.py ==================================================================================== type bug fix support yesterday as search time ========== enstore_make_plot_page.py ==================================================================================== add support for the quota plot page ========== enstore_saag.py ==================================================================================== add other links to saag page ========== enstore_up_down.py ==================================================================================== add a test alarm for auto ticket generation remove traceback server.movers is a list do not turn lm yellow if we are watching no movers Modified some literal strings to use strings from enstore_constants. ========== entv.py ==================================================================================== Fixed a problem when a mover belonged to multiple libraries. Phantom spaces for movers were in the display for these duplicate movers. This was fixed by forcing uniques on the list of names. Bug fix for previous entry. When the reupdate occurs an exception occured. Fixed the problem where entv forgot about the mover outline colors that movers from a library manager share. Added some dels to prevent eronious warning messages being printed out to the terminal with python 2.2. Removed references to fcntl module since it wasn't being used. Fix some resource leaks from "exc, msg, tb = sys.exc_info()" code. Also, uncomment the unsubscribe line for the event relay messages. Moved the setup_signal_handling and signal_handler functions into the delete_at_exit python module. This generalizes the code for encp, ensyc and entv. Fixed the repositioning of the window. Also, another attempt to fix the resource leaks. Added some missing exceptions that should have been caught. Major modifications toword the removal of resource leaks. There were two leaks fixed (a third was from generic_client) that stemmed from cyclic refereneces. Part of this change was to stop having entv exec itself on certain errors. Other fixes include removing the rate, buffer bar and progress bar from the display when the mover moves into IDLE, ERROR, OFFLINE or Unknown state. Also, most (but not all) blanket excepts now only catch specific exceptions. Most remaing blanket excepts catch KeyboardInterupt first and re-raise. Modified entv.py to stop creating resource leaks. This included explicitly creating a master frame for the Canvas. Entv reinitializations under any situations no longer exec itself (which doesn't close open files.) Modified to process event relay client errors. If the .entvrc file was missing entv would through a traceback. It now correctly skips this and uses the defaults. Modifications to pass mylint.py. Fixed potential race condition if multiple entvs try to write out .entvrc at once. Added code to color code groups of clients and movers. Improved error handling when reading in .entvrc file. Also, added functionality to remember if animation is on or off in the .entvrc file. Forgot to save one last change from previous commit. Fixed a bug if csc.get_movers() fails. 'Canned' version will now correctly loop back to the beginning of the commands file after it reaches the end. Bug fix for canned version. It will not try to resubscribe to a non- existant event relay when in 'canned' mode. Change to slow down the handle_message() thread when running canned version of entv. Added the ability for entv to read in a command file for running in canned mode. Entv now has a --commands-file similar to the --movers-file. Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly. Entv now uses interface.py for command line prarsing. The switch --dont-show allows the user to hide all movers belonging to a library manager. ========== enstore_functions.py ==================================================================================== Added new function get_dict_from_config_file(). It is similar to get_from_config_file(), but the new function returns the entire dictionary. Added function get_enstore_tmp_dir() to return the location of temporary enstore files. ========== event_relay_messages.py ==================================================================================== add a enstore command support for event relay ========== ftt_driver.py ==================================================================================== fixed problem with ftt returning None and mover tracing back when tape has no label. fix a potential memory leak Fixed a bug when causing incorrect mover behaviour in case of the failure while labeling tapes. added a log message check for extended vol header remove pdb import These files are partially modifed to address lingering cern wrapper bugs. added new method ========== inquisitor.py ==================================================================================== add accounting server and drivestat server to servers page add more debugging info add more debugging info moved tmp files to tmp area add migrated to VOLUME_STATES typo check other_saag_links in config file when marking things down fix getting lm info when lm is timing out attempt to fix problem wher lm is alive but then times out from getting additional information init a variable check for periodic tasks when system is backed up ========== inquisitor_plots.py ==================================================================================== reset acc_db after closing db connection do not open db unless need to remove enstore_date fix query string merge in acc_temp file make mount plot from accounting db info obtain encp plot info in a more scalable way ========== interface.py ==================================================================================== Include support for --ecrc in encp. This option (for reads) means that after the transfer is complete encp will run an ecrc check back over the file. Added --shortcut switch. This is used in conjunction with the --get-cache switch to bypass full file path extrapolation. This avoids linear searches of all pnfs ids for each component in the full file path. This works similar to the --get-bfid switch. If the --shortcut switch is not given then the original --get-cache functionality remains unchanged. This change has no effect on the --put-chache switch. Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp. These files (encp.py, interface.py, EXfer.c) allow for encp to accept which EXfer optimizations to use from the command line. ========== inventory.py ==================================================================================== fix the wrong referenc fix the wrong size bytes used does not depend on capacity any more fix sys.exec_info() change the semantics of blank fix a typo remove html tag for plain VOLUMES_DEFINED remove html tag for plain VOLUMES_DEFINED simply alarm message for too many mounts change too alarm client generate alarm when tape mount exceeds limit remove volume_clerk add VOLUMES_TOO_MANY_MOUNTS fix a typo again change format of volumes_defined again change format of volumes_defined fix a typo again fix a typo add mounts to volume_defined ignore missing key due to live backup add number of files to volume_size handle last_access more accurately for recycled volumes ========== libtpshelve.py ==================================================================================== do not assign trace back of exc_info() to local variable ========== alarm_client.py ==================================================================================== support using remedy_api add init of message text as text allow sending message in alarm client Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". added client name option ========== monitored_server.py ==================================================================================== add accounting server and drivestat server to servers page ========== mover.py ==================================================================================== fixed bug in a function call added low level diagnostics for FTT_EBLANK fixed incorrectly formatted eod_cookie mods for multiple LMs shuffle libraries in the idle state to implement a fair selection of LM for multiple LMs send a correct state in update_lm Fixed/modified the code for multiple LMs serving one mover. fixed bug causing mover to not error out if too long in ATIVE state reading tape fixed update_stats fixed update_stats fixed update_stats fixed update_stats put back Michaels changes that were accidentally erased trace to fix update_stats trace to fix update_stats These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. This is a patch to fix the previous patch. This one fixes the disk mover. The mover will now send a byte to the encp before it will start reading bytes from the network on writes into enstore. add trigger message for disk mover fixed problem with ftt returning None and mover tracing back when tape has no label. send message on the data socket when tape write starts for encp to calculate network rate replace trace with log to track why mover does not error out when read tape has finished and the client does not receive data for a long time adjust types of the stats to match with the DB definition convert write protection to int modified a log message added write protection to stats added trace added trace Catch an exception when ftt fails to read stats due to the hardware problem. In fact some data is present and some not. Use low leve diagnostics to identify the problem added log messages to figure out why mover gets stuck in the dismount_wait 1. If stats file path does not exist. Create it. It could have been deleted by tmpwatch. 2. If there is any error rading the volume label set volume to NOACCESS make mover to not send alarm "Too long in state..." while reading tape when data is read but network transfer has not completed because of quite slow transfer rates attempt to fix avery rare situation when some variables get modified in one thread while they are being looked at in another thread. Also included MOUNT_WAIT state to generate alarms when too long in state update LM 10 time less frequently in the OFFLINE state if in OFFLINE send error message to LM just once set mover offline after forced dismount if eject failed typo fixed added some log messages addded drivestat Fixed a bug when causing incorrect mover behaviour in case of the failure while labeling tapes. introduction of new state FINISH_WRITING has prevented the mover to go offline immediately after writing a file. Now it is fixed fixed (must have) false alarm when writing tape added trace to catch a bug added trace to catch a bug recover from error state if eventually mover gets out of a very long seek fixed a 'typo' bug removed test timeout fix problem with tape being set to naccess and not being considered by library manager check transfer rates and disconnect from encp if it is slow 1. Modifed format of low level diagn. message 2. Change in response to AML2 MC mount failure: go broke make last_error a 2 tuple change log to trace made sending statistics to the central db server configurable, by default do not send some more log messages added added log message and diagnostics to track tape writing and reading wait forever if vc does not respond to set_remaining_bytes in state FINISH_WRITE and generate alarm removed test statement 1. Corrected transition to error state. 2. Made transfer_failed more clear 3. Changed crc error alarm message and error source. handle mount error more gracefuly fixed accounting calls added accounting calls close open sockets if connection failed Made mover - encp connection timeout and number of retries configurable. added error processing for mount_volume, corrected some errors 1. Configurable restart on error. If restart_on_error key is set in the mover configuration mover will restart. Otherwise not. 2. Fixed (I think) processing of certain FTT errors in transfer_failed. 3. Improved change of the state in transfer_error 4. Modified offline() to make it thread compliant. reordered some lines in transfer_failed changed watch_syslog some fix to start some fix to start some fix to start some fixe to start 1. Added a low level diagnostics 2. Offline mover if FTT_EBUSY caused a transfer failure 3. do not restart mover if it is already running. This applies to a case when mover process could not complete because it was in the kernel (D state). set mover offline if consecutive_failures exceeded its threshold and error_times exceeded its threshold. This is to fix a problem when the mover was unable to mount a tape due to drive busy and caused a lot of tapes to be set to NOACCESS. fixed volume family for volume assert restart mover if it too long in the ERROR state extend NULL in the null mover served path to anything after NULL fixed a bug always read a block size number of bytes to fix a firmware bug for some tape drives that return not numbre of bytes requested but a many as left to the file mark removed sleep use drivestat only if needed use drivestat only if needed set volume to noaccess and mover in error state if mover is stuck in some intermediate state added mount counter to vc and corresponding method change rate report for read request modified Trace.notify messages for entv/ratekeeper. modified calculation of the tape transfer rate reverse name change for transfer_time and intrroduce drive_transfer_time measure time for the tape and overall transfer separately and return tape time as transfer_time for the backward compatibility. Replace old transfer_time with total_tranfser_time ========== mover_client.py ==================================================================================== changed TO and retry pars. ========== net_driver.py ==================================================================================== I know it is silly but mover gets float div. exceptions, and I do not understand where from ========== null_driver.py ==================================================================================== added diagnistics to close method added actve_time ========== null_wrapper.py ==================================================================================== These files are partially modifed to address lingering cern wrapper bugs. ========== plotter.py ==================================================================================== make mount plot from accounting db info ========== priority_selector.py ==================================================================================== increase max pri to 1000001 do not set adm pri unless it is already set or encp_daq is enabled ========== setpath.py ==================================================================================== Added \n to the end of the error strings printed. Fixed the os.getcwd() vulnerability. ========== verify_db.py ==================================================================================== fix sys.exec_info() change the error message ========== volume_clerk.py ==================================================================================== do not assign trace back of exc_info() to local variable add quit() add removing 'mounts' to check_record() fix wrong filed 'mounts' fix sg-count bug -- do not count deleted volumes add sum_mounts at addvol guard against None volume at update_counts() cope with migrated state add --migrated keep sg count even when quota is not enforced build sgdb at start up if it was not there new sgpg related changes temporarily fix the missing mounts problem fix mounts missing key -- backward compatible add 'mounts' to newly created vol add --show-quota to volume add default event processing routine fixed a bug fixed a bug add event relay subscription to changed configuration set initail value correctly added mount counter to vc and corresponding method add check_record() to trim obsolete fields clean up extra fields when recycling ========== pnfs.py ==================================================================================== Encp will now record the crc in layer 4 of the file. All internal encp checks have been updated to handle this. Also, all internal changes to pnfs.py have been made to handle this extra piece of data. pnfs.py now uses fullpath() from enstore_functions2.py. Fixed is_pnfs_path() to better handle automounting of pnfs directories AND handle as a local directory those with "pnfs" occuring in a path that is not a pnfs file (i.e. /data1/bja/pnfs/cdfen/filesets/GI/GI10/GI1073/GI1073.1). Various fixes for "get" to work. Minor changes were made for the encp functionality used by get. Modified is_pnfs_path() to OPTIONALY only check if the name is in the /pnfs area. Previously it would always return false when the target did not exist which was resulting in some incorrect/confusing error messages. Modified encp.py and pnfs.py to agree on what a pnfs directory is. Previously, encp would consider any directory named /pnfs to be a pnfs directory. The "enstore pnfs" commands were more stringent and required /pnfs to be the very first directory in the path. This current fix is based on the looser encp version. In addition to just checking the pathname, a check for a pnfs specific "hidden" file is also performed. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Removed the --pnfs-state and --enstore-state options. Fix for --tagchown to accept uid and gid. It always should have accepted both ids and names but that was not the case. Includes fix for --tags and --tag when a tag contains no/bad data. Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. Two bug fixes. One, relative paths w.r.t. tags are now respected. Two, empty tags will no longer result in IndexErrors being raised. They are now caught and re-raised as IOErrors. Fixed the os.getcwd() vulnerability. Removed the pstat_decode messages that are put in the log file. Also, did a little code cleanup. Includes a fix for the way the full file paths are put together. Also, there is now a --get-bfid option for encp. Added legal charset test when setting a tag. Allow the --size option to take a long integer instead of just an integer. ========== option.py ==================================================================================== --volume for encp was added. (This is for early get tests.) remove remedy_ticket option add support for remedy tickets Minor modifications for newer (test) version of pychecker. Added the encp option --override-ro-mount. do not assign trace back of exc_info() to local variable add --pvols [--just] Fixed some --help related items. Long text lines getting printed instead of cut off and wrapped correctly. Added a help string for --usage. Removed the = from the --help info. If the short options list is long, it gets substituted with "[OPTIONS]..." for usage output. allow sending message in alarm client add --keep-vol Quick mod. to get around termios vs. TERMIOS problem. Modified option.py to work under python 2.0 and 2.2 without printing a warning message in python 2.2. Included new switches for encp. This is the first attempt to move encp to use option.py. Also, the new --all for enstore start/stop. added client name option Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. add --migrated new sgpg related changes add GVOL add support for the quota plot page Added the ability for entv to read in a command file for running in canned mode. Entv now has a --commands-file similar to the --movers-file. Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly. Include options for entv. The option is --dont-show. Note: the cvs log for entv.py accidentally refered to the old interface.py file instead of the newer option.py file. add --show-quota to volume Added --just option for start/stop. Added long as a type for options. add --trim-obsolete ========== ratekeeper_client.py ==================================================================================== A fix was made to prevent a traceback from occuring when the user executes "enstore ratekeeper". In line with other clients it now performs and implicit --help. Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". ========== host_config.py ==================================================================================== Slight fix to test if the specified enstore.conf file exists AND is a regular file. Note: it is still bad to tell encp to use a file as its enstore.conf file that is not a real enstore.conf file. Added a function update_cached_config() that rereads the enstore.conf file and updates the global cached information. There was a bug were a local variable could be accessed before having been set due to socket error. Said variable is now given a default value. Put in a some time delayed waits in case the DNS server is down for restart. More straitforward handling of errors raised if popen failes. Attempt to get the FQDN while determining the default ip address. ========== monitor_client.py ==================================================================================== These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Handle a possible exception better. add debugging info after a keyerror Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". Fixed the socket.socket() vulernability. ========== volume_assert.py ==================================================================================== Modified the request timeout time for the volume clerk query. One more option vs generic_client cleanup. Have the Interface class inherit from generic_client instead of option. Include timeouts incase the configuration server does not respond. Volume assert was broken. Is fixed now. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. ========== generic_client.py ==================================================================================== This is another attempt at fixing the generic_client resource leak problems. There is also a modification to change a print statement to a sys.stderr.write() call. use timeouts properly Second half of previous entry. Second attempt at removing the cyclic reference if the generic client is also a configuration client. back out Removed a resource leak. If the generic_client was also a configuration client, a cyclic reference was created causing the reference leak. Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". These modifications are in support of the new enstore_start and enstore_stop. Mostly having to do with the event relay dealing with 'alive' and 'quit' messages. ========== log_trans_fail.py ==================================================================================== modificaions needed to process new sys log message include low level tape diagn ========== monitor_server.py ==================================================================================== These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Fixed the socket.socket() vulernability. ========== event_relay.py ==================================================================================== add a enstore command support for event relay ========== enstore_show_inventory_cgi.py ==================================================================================== add COMPLETE_FILE_LISTING add VOLUMES_TOO_MANY_MOUNTS change VOLUMES_DEFINED to VOLUMES_DEFINED.html ========== file_clerk.py ==================================================================================== do not assign trace back of exc_info() to local variable add quit() add FILE_DB_USE_INDEX to switch on index file if there is none ========== library_manager_client.py ==================================================================================== Modified these clients to print the full --help output when no options are given. (i.e. "enstore library") It used to print just the shorter usage help output. This is to have all the server clients conform to a single look and feel. fixed interface probelm for delete option remove enstore_functions by using e_errors.is_ok() changed TO and retry pars. Fixed the printing of the active asserts heading. Before any work at movers would cause it to print, now just active asserts to so. ========== enstore_display.py ==================================================================================== Fixed a problem when using python 2_2_3. A syntax error would show up. Also, a color problem was fixed when a mover belongs to multiple libraries. Fixed some discrepencies when moving between states. This involved removing some displayed objects when a message to do so was dropped. Fixed some resource leaks w.r.t. "exc, msg, tb = sys.exc_info(). Also, modified some of the "master" variable code for python 2.2. Added the states 'HAVE_BOUND', 'FINISH_WRITE' and 'CLEANING' to the list of states to remove connections, rates, progress bars, etc. Fixed the problem of states sometimes being displayed with full text instead of being truncated to fit the space. Modified to support three columns of movers. The connection line now weaves through correctly from the third column. Added 'Unknown' to list of states to include for automatic client disconnections. Fixed the repositioning of the window. Also, another attempt to fix the resource leaks. Added some missing exceptions that should have been caught. Major modifications toword the removal of resource leaks. There were two leaks fixed (a third was from generic_client) that stemmed from cyclic refereneces. Part of this change was to stop having entv exec itself on certain errors. Other fixes include removing the rate, buffer bar and progress bar from the display when the mover moves into IDLE, ERROR, OFFLINE or Unknown state. Also, most (but not all) blanket excepts now only catch specific exceptions. Most remaing blanket excepts catch KeyboardInterupt first and re-raise. Previous commit broke displaying the connection lines in different colors. Modified entv.py to stop creating resource leaks. This included explicitly creating a master frame for the Canvas. Entv reinitializations under any situations no longer exec itself (which doesn't close open files.) Also, modified to handle the new mover state FINISH_WRITE correctly. Previously, FINISH_WRITE and the rate could have been displayed on to of each other. One other thing is that the second collumn's last mover fits in the window. Half was being "drawn" becasuse it is lower than the first column. It was this first collumn where the spacial calculations were done, now they take into account that the second collumn is lower. Cleaned up the code that locates the tcl/tk libraries. Changed comparing types of variables to type(()), type([]) or type("") to the coresponding value in modules types. If the mover is in "ACTIVE", "SEEK", "SETUP", "loaded", "MOUNT_WAIT", or "DISMOUNT_WAIT" to long the timer color will turn red. Fixed the os.getcwd() vulnerability. Modification to bring it in sync with entv.py. Bug fix when the domain name server is not available. A variable would be accessed but because of the lookup failure was never created. Hopefully fixed a problem with "host not found" errors. Added fit_string(). It will truncate strings to fit into the provided space. It is used on the state and volume names. First attempt to truncate strings to fit. Now the connection lines match the outline of the clients. Also, the mouse clicking on a connection will invert the color. Added code to color code groups of clients and movers. Improved error handling when reading in .entvrc file. Also, added functionality to remember if animation is on or off in the .entvrc file. Adjust the width of the mover rectangles based on the number of columns. Support running entv in a 'canned' mode. Use --movers-file as the switch to turn this support on. Requires 'canned' event_relay to function properly. Entv now uses interface.py for command line prarsing. The switch --dont-show allows the user to hide all movers belonging to a library manager. Reinitalize the display every so often. If the width is very small, then the image for the active state is suppressed. I consider the rate information to be more informative in this case. Made changes to the positioning calulation of things in the mover section. Fixed relocation bugs of the differenct percentage bars. Fixed the "How's on top?" problem between the media and network bars. Fixed the un-initialization problem of the volume label font size. Added an instantaneous rate display. Also added another bar that indicated percent on mover buffer filled. For the progress bar, two progresses are tracked one for network and one for media. ========== udp_client.py ==================================================================================== Cleaned up some code like the following: exc, msg, tb = sys.exc_info() to look like the following: exc, msg = sys.exc_info()[:2] This should help to avoid resource leaks. Made a fix such that it would send all "done_cleanup" messages before closing the socket. Previously it would close the socket after sending just one. Modification to return better error message for misconfigured /etc/hosts file. This seems to happen with default 7.3 fermi linux installs. ========== event_relay_client.py ==================================================================================== protect from socket.socket and bind errors These modifications are in support of the new enstore_start and enstore_stop. Mostly having to do with the event relay dealing with 'alive' and 'quit' messages. dont do sys.exit add a enstore command support for event relay ========== configuration_server.py ==================================================================================== remove 2 traceback leak problems Fixed a bug when attempting to build a log message. The status was not wrapped in (status,) format (since status is already a tuple). Fixed a bug preventing a reload of the configuration file. Modified the configuration server to use the event relay client to help manage socket.socket() volnerability. Fixed problem in dump(). Trace.trace() call was modified and was given to many arguments. Now the correct number exist. removed annoying pprint ========== udp_server.py ==================================================================================== Modified some Trace.trace levels. Also, added a Trace.trace() to the reply_to_caller_using_interface_ip() function. These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. If the socket() call failed, do not attempt to use it in the fcntl() call. Include conditional import for fcntl functionality. The FCNTL module is depricated in 2.2. ========== ratekeeper.py ==================================================================================== Modified how tracebacks are handled in the mainloop. This stemmed from fixing a sys.exc_info()/tb resource leak fix. Now Trace.handle_error() and rk.serve_forever_error() are called. Modified to avoid things like: exc, msg, tb = sys.exc_info() and instead do something like: exc, msg = sys.exc_info()[:2] Otherwise letting the traceback get set to a local variable (tb) causes a cyclic reference. Ratekeeper now uses the event_relay_client for handling event_relay related handling of the socket.socket() vulnerability. ========== configuration_client.py ==================================================================================== removed a print statement add debugging for network malaise remove redundancy in check Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". dump() didn't catch a possible TCP_EXCEPTION from callback. ========== media_changer_test.py ==================================================================================== fixed convert_version These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. ========== super_remove.py ==================================================================================== add --keep-vol ========== enstore_status.py ==================================================================================== ignore alarm lines weed out non- ENCP_XFER messages do not process debug messages print out encp errors too on history page use encp_ip when available on encp history page set disk_rate support new encp log file line format ========== cpio_odc_wrapper.py ==================================================================================== These files are partially modifed to address lingering cern wrapper bugs. ========== cern_wrapper.py ==================================================================================== These files are partially modifed to address lingering cern wrapper bugs. remove spurious print ========== file_clerk_client.py ==================================================================================== handle a strange error Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". honor alive timeout and retry from command line ========== log_server.py ==================================================================================== do not output misc messages to the extra log files fix bug in startup of log server fix extra files when not asked for fix bugs when starting add support for no debug log skip file if too big at beginning fix orig save logfile name fix restarting of log file optionally close log file at certain size ========== disk_driver.py ==================================================================================== fixed to comply with new mover reqs ========== enstore_functions2.py ==================================================================================== Fixed fullpath(). Moved the fullpath function from encp.py (and pnfs.py) to this file. add support for actions in alarm server ========== show_volume_cgi.py ==================================================================================== make pychecker happy make pychecker happy fix minor bugs using --gvol instead ========== checkBackedUpDatabases.py ==================================================================================== better way to determine if main backup is done ========== makeplot.py ==================================================================================== A full disk could cause this to traceback. Not it errors out gracefully. Fixed a bug that caused it to include to many days worth of files in the plot than it is supposed to. make plot backgrounds grey remove -modulate for new convert ========== log_client.py ==================================================================================== attempt to fix start/stop from unnecessary instantiation of lots of clients and delays cusing them to waste about a minute before starting doing anything Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". added client name option support yesterday as search time ========== scanfiles.py ==================================================================================== reduce unnecessary stat handle files younger than 1 hour try back door to gain permission exclude .B_* and .A_* ========== media_changer.py ==================================================================================== These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. diagnostics added be aware of 9940B media. Added trace ignore Enter fixed a bug in cleanCycle of stk class fixed a bug in cleanCycle of stk class added clean method to stk class add migrated state ========== volume_clerk_client.py ==================================================================================== fix a typo add --pvols [--just] allow L2 suffix for volume labels Mulitple client changes for support of improved start/stop. Mostly, theses changes have to do with timeouts of various functions. One notable change is that the monitor_server now looks for "monitor_server" in the config file instead of just "monitor". honor alive timeout and retry from command line tunning timeout and retry tunning timeout and retry tuning the timeout and retry add --migrated new sgpg related changes fix previous log message which is supposed to be: fixing missin si_time volume_clerk_client.py add --gvol add timeout for self.send fix --clear add --show-quota to volume added mount counter to vc and corresponding method add --trim-obsolete ========== enstore_overall_status.py ==================================================================================== change enstore to mass storage ========== media_changer_client.py ==================================================================================== Modified these clients to print the full --help output when no options are given. (i.e. "enstore library") It used to print just the shorter usage help output. This is to have all the server clients conform to a single look and feel. ========== get_total_bytes_counter.py ==================================================================================== add new libraries display TB on system pages transfer number as bytes not tb ========== get_all_bytes_counter.py ==================================================================================== display TB on system pages output in correct format strip off newline transfer number as bytes not tb change name of bytes file make a file with total bytes as an integer for sc2002. ========== enstore_show_inv_summary_cgi.py ==================================================================================== add COMPLETE_FILE_LISTING add VOLUMES_TOO_MANY_MOUNTS change VOLUMES_DEFINED to VOLUMES_DEFINED.html ========== library_manager.py ==================================================================================== do nothing in bad_volume. This is taken care of in mover_error changed version comparison log errror message fixed convert_version fixed convert_version code changes to address tape ingest issues code changes to address tape ingest issues do not set volume to noaccess while processing requests. This must be done in the error processing attempt to identify unpack non-sequence type error in bad_volume call changes in mover_error to process FTT_EBLANK error some mods for ktev request porcessing allow setting max_suspect_movers and max_suspect_volumes in the configuration file modified mover_busy to clean up at movers list for multiple LM per mover mods for multiple LMs Fixed /modified the code for multiple LMs serving one mover. These files have been modified to corectly handle which versions of python the FCNTL module should and should not be imported by. This is in responce to the FCNTL being depricated with 2.2 versions of python. fixed bug in delete_mover 5. Found why the requests may not go to movers even if the file family width allows. At least one case is understood.If the mover fails to mount tape it sends an error message to the library manager and then goes to the idle state. In the error message the mover,volume pair is mover,current_volume. In the idle message this pair is mover,previous_volume. When removing the entry from volumes_at_movers queue LM looks up for the pair mover,volume and it may not get removed because of different combinations described in the mover message. Now this is fixed. modified get_work_at_movers to search for pairs: label, mover. Before it was searching only for the label. This was wrong. Consider the following. Mover1 fails on VOL1. It send error rq and lm removes VOL1 from work at movers. Suppose VOL1 was dismounted and is available. Mover2 gets the VOL1 and now it is again in work at movers. But the error message from mover1 comes and removes this work from work at movers. trace added some minor changes some more logging replace 33 with 31 in trace replaced 13 with 33 in trace. 13 is used in config client and inteferes fixed bug added some diagnostics create path for a lock file if does not exist remove double entry fix problem with tape being set to naccess and not being considered by library manager added retrn value checks for vol_info add migrated state fix bug in mover_bound_volume causing LM to not update volumes_at_movers for HIPRI requests, which in turn causes a wrong calculation of busy volumes do not allow any, hence hipr rq. go if it is not already in the list of active tapes fix ambiguous logic in busy_volumes changed trace level for assigning write rqs LM with new queue manager added trace added some traces changed get in manage queue removed some traces to speedup the server added trace next_work_any_volume added traces to process_read_request some more mods for next_work_this_volume some more mods for next_work_this_volume modified next_work_this_volume to chech for restricted host access ========== generic_server.py ==================================================================================== Modified the generic server to avoid a sys.exc_info()/tb resource leak. Another still remains. ========== enstore_plots.py ==================================================================================== do not write a plot if there is no data make mount plot from accounting db info make plot backgrounds grey remove -modulate for new convert translate a / to a - in drive_id ========== manage_queue.py ==================================================================================== removed the remnants of the old adm_queue, and fixed the self.admin_queue.find call increase max pri to 1000001 removed some heavy diagnistics LM with new queue manager bakked off changes cnaged trace modified get and get_next added some traces added some traces added some traces added some traces added some traces added some traces added some traces changed get in manage queue bakked off the change removed some traces to speedup the server fixed a typo When request qeue contains admin and not admin requests, there was incorrect processing of ordinary requests skipping the 1st one in the queue. Now it is fixed in Request_Queue.get ========== dispatching_worker.py ==================================================================================== fix the annoying fd is not integer problem add debugging information If the UDPServer client failed to initialize (specifically to obtain its udp socket) then exit the server. Added a comment about a line of code.