ENCP release notes, from v3_8b to v3_9 Encp changes: ============= This encp version will work better with the new soon-to-be-in-production library manager, especially when Enstore is under heavy load. The new multi-threaded library manager can tell a mover to start a transfer for an encp still waiting for a response from the library manager. Older versions of encp only wait for one response at a time and may see increased timeouts; this version of encp can wait for both. dCache use only: If the owner of the output file in PNFS did not match that owner of the output directory in PNFS encp transfers would fail with an incorrect permission error. Now they will succeed. In previous versions, if the first attempt to write the original copy of a multiple copy write failed with a retry and the original copy retry succeeded; then attempts to write the multiple copies would not be attempted. (Such failures are recorded and external means are used to get all intended copies to tape.) Starting with this version if the original copy succeeds, regardless of the number of retries, the multiple copies will be attempted. The timeout for waiting for the control socket message has been reduced from 15 minutes to 30 seconds. It was observed that security scans were connecting to the listening socket and both sides would wait resulting in a denial of service attack. By reducing the timeout, the correct connection should be identified. v3_9a is v3_9, but with one important patch included that will keep encp, in some situations, from getting into a state where it may never fail and never succeed (bug #508). v3_9b is the same as v3_9a, but with a correctly compiled enstore command. v3_9c is almost the same as v3_9b, but with a patch to a sub-module that will keep encp from depending on the info_server in some rare situations (bug #544). Misc.: ====== New switches have been added to allow users to monitor the duplicated status of their files. "enstore info" --find-all-copies, --find-copies, --find-duplicates, --find-original, --find-the-orginal and --show-copies switches have been added. Detailed cvs commit logs ========== ecron_util.py ==================================================================================== extract only hosbname, drop domain take into accoutn that node can be fully qualified or not ========== duplicate.py ==================================================================================== Switched "yes" to migrate.YES from Sasha's comments in bugzilla 418. Compatibility changes with migrate.py 1.202. The migrate.py version of is_expected_volume() now takes a db argument. Even though this version doesn't use db, the argument list was modified to remain compatible. 1) Allow for the duplication of a duplicate. 2) Allow for make additional duplicates (i.e. _copy_2) if _copy_1 already exists). 3) Make --make-failed-copies work. For migration and duplication: 1) Fixed migrating deleted files. For duplication: 1) Honor --use-volume-assert for scanning. This involved reworking the code so duplicate.py used final_scan_volume() and final_scan_file() from migrate.py instead of its own (obsolete) duplicate.py versions. For Migration: 1) Prevent migrate.py --restore from working on duplicated files. Enable --force to work on supposidly completed destination duplicated volumes. Also, enable --scan-vol to work when the source and destination files have already been swapped. Fixed a bug where err_msg could be used before being set while 'swapping' metadata. Changed Trace.init(MIGRATION_NAME) to Trace.init(migrate.MIGRATION_NAME) so that duplication works again. Changed Trace.init("DUPLICATION") to Trace.init(MIGRATION_NAME). ========== enstore_functions3.py ==================================================================================== Added is_ip_addr(), which returns true if the string passed to it matches an IP V4 dotted decimal address. False otherwise. Fix to is_volume*() to accept deleted volumes as valid volume names. Allow old bfid formats in D0en to pass is_bfid(). Import just is_pnfsid() from pnfs.py. ========== duplication_util.py ==================================================================================== 1) Allow for the duplication of a duplicate. 2) Allow for make additional duplicates (i.e. _copy_2) if _copy_1 already exists). 3) Make --make-failed-copies work. Replaced using option.Interface() with enstore_functions2.default_host() and enstore_functions2.default_port() to avoid issues with swap_original_and_copy.py --help. Allow for swap_original_and_copy.py to fix duplications where the primary and multiple copies are out of sync with the contents of layer 4. Fixed a bug where a consistency check before duplication 'swapping' would fail incorrectly. The pnfs and database paths were getting normalized to the mountpoints currently mounted inconstently, leading to failures that should not happen. ========== dispatching_worker.py ==================================================================================== revert recent changes Replaced python 2.6 specific threading.current_thread() with python 2.4 compatible threading.currentThread(). udp_server.py: 1) Eliminated self.reply_address, self.current_id and self.client_number since they are not thread safe. 2) Added reply_to_caller2(). Differs from reply_to_caller() by taking two arguments, the first is the object to send, the second is a dictionary with the 'r_a' element. 3) Added a self.tsd (Thread Safe Data) member variable. Created the init_thread_specific_data() function to handle the initializing of the thread specific data. dispatching_worker.py: 1) Eliminated use of self.reply_address and other self.xxx member varialbles, since they are not threadsafe. 2) Updated invoke_function() to handle starting a function in another thread/process if the function name is in the self.run_in_parallel list. The default is to run it in the main thread. 3) Extend self.tsd.is_child to include spawned threads. 4) Added __getattr__() and __setattr__() to handle looking at self.xxx or self.tsd.xxx. 5) Removed dispatching_worker.run_in_thread(). dispatching_worker.DispatchingWorker.run_in_thread should be used instead. Moved the function processing logging/tracing time code to __invoke_function(). This will allow it to be accurate for all functions run in parallel, instead of reporting the time to create a new thread/process. Since the code is in test mode, the ability to run a function in thread or process has been added. At this time only the configuration server optionally uses this new feature. Limit the number of threads for the Enstore processes, when thread_name is None. If thread_name is given, then allow only one thread. revert back from Trace.log to Trace.trace in process_request replaced Trace.trace(5,...) with Trace.log(e_errors.INFO) Modified send_reply_with_long_answer_part1() to resend the udp reply every 5 seconds while waiting for the client to connect over UDP. make sure argument is tuple bug fix, use apply(function,args) use thread_wrapper for function invocation (needed to call after function) followed up on our e-mail exchange this morning - added function invoke_functon that can be overriden by subclasses. Modified file_clerk accordingly Patched generic_client and dispatching_worker to extend the client/server protocol to handle messages that are too long for UDP. The send() function in generic_client.py was modified and so was dispatching_worker.send_reply(). The function callback.connect_to_callback() handles most of the connection issues to establish the socket connection. This is for the opposite side of the already existing callback.get_callback() function. The connect_to_callback() function is used by generic_client.send(). In dispatching_worker.py, send_reply() may call the new function send_reply_with_long_answer() if reply_to_caller() fails becuase the reply message is too long. log error if thread fails to start code review changes removed confusing log lessage put get_fd_message into try --- except wrapper renamed run_in_thread to _run_in_thread to hide it from subclasses If dispatching worker is configured to use raw input it can run callback messages processing in a separate thread (event relay messages). It is enabled in raw mode. If there are multiple instances of we need only one callback messages processing thread. To disable spawnig this thread use disable_callback This use If using rwaUDP rawUDP.receiver starts in serve_forever method. This is done to provide spawning receiver thread (process) in a thread. Before it was done when UDPServer onbject was created (in main thread) define request in do_one_request more code for event_relay messages Receive event_relay messages on the specified in configuration port, Rather than using callback. This allows to receive and process messages via rawUDP In qiut() kill chidlen if using multiprocessing anf python version is 2.6 and later fixed a typo improved interval functions handling in combination with rawUDP some changes to better use rawUDP more modofication to use with rawUDP.c ========== configuration_client.py ==================================================================================== Corrected a comment. Added reply_serverlist(), only the monitor has used this configuration server feature and did so by using generic client. Also, added --copy to be able to specify the configuration_server's internal configdict copy level. These patches are related to getting the configuration server switched to using dispatching_worker's new send_reply_with_long_answer() function. Backward compatiblity with old clients and servers is still there. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Patched get_movers2() to honor the library_manager parameter. Removed most (but not quite all) traces of string exceptions. Added a log message after a Traceback was found 1-12-2009 on STKen. This will hopefully allow for a fix if it ever happens again. ========== inventory.py ==================================================================================== added standard header I renamed the new variables: rc_file3 => rc_contents_list3 n_recyclable3 => number_of_recyclable3 Added third category of recyclable volumes. This new category is for migrated, duplicated or cloned volumes. modify output log to indicate which volume is skipped and which volume is fully processed replaced last_access with modification_time ========== file_utils.py ==================================================================================== intriduce euid/geuid protected wrapper to call os module functions should replace many functions in this module. Make set_euid_egid reset the euid back to root if the egid needs to be changed, otherwise we don't have the permissions to change the egid. 1) added function rmdir to file_utils that effectively executes "rm -rf path" 2) modified checkdb.py so that it cleans up after itself not leaving old stuff behind. This change makes it immune to postgresql version changes These changes are in response to Sasha's review comments in preperation for releasing encp v3_9. Any exceptions from os.seteuid() and os.setegid() are now caught, the lock released and then the exception is re-raised. Also, the descriptive comment for the listdir() function was corrected. Added a wrapper for os.listdir(). Also, added an option to get_stat() to be able to use os.stat() by default or os.lstat() if need be. Added a flags argument to atomic.open2() to match that of the os.open() function. Modified file_utils.py, pnfs.py and enmv.py to use the new format. Added functions: open(), chmod(), chown(), utime() and remove() to handle locking the use of os.seteuid() around file operations. If the new euid and egid already are correct, skip setting it again. Refine the seteuid locking code. When going becomming unprivledge effective user from root, be sure to change the gid before uid. If the user id is changed first, the group id change fails. Removed leftover debugging print statement. ========== enstore_functions.py ==================================================================================== When we have an IP address, it is correct to use socket.gethostbyaddr() instead of gethostbyname[_ex](), since we have an IP address and not a hostname. This is for get_enstore_tmp_dir(). Modified get_enstore_tmp_dir() to compare with all the IP addresses for the node, not just the default hostname. The consequence of this was that it was trying to contact the configuration server when it should have looked at the local conf file and was causing a long delay. ========== pnfs.py ==================================================================================== Restrict a user from setting a file_family tag to end in -MIGRATION or _copy_# as these are mangled migration/duplication file_family patterns. This is for bugzilla #143. Added a flags argument to atomic.open2() to match that of the os.open() function. Modified file_utils.py, pnfs.py and enmv.py to use the new format. Handle the empty string passed as a pathname to is_pnfs_path(). Use file_utils.py functions open(), chown(), chmod(), utime() and remove() instead of the python provided versions. ========== info_client.py ==================================================================================== The commands --find-original, --find-the-original, --find-duplicates, --find-all-copies and --find-copies commands are now available to users through the info server. Admins can now use them using the "enstore file" or "enstore info" commands (though the "enstore file" variants points to the info server). In addition to this the enstore info --show-copies now works like it was supposed to. made indend 4 spaces to be consistent with the rest of the code fix bugzilla http://www-ccf.fnal.gov/Bugzilla/show_bug.cgi?id=437 print modification_time in enstore info -gvol ... in human readable format Fixed typos. long_answer should be long_reply. Use the new generic_client.send() long_reply argument for those functions that do need to obtain reponses with more information than UDP can handle. These changes have these Enstore servers using dispatching_worker.send_reply_with_long_answer() instead of their local copy of this function. Also, the commented out duplicated code has been removed too. file_clerk - minor printout change info_server, info_client: noticed that command like "enstore info --show-file CDMS115209320400002" do not work Fixed. Removed code after the return statement in tape_list3(). This somehow got left in. Allow for send_with_long_answer() and tape_list() to accept optional timeout and retry values. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Added backward compatiblity for new clients to fallback to trying old protocols for inquires with long answers. This is a very large patch. 1) The clients now use send_with_long_answer() to generically handle requests where the response is large. Servers, now have reply_to_caller_with_long_answer() to handle the same thing on their side. These functions use an "active" TCP protocol. 2) Better error checking for erroneous values. For example, if a command required a bfid but is supplied a pnfsid the user would get a cryptic value (at best) or no information from the DB query. Now, they would get something like "valueis not a bfid". This is done with the extract_value_from_ticket() family of functions. ========== operation.py ==================================================================================== Changed tab flipping output CAP for SL8500 at GCC to 0,5,0 (was 0,1,0), as per changes discussed at 10:30 Fri Nov 6 status meeting and recorded in elog http://ccfsrv2.fnal.gov:8181/Tapes+Log/7168 . ========== generic_client.py ==================================================================================== Removed unused import time When testing enstore_start.py it was noticed that in case of alive timeout the return value was always returnig "configuration_server". This fix will return a real offender name. Added to optional argument long_reply to send(). If None, then the default behavior of looking at the short reply to determine if a long answer needs to be obtained. Zero, for never getting the long answer . One, for always getting the long answer. In send(), don't use e_errors.ERROR for the unknown errors, use e_errors.UNKOWN instead. e_errors.ERROR is the log severity and not an error. This change is in response to Sasha's review comments in preperation for releasing encp v3_9. The send() function creates a connect_socket, but one error case referenced control_socket instead. Patched generic_client and dispatching_worker to extend the client/server protocol to handle messages that are too long for UDP. The send() function in generic_client.py was modified and so was dispatching_worker.send_reply(). The function callback.connect_to_callback() handles most of the connection issues to establish the socket connection. This is for the opposite side of the already existing callback.get_callback() function. The connect_to_callback() function is used by generic_client.send(). In dispatching_worker.py, send_reply() may call the new function send_reply_with_long_answer() if reply_to_caller() fails becuase the reply message is too long. Fixed occurences of msg.errno when msg has no errno attribute. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Catch TypeError exception. Without this the code runs into string exception when config server has no requested entry Removed most (but not quite all) traces of string exceptions. Fixed the recent change to have BAD STATUS go to stderr. Also, moved the exception catches of errno.errorcode[errno.ETIMEDOUT] to be the last exception tried. ========== enstore_html.py ==================================================================================== Added the ability for the new home of the weekly summary reports to be linked to from top enstore pages. To achieve this, html_main.py now accepts the --pages switch for creation of the generated html pages. ========== option.py ==================================================================================== Added --scan as a more generic switch to replace migrate.py's --scan-volumes switch. Added --debug for migration. Get --eject and --insert working for media_changer_client. Added --ioarea and --remove for media_changer_client. Added the ability for the new home of the weekly summary reports to be linked to from top enstore pages. To achieve this, html_main.py now accepts the --pages switch for creation of the generated html pages. Fixed a typo for --source-only to work correctly. Added --use-volume-assert for migration. Modified comment to reflect that encp has --skip-pnfs now. Added --destination-only and --source-only for migration. Added --insert and --eject for media_changer_client. Added the hidden --proceed-number option for migration. Added --use-disk-files for migration. Added --crc-check for volume_assert. ========== aml2.py ==================================================================================== Get eject() and insert() working. The AML/2 is going away soon, but at least the media_changer_client.py has been tested. ========== net_driver.py ==================================================================================== Changed receive (select) timeout from 5 to 1 min. This should be enough to receive a block of the data from network. ========== drive_utilization_plotter_module.py ==================================================================================== fix cut&paste problem ========== enstore_status.py ==================================================================================== fix the issue having to do with particular order of columns in "select acc_daily_summary.py accounting_client.py accounting_client.py.~1.21.~ accounting_client.pyc accounting.py accounting.py.~1.16.~ accounting.pyc accounting_query.py accounting_query.pyc accounting_server.py accounting_server.py.~1.29.~ accounting_server.pyc aci.py alarm_client.py alarm_client.pyc alarm.py alarm.py.~1.51.~ alarm.pyc alarm_server.py alarm_server.py.~1.53.~ alarm_server.pyc aml2_dummy.py aml2_log.py aml2.py atomic.py atomic.pyc backup_backup.py backup_client.py backup_client.pyc backup.py backup.pyc bfid_db.py blah.ps bytes_per_day_plotter_module.py bytes_per_day_plotter_module.pyc callback.py callback.py~ callback.py.~1.132.~ callback.pyc cern_wrapper.py change_loc_cookie.py change_s_i.py charset.py charset.pyc checkdb.py checkdbs.py check_pnfs_db.py cleanUDP.py cleanUDP.py.~1.35.~ cleanUDP.pyc config-filec config.o configuration_client.py configuration_client.py.~1.139.~ configuration_client.pyc configuration_server.py configuration_server.py.~1.143.~ configuration_server.pyc core.17621 cpio_odc_wrapper.py create_sg_db.py CVS db_compare.py db_dump.py db.py db_retrieve_backup.py dbs.py dcache_make_queue_plot_page.py dcache_monitor.py delete_at_exit.py delete_at_exit.py.~1.23.~ delete_at_exit.pyc discard_copy.py discipline.py discipline.pyc disk_driver.py disk_driver.pyc dispatching_worker.py dispatching_worker.py.~1.191.~ dispatching_worker.py.bak_before_merge dispatching_worker.pyc dispatching_worker.py.new drivestat2.py drivestat_client.py drivestat_client.pyc drivestat_server.py drive_utilization_plotter_module.py drive_utilization_plotter_module.pyc duplicate.py duplicate.py.~1.26.~ duplicate.pyc duplication_util.py duplication_util.pyc ecron_util.py edb.py edb.py.~1.35.~ edb.pyc e_errors.py e_errors.py.~1.101.~ e_errors.pyc ejournal.py ejournal.pyc encp_admin.py encp_admin.py.~1.1.~ encp_admin.pyc ENCPBIN encp.py encp.py~ encp.py.1.829 encp.py.~1.973.~ encp.py.bak2 encp.py.broken encp.pyc encp_rate_multi_plotter_module.py encp_rate_multi_plotter_module.pyc encp_ticket.py encp_ticket.pyc encp_user2.py encp_user2.py.~1.1.~ encp_user2.pyc encp_user.py encp_user.py.~1.1.~ encp_user.pyc encp_wrapper.py encp_wrapper.py.~1.7.~ encp_wrapper.pyc en_eval.py en_eval.pyc enmail.py enmv_admin.py enmv.py enmv_user.py enstore_admin.py enstore_alarm_cgi.py enstore_alarm_search_cgi.py ENSTORE_BIN ENSTORE_BIN_TEMP enstore_constants.py enstore_constants.py~ enstore_constants.pyc enstore_display.py enstore_display.py.~1.172.~ enstore_display.pyc enstore_display.py.test enstore_erc_functions.py enstore_erc_functions.pyc enstore_file_listing_cgi.py enstore_files.py enstore_files.py~ enstore_files.pyc enstore_functions2.py enstore_functions2.py~ enstore_functions2.py.~1.24.~ enstore_functions2.pyc enstore_functions3.py enstore_functions3.py.~1.7.~ enstore_functions3.pyc #enstore_functions.py# enstore_functions.py enstore_functions.py.~1.56.~ enstore_functions.pyc enstore_html.py enstore_html.py~ enstore_html.pyc enstore_log_file_search_cgi.py enstore_mail.py enstore_mail.pyc enstore_make_generated_page.py enstore_make_generated_page.pyc enstore_make_log_calendar.py enstore_make_plot_page.py enstore_make_plot_page.py~ enstore_make_plot_page.pyc enstore_overall_status.py enstore_pg.py enstore_pg.pyc enstore_plots.py enstore_plots.pyc enstore_plotter_framework.py enstore_plotter_framework.pyc enstore_plotter_module.py enstore_plotter_module.pyc enstore.py enstore.py~ enstore.py.~1.83.~ enstore.pyc enstore_restart.py enstore_restart.pyc enstore_saag_network.py enstore_saag_network.pyc enstore_saag.py enstore_saag.pyc enstore_show_inventory_cgi.py enstore_show_inventory_cgi.pyc enstore_show_inv_summary_cgi.py enstore_start.py enstore_start.py.~1.67.~ enstore_start.pyc enstore_status.py enstore_status.pyc enstore_stop.py enstore_stop.py.~1.32.~ enstore_stop.pyc enstore_system_html.py enstore_system_html.py~ enstore_system_html.pyc enstore_up_down.py enstore_up_down.pyc enstore_user2.py enstore_user_cgi.py enstore_user.py enstore_utils_cgi.py ensync_admin.py ensync.py ensync_user.py ENTV_BIN entv.py entv.py.~1.128.~ entv.pyc entv.py.from_sasha entv.tar esgdb.py esgdb.pyc espion.py estart.py estop.py event_relay_client.py event_relay_client.py.~1.32.~ event_relay_client.pyc event_relay_messages.py event_relay_messages.pyc event_relay.py fake_quota.py fake_quota.pyc fdbdump.py file_clerk_client.py file_clerk_client.py.~1.183.~ file_clerk_client.pyc file_clerk.py file_clerk.py.~1.286.~ file_clerk.pyc file_family_analysis_plotter_module.py file_family_analysis_plotter_module.pyc fileinfo.py #file_utils.py# file_utils.py file_utils.py.~1.14.~ file_utils.pyc find_pnfs_file.py find_pnfs_file.py.~1.11.~ find_pnfs_file.pyc flip_tab_acsls.sh flip_tab_das.sh frozen.o fs.py fs.py~ fs.pyc ftt_driver.py ftt_driver.pyc ftt.py ftt.py.~1.13.~ ftt.pyc generic_alarm.py #generic_client.py# generic_client.py generic_client.py.~1.71.~ generic_client.pyc generic_driver.py generic_driver.pyc generic_server.py generic_server.py.~1.48.~ generic_server.pyc get_all_bytes_counter.py get_cron_title.py get.py get.py.~1.137.~ get.py.2 get.pyc get_total_bytes_counter.py hello histogram.py histogram.pyc hostaddr.py hostaddr.pyc host_config.py host_config.pyc html_cmd.py~ html_cmd.pyc html_main.py idlemovers.py info_client.py info_client.py.~1.38.~ info_client.pyc info_server.py info_server.py~ info_server.py.~1.56.~ info_server.pyc inquisitor_client.py inquisitor_client.pyc inquisitor_plots.py inquisitor_plots.py~ inquisitor_plots.pyc inquisitor.py inquisitor.py~ inquisitor.py.~1.346.~ inquisitor.pyc interface.py inventory.py inventory.py~ inventory.py.~1.170.~ inventory.pyc journal.py label_tape.py library_manager_client.py library_manager_client.py.~1.163.~ library_manager_client.pyc #library_manager.py# library_manager.py library_manager.pyc library_manager.py.prod_updated library_manager.py.sahsa_new library_manager.py.sahsa_original lintit lm_list.py lm_list.pyc lm_que_length.py log_client.py log_client.py.~1.93.~ log_client.pyc log_finish_event.py log_server.py log_server.py.~1.78.~ log_start_event.py log_trans_fail.py log_trans_fail.pyc m2.py M_abc.o M__abcoll.o Makefile Makefile.~1.66.~ make_ingest_rates_html_page.py make_migrated_as_duplicate.py make_original_as_duplicate.py makeplot.py make_pyc_files.py manage_queue.py manage_queue.pyc match_syslog.py M_base64.o M_BaseHTTPServer.o M_bdb.o M_bisect.o M_cmd.o M_codecs.o M_collections.o M_copy.o M_copy_reg.o M_ctypes___endian.o M_ctypes__macholib__dyld.o M_ctypes__macholib__dylib.o M_ctypes__macholib__framework.o M_ctypes__macholib.o M_ctypes.o M_ctypes__util.o M_difflib.o M_dis.o M_distutils__dep_util.o M_distutils__errors.o M_distutils__log.o M_distutils.o M_distutils__spawn.o M_distutils__sysconfig.o M_distutils__text_file.o M_distutils__util.o M_doctest.o M_dummy_threading.o M_dummy_thread.o media_changer_client.py media_changer_client.py.~1.106.~ media_changer_client.pyc media_changer.py media_changer.py~ media_changer.py.~1.281.~ media_changer.pyc media_changer_test.py media_changer_test.py~ media_changer_test.pyc M_email__base64mime.o M_email__charset.o M_email__encoders.o M_email__errors.o M_email__feedparser.o M_email__message.o M_email__mime.o M_email.o M_email___parseaddr.o M_email__parser.o M_email__quoprimime.o M_email__utils.o M_encodings__aliases.o M_encodings.o M_FixTk.o M_fnmatch.o M_formatter.o M_ftplib.o M_functools.o M___future__.o M_genericpath.o M_getopt.o M_getpass.o M_gettext.o M_glob.o M_heapq.o M_httplib.o migrate.py migrate.py~ migrate.py.~1.202.~ migrate.pyc migration_scope.py migration_summary_plotter_module.py migration_summary_plotter_module.py~ migration_summary_plotter_module.py.~1.15.~ migration_summary_plotter_module.pyc M_inspect.o M_keyword.o M_linecache.o M_locale.o M_macurl2path.o M___main__.o M_mimetools.o M_mimetypes.o M_ntpath.o M_nturl2path.o module_trace.py module_trace.py.~1.2.~ monitor_client.py monitor_client.py.~1.91.~ monitor_client.pyc monitored_server.py monitored_server.pyc monitor_server.py M_opcode.o M_optparse.o M_os2emxpath.o M_os.o mounts_plot.py mounts_plotter_module.py mounts_plotter_module.pyc movcmd_mc.py mover_client.py mover_client.pyc mover_constants.py mover_constants.pyc mover-nanny.py mover.py mover.py.~1.1109.~ mover.pyc mover_summary_plotter_module.py M_pdb.o M_pickle.o M_pkgutil.o M_posixpath.o M_pprint.o mpq.py mpq.pyc M_py_compile.o M_pydoc.o M_pydoc_topics.o M_quopri.o M_random.o M_re.o M_repr.o M_rfc822.o M_shlex.o M_site.o M_socket.o M_SocketServer.o M_sre_compile.o M_sre_constants.o M_sre_parse.o M_ssl.o M_stat.o M_StringIO.o M_string.o M_struct.o M_subprocess.o M_tempfile.o M_textwrap.o M__threading_local.o M_threading.o M_Tkconstants.o M_Tkinter.o M_tokenize.o M_token.o M_traceback.o M_tty.o M_types.o multiple_interface.py multiple_interface.pyc M_unittest.o M_urllib.o M_urlparse.o M_UserDict.o M_uu.o M_warnings.o M_webbrowser.o net_directory.py net_driver.py net_driver.pyc null_driver.py null_driver.pyc null_wrapper.py null_wrapper.pyc on-call.py operation.py option.py option.py~ option.py.~1.178.~ option.pyc plotter_main.py plotter_main.py~ plotter_main.pyc plotter.py plotter.py~ plotter.pyc pnfs_agent_client.py pnfs_agent_client.py~ pnfs_agent_client.py.~1.21.~ pnfs_agent_client.pyc pnfs_agent.py pnfs_agent.py.~1.32.~ pnfs_agent.pyc pnfs_backup_plot.py pnfs_backup_plotter_module.py pnfs_backup_plotter_module.pyc pnfsidparser.py #pnfs.py# pnfs.py pnfs.py~ pnfs.py.~1.291.~ pnfs.pyc pnfs.py.saved priority_selector.py priority_selector.pyc put.py put.py.~1.4.~ put.pyc quickquota.py quota.py quota.pyc quotas_plotter_module.py quotas_plotter_module.pyc ratekeeper_client.py ratekeeper_client.pyc ratekeeper_plotter_module.py ratekeeper_plotter_module.pyc ratekeeper.pts ratekeeper.py ratekeeper.py.~1.41.~ ratekeeper.pyc ratemeter.py rate_test.py rawUDP_p.py rawUDP_p.pyc rawUDP.py rawUDP.pyc readonly_volumes.py readonly_volumes.py~ readonly_volumes.pyc recent_file_listing.py recent_file_listing.py.~1.9.~ recent_file_listing.pyc report_volume_quotas.py restoredb.py rm_volmap.py run1.py safe_dict.py scanfiles.py scanfiles.py~ scanfiles.py.~1.149.~ scanfiles.pyc scanfiles.py.SAVED send_alive.py set_lm_noread.py setpath.py setpath.pyc sg_db.py show_file_cgi.py show_volume_cgi.py sljdfsf slots_usage_plotter_module.py slots_usage_plotter_module.pyc stethoscope.py strace_out string_driver.py string_driver.pyc summary_bpd_plotter_module.py summary_burn_rate_plotter_module.py summary_burn_rate_plotter_module.py~ summary_burn_rate_plotter_module.pyc summary_plots.py swap_original_and_copy.py swap_original_and_copy.py.~1.2.~ swap_original_and_copy.pyc tab_flipping_exemption.py tab_flipping_nanny.py take_out_acsls.sh take_out_das.sh tapes_burn_rate_plotter_module.py tapes_burn_rate_plotter_module.py~ tapes_burn_rate_plotter_module.pyc time_fifo.py timeofday.py timeofday.pyc Trace.py Trace.py.~1.81.~ Trace.pyc udp_client.py udp_client.py.~1.170.~ udp_client.pyc udp_cl.py udp_common.py udp_common.py.~1.13.~ udp_common.pyc udp_load_test.py udp_load_test.py.~1.1.~ udp_load_test.pyc #udp_server.py# udp_server.py udp_server.py.~1.58.~ udp_server.pyc udp_server_test_15118 udp_server_test_16059 udp_srvr.py vdbdump.py volume_assert.py volume_assert.py.~1.25.~ volume_assert.pyc volume_assert_wrapper.py volume_assert_wrapper.pyc volume_clerk_client.py volume_clerk_client.py.~1.272.~ volume_clerk_client.pyc volume_clerk.py volume_clerk.py~ volume_clerk.py.~1.445.~ volume_clerk.pyc volume_family.py volume_family.pyc web_server.py web_server.pyc weekly_summary_report.py write_protection_alert.py www_server.py www_server.pyc YANKED yank.py from encp_xfer" and "select acc_daily_summary.py accounting_client.py accounting_client.py.~1.21.~ accounting_client.pyc accounting.py accounting.py.~1.16.~ accounting.pyc accounting_query.py accounting_query.pyc accounting_server.py accounting_server.py.~1.29.~ accounting_server.pyc aci.py alarm_client.py alarm_client.pyc alarm.py alarm.py.~1.51.~ alarm.pyc alarm_server.py alarm_server.py.~1.53.~ alarm_server.pyc aml2_dummy.py aml2_log.py aml2.py atomic.py atomic.pyc backup_backup.py backup_client.py backup_client.pyc backup.py backup.pyc bfid_db.py blah.ps bytes_per_day_plotter_module.py bytes_per_day_plotter_module.pyc callback.py callback.py~ callback.py.~1.132.~ callback.pyc cern_wrapper.py change_loc_cookie.py change_s_i.py charset.py charset.pyc checkdb.py checkdbs.py check_pnfs_db.py cleanUDP.py cleanUDP.py.~1.35.~ cleanUDP.pyc config-filec config.o configuration_client.py configuration_client.py.~1.139.~ configuration_client.pyc configuration_server.py configuration_server.py.~1.143.~ configuration_server.pyc core.17621 cpio_odc_wrapper.py create_sg_db.py CVS db_compare.py db_dump.py db.py db_retrieve_backup.py dbs.py dcache_make_queue_plot_page.py dcache_monitor.py delete_at_exit.py delete_at_exit.py.~1.23.~ delete_at_exit.pyc discard_copy.py discipline.py discipline.pyc disk_driver.py disk_driver.pyc dispatching_worker.py dispatching_worker.py.~1.191.~ dispatching_worker.py.bak_before_merge dispatching_worker.pyc dispatching_worker.py.new drivestat2.py drivestat_client.py drivestat_client.pyc drivestat_server.py drive_utilization_plotter_module.py drive_utilization_plotter_module.pyc duplicate.py duplicate.py.~1.26.~ duplicate.pyc duplication_util.py duplication_util.pyc ecron_util.py edb.py edb.py.~1.35.~ edb.pyc e_errors.py e_errors.py.~1.101.~ e_errors.pyc ejournal.py ejournal.pyc encp_admin.py encp_admin.py.~1.1.~ encp_admin.pyc ENCPBIN encp.py encp.py~ encp.py.1.829 encp.py.~1.973.~ encp.py.bak2 encp.py.broken encp.pyc encp_rate_multi_plotter_module.py encp_rate_multi_plotter_module.pyc encp_ticket.py encp_ticket.pyc encp_user2.py encp_user2.py.~1.1.~ encp_user2.pyc encp_user.py encp_user.py.~1.1.~ encp_user.pyc encp_wrapper.py encp_wrapper.py.~1.7.~ encp_wrapper.pyc en_eval.py en_eval.pyc enmail.py enmv_admin.py enmv.py enmv_user.py enstore_admin.py enstore_alarm_cgi.py enstore_alarm_search_cgi.py ENSTORE_BIN ENSTORE_BIN_TEMP enstore_constants.py enstore_constants.py~ enstore_constants.pyc enstore_display.py enstore_display.py.~1.172.~ enstore_display.pyc enstore_display.py.test enstore_erc_functions.py enstore_erc_functions.pyc enstore_file_listing_cgi.py enstore_files.py enstore_files.py~ enstore_files.pyc enstore_functions2.py enstore_functions2.py~ enstore_functions2.py.~1.24.~ enstore_functions2.pyc enstore_functions3.py enstore_functions3.py.~1.7.~ enstore_functions3.pyc #enstore_functions.py# enstore_functions.py enstore_functions.py.~1.56.~ enstore_functions.pyc enstore_html.py enstore_html.py~ enstore_html.pyc enstore_log_file_search_cgi.py enstore_mail.py enstore_mail.pyc enstore_make_generated_page.py enstore_make_generated_page.pyc enstore_make_log_calendar.py enstore_make_plot_page.py enstore_make_plot_page.py~ enstore_make_plot_page.pyc enstore_overall_status.py enstore_pg.py enstore_pg.pyc enstore_plots.py enstore_plots.pyc enstore_plotter_framework.py enstore_plotter_framework.pyc enstore_plotter_module.py enstore_plotter_module.pyc enstore.py enstore.py~ enstore.py.~1.83.~ enstore.pyc enstore_restart.py enstore_restart.pyc enstore_saag_network.py enstore_saag_network.pyc enstore_saag.py enstore_saag.pyc enstore_show_inventory_cgi.py enstore_show_inventory_cgi.pyc enstore_show_inv_summary_cgi.py enstore_start.py enstore_start.py.~1.67.~ enstore_start.pyc enstore_status.py enstore_status.pyc enstore_stop.py enstore_stop.py.~1.32.~ enstore_stop.pyc enstore_system_html.py enstore_system_html.py~ enstore_system_html.pyc enstore_up_down.py enstore_up_down.pyc enstore_user2.py enstore_user_cgi.py enstore_user.py enstore_utils_cgi.py ensync_admin.py ensync.py ensync_user.py ENTV_BIN entv.py entv.py.~1.128.~ entv.pyc entv.py.from_sasha entv.tar esgdb.py esgdb.pyc espion.py estart.py estop.py event_relay_client.py event_relay_client.py.~1.32.~ event_relay_client.pyc event_relay_messages.py event_relay_messages.pyc event_relay.py fake_quota.py fake_quota.pyc fdbdump.py file_clerk_client.py file_clerk_client.py.~1.183.~ file_clerk_client.pyc file_clerk.py file_clerk.py.~1.286.~ file_clerk.pyc file_family_analysis_plotter_module.py file_family_analysis_plotter_module.pyc fileinfo.py #file_utils.py# file_utils.py file_utils.py.~1.14.~ file_utils.pyc find_pnfs_file.py find_pnfs_file.py.~1.11.~ find_pnfs_file.pyc flip_tab_acsls.sh flip_tab_das.sh frozen.o fs.py fs.py~ fs.pyc ftt_driver.py ftt_driver.pyc ftt.py ftt.py.~1.13.~ ftt.pyc generic_alarm.py #generic_client.py# generic_client.py generic_client.py.~1.71.~ generic_client.pyc generic_driver.py generic_driver.pyc generic_server.py generic_server.py.~1.48.~ generic_server.pyc get_all_bytes_counter.py get_cron_title.py get.py get.py.~1.137.~ get.py.2 get.pyc get_total_bytes_counter.py hello histogram.py histogram.pyc hostaddr.py hostaddr.pyc host_config.py host_config.pyc html_cmd.py~ html_cmd.pyc html_main.py idlemovers.py info_client.py info_client.py.~1.38.~ info_client.pyc info_server.py info_server.py~ info_server.py.~1.56.~ info_server.pyc inquisitor_client.py inquisitor_client.pyc inquisitor_plots.py inquisitor_plots.py~ inquisitor_plots.pyc inquisitor.py inquisitor.py~ inquisitor.py.~1.346.~ inquisitor.pyc interface.py inventory.py inventory.py~ inventory.py.~1.170.~ inventory.pyc journal.py label_tape.py library_manager_client.py library_manager_client.py.~1.163.~ library_manager_client.pyc #library_manager.py# library_manager.py library_manager.pyc library_manager.py.prod_updated library_manager.py.sahsa_new library_manager.py.sahsa_original lintit lm_list.py lm_list.pyc lm_que_length.py log_client.py log_client.py.~1.93.~ log_client.pyc log_finish_event.py log_server.py log_server.py.~1.78.~ log_start_event.py log_trans_fail.py log_trans_fail.pyc m2.py M_abc.o M__abcoll.o Makefile Makefile.~1.66.~ make_ingest_rates_html_page.py make_migrated_as_duplicate.py make_original_as_duplicate.py makeplot.py make_pyc_files.py manage_queue.py manage_queue.pyc match_syslog.py M_base64.o M_BaseHTTPServer.o M_bdb.o M_bisect.o M_cmd.o M_codecs.o M_collections.o M_copy.o M_copy_reg.o M_ctypes___endian.o M_ctypes__macholib__dyld.o M_ctypes__macholib__dylib.o M_ctypes__macholib__framework.o M_ctypes__macholib.o M_ctypes.o M_ctypes__util.o M_difflib.o M_dis.o M_distutils__dep_util.o M_distutils__errors.o M_distutils__log.o M_distutils.o M_distutils__spawn.o M_distutils__sysconfig.o M_distutils__text_file.o M_distutils__util.o M_doctest.o M_dummy_threading.o M_dummy_thread.o media_changer_client.py media_changer_client.py.~1.106.~ media_changer_client.pyc media_changer.py media_changer.py~ media_changer.py.~1.281.~ media_changer.pyc media_changer_test.py media_changer_test.py~ media_changer_test.pyc M_email__base64mime.o M_email__charset.o M_email__encoders.o M_email__errors.o M_email__feedparser.o M_email__message.o M_email__mime.o M_email.o M_email___parseaddr.o M_email__parser.o M_email__quoprimime.o M_email__utils.o M_encodings__aliases.o M_encodings.o M_FixTk.o M_fnmatch.o M_formatter.o M_ftplib.o M_functools.o M___future__.o M_genericpath.o M_getopt.o M_getpass.o M_gettext.o M_glob.o M_heapq.o M_httplib.o migrate.py migrate.py~ migrate.py.~1.202.~ migrate.pyc migration_scope.py migration_summary_plotter_module.py migration_summary_plotter_module.py~ migration_summary_plotter_module.py.~1.15.~ migration_summary_plotter_module.pyc M_inspect.o M_keyword.o M_linecache.o M_locale.o M_macurl2path.o M___main__.o M_mimetools.o M_mimetypes.o M_ntpath.o M_nturl2path.o module_trace.py module_trace.py.~1.2.~ monitor_client.py monitor_client.py.~1.91.~ monitor_client.pyc monitored_server.py monitored_server.pyc monitor_server.py M_opcode.o M_optparse.o M_os2emxpath.o M_os.o mounts_plot.py mounts_plotter_module.py mounts_plotter_module.pyc movcmd_mc.py mover_client.py mover_client.pyc mover_constants.py mover_constants.pyc mover-nanny.py mover.py mover.py.~1.1109.~ mover.pyc mover_summary_plotter_module.py M_pdb.o M_pickle.o M_pkgutil.o M_posixpath.o M_pprint.o mpq.py mpq.pyc M_py_compile.o M_pydoc.o M_pydoc_topics.o M_quopri.o M_random.o M_re.o M_repr.o M_rfc822.o M_shlex.o M_site.o M_socket.o M_SocketServer.o M_sre_compile.o M_sre_constants.o M_sre_parse.o M_ssl.o M_stat.o M_StringIO.o M_string.o M_struct.o M_subprocess.o M_tempfile.o M_textwrap.o M__threading_local.o M_threading.o M_Tkconstants.o M_Tkinter.o M_tokenize.o M_token.o M_traceback.o M_tty.o M_types.o multiple_interface.py multiple_interface.pyc M_unittest.o M_urllib.o M_urlparse.o M_UserDict.o M_uu.o M_warnings.o M_webbrowser.o net_directory.py net_driver.py net_driver.pyc null_driver.py null_driver.pyc null_wrapper.py null_wrapper.pyc on-call.py operation.py option.py option.py~ option.py.~1.178.~ option.pyc plotter_main.py plotter_main.py~ plotter_main.pyc plotter.py plotter.py~ plotter.pyc pnfs_agent_client.py pnfs_agent_client.py~ pnfs_agent_client.py.~1.21.~ pnfs_agent_client.pyc pnfs_agent.py pnfs_agent.py.~1.32.~ pnfs_agent.pyc pnfs_backup_plot.py pnfs_backup_plotter_module.py pnfs_backup_plotter_module.pyc pnfsidparser.py #pnfs.py# pnfs.py pnfs.py~ pnfs.py.~1.291.~ pnfs.pyc pnfs.py.saved priority_selector.py priority_selector.pyc put.py put.py.~1.4.~ put.pyc quickquota.py quota.py quota.pyc quotas_plotter_module.py quotas_plotter_module.pyc ratekeeper_client.py ratekeeper_client.pyc ratekeeper_plotter_module.py ratekeeper_plotter_module.pyc ratekeeper.pts ratekeeper.py ratekeeper.py.~1.41.~ ratekeeper.pyc ratemeter.py rate_test.py rawUDP_p.py rawUDP_p.pyc rawUDP.py rawUDP.pyc readonly_volumes.py readonly_volumes.py~ readonly_volumes.pyc recent_file_listing.py recent_file_listing.py.~1.9.~ recent_file_listing.pyc report_volume_quotas.py restoredb.py rm_volmap.py run1.py safe_dict.py scanfiles.py scanfiles.py~ scanfiles.py.~1.149.~ scanfiles.pyc scanfiles.py.SAVED send_alive.py set_lm_noread.py setpath.py setpath.pyc sg_db.py show_file_cgi.py show_volume_cgi.py sljdfsf slots_usage_plotter_module.py slots_usage_plotter_module.pyc stethoscope.py strace_out string_driver.py string_driver.pyc summary_bpd_plotter_module.py summary_burn_rate_plotter_module.py summary_burn_rate_plotter_module.py~ summary_burn_rate_plotter_module.pyc summary_plots.py swap_original_and_copy.py swap_original_and_copy.py.~1.2.~ swap_original_and_copy.pyc tab_flipping_exemption.py tab_flipping_nanny.py take_out_acsls.sh take_out_das.sh tapes_burn_rate_plotter_module.py tapes_burn_rate_plotter_module.py~ tapes_burn_rate_plotter_module.pyc time_fifo.py timeofday.py timeofday.pyc Trace.py Trace.py.~1.81.~ Trace.pyc udp_client.py udp_client.py.~1.170.~ udp_client.pyc udp_cl.py udp_common.py udp_common.py.~1.13.~ udp_common.pyc udp_load_test.py udp_load_test.py.~1.1.~ udp_load_test.pyc #udp_server.py# udp_server.py udp_server.py.~1.58.~ udp_server.pyc udp_server_test_15118 udp_server_test_16059 udp_srvr.py vdbdump.py volume_assert.py volume_assert.py.~1.25.~ volume_assert.pyc volume_assert_wrapper.py volume_assert_wrapper.pyc volume_clerk_client.py volume_clerk_client.py.~1.272.~ volume_clerk_client.pyc volume_clerk.py volume_clerk.py~ volume_clerk.py.~1.445.~ volume_clerk.pyc volume_family.py volume_family.pyc web_server.py web_server.pyc weekly_summary_report.py write_protection_alert.py www_server.py www_server.pyc YANKED yank.py from encp_error" by using dictionaries instead of bare data. ========== file_clerk_client.py ==================================================================================== The commands --find-original, --find-the-original, --find-duplicates, --find-all-copies and --find-copies commands are now available to users through the info server. Admins can now use them using the "enstore file" or "enstore info" commands (though the "enstore file" variants points to the info server). In addition to this the enstore info --show-copies now works like it was supposed to. http://www-ccf.fnal.gov/Bugzilla/show_bug.cgi?id=422 take into account Mike's comments: Provided a mechanism for talking to the file_clerk or volume_clerk instead of the info_server. The default remains the info_server for commands that don't modify anything, but allowing a means for the clerks to be used in testing is useful. The trigger is --force on the command line; --force will not be listed by --help. added new constant : FILE_DELETED_FLAGS list of supported delete flags check value of deleted string in file_clerk_client and file_clerk. Added these functions to support migration working through the file clerk: find_migration_info() set_copied() unset_copied() set_swapped() unset_swapped() set_checked() unset_checked() set_closed() unset_closed() These changes have these Enstore servers using dispatching_worker.send_reply_with_long_answer() instead of their local copy of this function. Also, the commented out duplicated code has been removed too. Corrected the --mark-bad and --unmark-bad help strings to be more useful. Specifically in the area of when the optional [bfid] parameter should/needs to be included. Handle records with empty paths and pnfsid to avoid tracebacks. Some seteuid() patches to make --restore work in more situations. Fixed --list from undefined variable. This is a very large patch. 1) The clients now use send_with_long_answer() to generically handle requests where the response is large. Servers, now have reply_to_caller_with_long_answer() to handle the same thing on their side. These functions use an "active" TCP protocol. 2) Better error checking for erroneous values. For example, if a command required a bfid but is supplied a pnfsid the user would get a cryptic value (at best) or no information from the DB query. Now, they would get something like "value is not a bfid". This is done with the extract_value_from_ticket() family of functions. Modified the --restore command. If the --force switch is also used, then the file does not need to be marked deleted in the enstore DB nor removed from PNFS. --force is a hidden switch due to the obvious potential for abuse. ========== null_driver.py ==================================================================================== do not try to close file if its id is -1 ========== media_changer_client.py ==================================================================================== Return the location information for STK robots for the --show-volume command. Get --eject and --insert working for media_changer_client. Added --ioarea and --remove for media_changer_client. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Added timeout parameters to viewvol(). Removed most (but not quite all) traces of string exceptions. Go back to using list_volumes instead of list_volumes2. Added the internal pieces for insert and eject to work, but the key part of the code is still commented out to prevent their use. Added missing "self.list_clean = 0". ========== log_server.py ==================================================================================== revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. ========== aci.py ==================================================================================== Get eject() and insert() working. The AML/2 is going away soon, but at least the media_changer_client.py has been tested. ========== enstore_files.py ==================================================================================== Added the ability for the new home of the weekly summary reports to be linked to from top enstore pages. To achieve this, html_main.py now accepts the --pages switch for creation of the generated html pages. ========== e_errors.py ==================================================================================== Instead of having the python version exactly match (2, 6) it must be greater than or equal to (2, 6). Also, fixed the EnstoreErrors string representation to be EnstoreError instead of EncpError. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. New LM state added: MOVERLOCKED. In this state requests do not get expedited to movers. This state is used for testing. It helps to accumulate any number requests. Added new errors: TOO_MANY_FILES TOO_MANY_VOLUMES DATABASE_ERROR INVALID_ACTION ========== cleanUDP.py ==================================================================================== Removed Trace.handle_error() calls. Now that dispatching_worker and generic_client can handle long messages, these errors need to be passed up to those layers so that they can do the correct thing. removed debuggging print statement Fixed occurences of msg.errno when msg has no errno attribute. Removed most (but not quite all) traces of string exceptions. If sendto() gets a EMSGSIZE error, don't retry. Also, log information that would be useful for debugging: where the message was to be sent and the current stack trace. ========== event_relay_client.py ==================================================================================== Allow transferring messages on specofied port rather than selected internally. This allows communucation usin rawUDP ========== enstore_system_html.py ==================================================================================== Added link to active_volumes.sh (ticket 442) don not count shelf and test libraries use new fields in volume table Modified the creation of the top Enstore web page, so that the new home of the Weekly Summary Reports have a link pointing to them. Also, removed some code with pychecker warnings. use fast pizzabyte query on stken system ========== migration_summary_plotter_module.py ==================================================================================== Added a comment explaining why some min() functions were changed to max() functions. It explains that for migrated, recycled and migrated (again) tapes XXOO00 contains the combined volume history for XXOO00 and XXOO00.deleted and that max() is needed to correctly reported their dates of migration. 1) Corrected the started and migrated totals reported. 9940 tapes migrated and recycled back to 9940 tapes were resulting in an under-reported values. 2) The closed total was over-reporting the numbers. As long as one destination tape was scanned and recorded in the migration_history table the source tape was reported as scanned. Now it checks that at least one destination tape was scanned and closed AND that no source-to-destination pairs are left unclosed for the source tapes. Fixed plots when all the migrated tapes are ejected. ========== enstore.py ==================================================================================== Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Removed most (but not quite all) traces of string exceptions. ========== show_volume_cgi.py ==================================================================================== print show_size(total) change format switch to using client intertace rather than call command line ========== library_manager.py ==================================================================================== On line 3120 Trace.alarm was replaced with Trace.log (ticket # 454) Corrected a bug found during code review of bugzilla ticket 431. More changes to make Admin (HIPRI) requests to correctly override regular priority requests. Fixed bug in mover_error "IDLE" instead of IDLE Fixed bug in get_host_name_from_ticket created a method get_host_name_from_ticket to replace multiple occurences of the same code, recommended by Michael. 1. Backed off to rev. 1.658 2. If there are idle movers then do not attempt to preempt a volume, for which last work had regular pririty, by Admin Priority request. This dimishes the number of mounts/dismounts and improves the overall rates. In restric_host_access try to get hostname from callback address, this is important for multihomed hosts. Changed list to look in for busy movers in restrict_host_access. (bugzilla # 431) Do not make preemptive tape dismount if admin priority request is for would get rejected by discipline for idle mover. (bugzilla # 431) revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. When mover is removed from at_movers list do not send alarm, just log the error in check send alarm only once comments corrected Library manager may not be able to respond to a mover on time which results in the mover retry message. The mover requests may also be kept in the udp buffer before getting processed by the library manager. Both cases result in sending a work to the mover and then removing work from the active list as a result of the second idle request to avoid this problem compare the time of the mover request submission with the time when the work for the same mover became active. This code requires a new key in the mover ticket send alarm for a given transfer only once Do not remove work from active list, only send an alarm. Removing from active list causes tape mounts while they are still in use in another drive. in check() run everything in try - except to avoid problems when at_movers can be modified by another thread. It is not feasible to use locks in this case. 'MOUNT_WAIT' and 'DISMOUNT_WAIT' added to the check for max_time_in_active If mover is in state 'MOUNT_WAIT'or 'DISMOUNT_WAIT' for more that 10 minutes and still sending updates do not remove it from active movers list. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. if 'use_raw_input'=1 is specified the use rawUDP else do as before Reenabled discipline in HAVE_BOUND state. This is dynamically controlled by 'restrict_access_in_bound' entry in the configuration, and changes when configuration is reloaded. 'restrict_access_in_bound' can be set to True, False, None Removed most (but not quite all) traces of string exceptions. Corrected some log messages related to the handling on new config messages. ========== show_file_cgi.py ==================================================================================== switch to using client intertace rather than call command line ========== accounting.py ==================================================================================== added standard header take into account Mike's commens in bugzilla ticket 479 fix issue with IndexError reported by Mike process and fill library field in encp_xfer and encp_error table - tested on gccensrv1 ========== cern_wrapper.py ==================================================================================== replaced string exceptions ========== atomic.py ==================================================================================== Added a flags argument to atomic.open2() to match that of the os.open() function. Modified file_utils.py, pnfs.py and enmv.py to use the new format. 1. Added fuction for creating a unique file name. Before the file name was not unique. 2. Added some traces. ========== enstore_constants.py ==================================================================================== Added a comment per code review for ticket 435 Moved some common for library_manager.py and volume_clerk.py constants to enstore_constants.py This was done as followup on library_manager.py code review for LM scalability, bugzilla ticket 435 added new constant : FILE_DELETED_FLAGS list of supported delete flags check value of deleted string in file_clerk_client and file_clerk. Fixed the rate constants for LTO3 and LTO4 drives. The new values match those in the configuration. Modified the creation of the top Enstore web page, so that the new home of the Weekly Summary Reports have a link pointing to them. Also, removed some code with pychecker warnings. Added WEEKLY_SUMMARY_SUBDIR. A few other examples were listed, but they won't be used yet. ========== monitor_client.py ==================================================================================== Catch exceptions with respect to the recent python 2.6 changes. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Removed most (but not quite all) traces of string exceptions. ========== delete_at_exit.py ==================================================================================== Removed the now unused Container class. Replaced the local Container() class for providing thread specific data with the threading.local() class. Modified the detection of stale information in the thread_specific_data object. Added an explicit test in get_deletion_lists() if the obtained container object's thread ID does not match the current thread's ID. Don't depend on SIGRTMIN to be defined, but use it if available. Linux uses some signals not included in signal.h (and not included in the python signal module either) that are used to syncronize information between threads (like updating the uid & gid). We shouldn't install our own signal handlers for these signals. If the filename is a .(access)() name, this patch now finds the .(access)() name of the directory and the real basename of the file. For some reason, removing a .(access)() name from pnfs fails; it also fails to generate any type of error message. Hope this workaround helps. ========== tab_flipping_exemption.py ==================================================================================== Change the way to deal with the complaint of missing kerberos 4 ticket from certain version of klist. Now os.popen3() is used and the stderr is ignored. ========== encp.py ==================================================================================== Comment added in response to bugzilla #476 comment #5. This describes that USE_NEW_EVENT_LOOP, USE_LMC_CACHE, et. al. are not modified by the code, but are intended to be used by a developer/tester to intentionally "break" encp for a particular test. Modifications from Sasha's suggestions in Comment #8 of bugzilla #465. Additional patch for bugzilla #465 issue two. The original patch (1.965) worked for non-mulitple copy writes, but multiple copy writes need these extra changes. 1) If the library manager does not reply within 15 minutes of the initial message being sent; consider it a timeout error, reset the global __lmc variable and retry. 2) If handle_retries() gets an incomplete request ticket, see #1 for an example, assign the blame to the first non-completed request in the request list. Otherwise, the retry count updates are lost and encp may end up spinning endlessly without a chance of succeeding or giving an error. These are for bugzilla #476. Additional edge case to previous commit. If we had less then 30 seconds left before the 15 minute timeout expires while opening the control socket when a fake mover connection from the security scans connects; the loop would exit as if it had a successful connection and not an error. Now it raises a ETIMEDOUT error for this situation. Reduce the timeout period waiting for the control socket ticket to 30 seconds from the default 15 minutes. It was observed that security scans were connecting to encp and the bogus connections were preventing the movers from contacting encp. Removed some debuging output added to investigate bugzilla #465. Use the new pnfs_agent_client timeout specification. Previously, all timeouts were assumed to be "never." (bugzilla 465) Added an inner loop to write_to_hsm(). This change is to prevent double write re-submissions to the library manager. (bugilla 465) Handle errors from the LM better when writing multiple copies. This is for bugzilla #448. Patched encp to write any multiple copies if the original had a retry. Previously, if the original had a retry, then the internal accounting that the original had succeeded was lost when it came time to write the multiple copy. See bug #369. This patch is in response to bugzilla #365. It was possible for encp to use the wrong volume_clerk if the requested file was located on a tape in one Enstore system, but another Enstore system had a different tape with the same external label. This had been observed with NULL tapes, like NUL027 existing in GCCen and CDFen. Correct the bug report 361. For writes, some errors where being reported incorrectly. The known case was for writes where the tape mount took longer than the 15 minutes encp was waiting. The incorrect error was: ('TOO MANY RETRIES', ('NET_ERROR', 'Bad file descriptor')) It is now: ('TOO MANY RETIRES', ('UNKNOWN', "No data written to mover.") Reads did not have this problem. Modification to have encp return the correct error exit status (1 vs. 2) when used for migration. Fixed --max-resubmits and --resubmit-timeout. Modifications from suggestions from Sasha. Some exception raises in start() were replaced with "return 1". Calls to os.seteuid() and os.setegid() were wrapped with try/except. The list of handled exceptions from create_zero_length_pnfs_files() now has IOError and EncpError appened to it. When running the regression tests using pnfs_agent, a permission error was occuring; now the offending function, file_utils.chown(), is executed only if the real uid/gid is 0 (root). Major bug 147 addressed. encp.py revision 1.889 introduced a defect that only effected encp when encp_admin.py, encp_user.py or encp_user2.py were used to make a frozen version. The problem was that encp would always return 0, even for failures. This commit is just to reflect that v3_8 was actually made with encp.py 1.922 and not 1.955. v3_8 was made with code tagged bakken_tag_1_6_2009. bumping version to v3_8 because of encpCut Address bug #313. There were two spots in the code where the variable err_msg was being set to a string. It should have been setting one element in the err_msg dictionary to that string. Coordinate errors from encp to migration better. If encp returns an exit status of 2 the migration should not retry the encp. (This does the same thing that get and sdsscp did for the SDSS migration.) If necessary extract the starting copy number from the _copy_# file family passed in using --file-family. (This should only be used for duplcation.) Modified do_layers_exists() to work with pnfs or pnfs_agent. Instead of having the python version exactly match (2, 6) it must be greater than or equal to (2, 6). Replace calls to readlayer() with stat() calls. All that is needed is to know if there is anything in the layers and not the actual contents. Hopefully by avoiding reading in the entire layer, the performance will improve when there is heavy load. Added additional "Time to" messages for some checking steps inside handle_retries(). Also, added EnstoreError to the list of possible Exceptions caught from read_tcp_obj(). Call file_utils.py functions for hidding the modification of the euid/egid. Use the new file_utils.py chown(), chmod(), utime(), open() and remove() functions. 1) Handle EINTR if raised from select() in open_data_socket(). 2) Added some logging in stall_read_transfer() and stall_write_transfer(). Correct the usage of the file_utils.py seteuid locking functionality. Send to the debug log, information about the transaction id(s) we are waiting for. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Fixed a bug to get --override-deleted working again. This was likely introduced in revision 1.916. Fixed the previous commit to work on writes too. 1) Patch to put the non-migration path in the wrapper on the tape when writing for migration. 2) Set non-root uid/gid in the Enstore DB for migration. 3) Patches for reading deleted files with --override-deleted. Modifed encp to include the library information for encp_error and encp_xfer. Patch submit_one_request() to return the correct number of elements on error. Modifed the wait_for_message() event loop to use the udp_client repeat_deferred() and drop_deferred() functions. A geometric backoff of resending outstanding requests has also been implimented, similar to the one in UDPClient.send(). Added some diagnostics when becomming a different user after the file is transfered, but before the final stages are completed. Split up the last access and modification time updates into two functions. They can fail do to permissions when the euid does not match that of the file. For reads with --skip-pnfs, pass the pnfs_name0 value to appease NullMovers. Added --skip-pnfs as a hidden option. This only can take effect when --get-bfid is also used. When reading, if a directory in the target path does not exist, catch the OSError that verify_read_request_consistancy() raises. (The error is raised trying to determine if there is sufficent space on the filesystem for the file in question.) Modified the --copy switch help string have "(0 = primary)" instead of "(0 = original)". Added a missing return. Otherwise non-retriable errors trying to connect with the mover go ignored. Added additional "Time to" message for adjust_resubmit_request(). Also, Fixed "Atempting" to be "Attempting". These changes re-work the the looping of the code. The main change was the addition of wait_for_messages() that can wait for both the LM submision reply and the mover callback. Added "Time to" debug log messages for stall_read/write_transfer() functions. Also, upon some network errors, use callback.py's new MAC address finding capability to report additional information. Handle switching effective IDs when checking if layers 1 & 4 are emtpy or not. Modified encp when writing to add a handle_retries() call. The original error itself (in the known case the pnfs file was removed while encp was still running) was handled correctly, but file_utils.end_euid_egid() was called while not having the euid_lock acquired. Modified clients() so that volume_assert can use it without throwing tracebacks. ========== encp_admin.py ==================================================================================== Major bug 147 addressed. encp.py revision 1.889 introduced a defect that only effected encp when encp_admin.py, encp_user.py or encp_user2.py were used to make a frozen version. The problem was that encp would always return 0, even for failures. ========== udp_common.py ==================================================================================== changed one Trace call Added agrguments for modified en_enal call timing added ========== volume_assert.py ==================================================================================== This patch is in response to bugzilla #392. Volume assert now has more patience when submitting requests to the LM. Also, if a request is not successfully made, handle the situation better. Added stall_volume_assert(). If the tape is long, then we need to wait a while instead of encp's default 15 minutes which --crc-check is used. Also, removed the location_cookie list part of the --crc-check since the mover doesn't support that yet. Added a do_work() function to volume_assert similar to encp's do_work(). Sort the location cookies when outputing the results when --crc-check is used. Fixed a beginning message issue by using encp.clients(). Also, --crc-check will now request files be checked for CRC_MISATCH in addition to just the volume label. ========== ftt_driver.py ==================================================================================== Replaced string exceptions with exception class. Python does not support string exceptions beginning with version 2.6 Added log message to verify_label ========== ratekeeper.py ==================================================================================== revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. Added functionality for the ratekeeper to reconnect to the database after the connection to the database is lost. ========== accounting_client.py ==================================================================================== Modifed encp to include the library information for encp_error and encp_xfer. ========== enstore_start.py ==================================================================================== When we have an IP address, it is correct to use socket.gethostbyaddr() instead of gethostbyname[_ex](), since we have an IP address and not a hostname. Fixed enstore_start.this_host() to report all hostname and IP addresses for the node. The use_alias variables are no longer needed since all hostname and IP information is used every time. fixed syntax error on line 411 Post code review changes (bugzilla ticket 388). modified command on line 411 Post code review changes (bugzilla ticket 388). 1. Changed output: "Will execute:", cmd_list 2. Modified commands to exclude "enstore start", "enstore stop", "enstore restart" For multi-homed Enstore nodes, all IP addresses need to be checked, not just the default IP address. Removed some debug outputs. Removed some commented lines. 1. Better expression for checking if the componets to be started is not already running on start: if there is no response to alive, check with EPS before starting Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Replaced popen2.Popen3 with subprocess.Popen and pipeoblect.communicate and fixed bugs. import subrocess import subrocess popen2 is deprecated in last python releases. Replaced with subprocess Removed most (but not quite all) traces of string exceptions. ========== volume_clerk_client.py ==================================================================================== Provided a mechanism for talking to the file_clerk or volume_clerk instead of the info_server. The default remains the info_server for commands that don't modify anything, but allowing a means for the clerks to be used in testing is useful. The trigger is --force on the command line; --force will not be listed by --help. Added these functions to support migration working through the volume clerk: list_migrated_files() list_duplicated_files() set_migration_history() set_migration_history_closed() These changes have these Enstore servers using dispatching_worker.send_reply_with_long_answer() instead of their local copy of this function. Also, the commented out duplicated code has been removed too. handle modification_time correctly in gvol call This is a very large patch. 1) The clients now use send_with_long_answer() to generically handle requests where the response is large. Servers, now have reply_to_caller_with_long_answer() to handle the same thing on their side. These functions use an "active" TCP protocol. 2) Better error checking for erroneous values. For example, if a command required a bfid but is supplied a pnfsid the user would get a cryptic value (at best) or no information from the DB query. Now, they would get something like "value is not a bfid". This is done with the extract_value_from_ticket() family of functions. ========== pnfs_agent.py ==================================================================================== revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. use run_in_thread from dispatching_worker removed confusing log lessage turned off debug logging Reverted some changes. Changes triggered by py_checker 1. Modified creat 2. Adeed thread name to debug messages 3. Added debug messages set uid, gid when creating directories Fixed some key inconsistencies between client and server ========== en_eval.py ==================================================================================== Changes after code review 1. Modified function patterm 2. Use Types 3. A bit more optimal code 1. Additional arguments in en_eval check=True -- check for the presense of the function in expression. when rawUDP is used there is no reason to check incomong message 2 times: int rawUDP.put and udp_server.get_message compile=False use complier to check for the presense of the function in expression. Using compiler is very time consuming operation and anyways it is deprecated. If set to False (default) - use re.search for a function pattern this is about 20 times faster comparing to compiler Do not reraise NameError exception. Just return the value and let others to take care of this. This is related to event_relay messaging that has it own message format. reraise NameError if so happens timing added ========== enmv.py ==================================================================================== Added a flags argument to atomic.open2() to match that of the os.open() function. Modified file_utils.py, pnfs.py and enmv.py to use the new format. ========== get_total_bytes_counter.py ==================================================================================== increase timout communicating to configuration server correct incorrect if condition. Do float(row[0]) to avoid problem with Decimal do not count shelf and test libraries use new fields in volume table to extract pizzabytes use fast query on stken system to get byte counts ========== get.py ==================================================================================== Major bug 147 addressed. encp.py revision 1.889 introduced a defect that only effected encp when encp_admin.py, encp_user.py or encp_user2.py were used to make a frozen version. The problem was that encp would always return 0, even for failures. Removed most (but not quite all) traces of string exceptions. Removed a debugging print statement. These changes re-work the the looping of the code. The main change was the addition of wait_for_messages() that can wait for both the LM submision reply and the mover callback. Added setting request['method'] explicitly to "read_tape_start" before submitting to the LM. encp.py has changed and does not do this for us anymore. ========== scanfiles.py ==================================================================================== Catch exception to avoid deadlock. The exception itself is not understood, since root should never fail to set an effective ID; thus more error reporting is also added. Patched such that Ctrl-C works again. Also, moved file system related calls to use file_utils.py. Avoid tracebacks for partially written layer 4 information. Also, handle the situation for mulitple copy files that have attempted to be migrated. 1) Added code to give more accurate errors and warnings for files in the middle states of being migrated/duplicated. 2) Added the ability to make additional duplicates. This would involve making a _copy_2 file if the _copy_1 duplicate already exists. 3) Patched get_pnfsid() to use file_utils.open(). Deal with some reporting issues regarding migration/duplication files. Correctly identify situations were the duplicated files primary/copy status is out-of-sync between layer 4 and the active_files_copy table. Also, correctly identify where migration files are out-of-sync between layer 4 and the migration table. Added checks to make sure that extra lines are not existing in layer 1 or layer 4. The migration is picky. Fixed a traceback if the user specifies an input file that does not exist. Now an error message is displayed and the program exits (with error exit status) gracefully. Fixed a number of little issues regarding the various states of files during migration. It was reporting errors, just not always the correct one(s). Patch to prevent an uninitialized variable from being accessed. Handle multiple hsm, crc and size values in layer 2. Handle an error when the layer 2 size is set to something other than a number. Set the egid and euid back to root before listing a directory if the user has not given themselves permissions to list their own directory. Fixed a copy/paste error on a warning message. "size" => "crc" ========== accounting_server.py ==================================================================================== revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. process and fill library field in encp_xfer and encp_error table - tested on gccensrv1 ========== file_clerk.py ==================================================================================== http://www-ccf.fnal.gov/Bugzilla/show_bug.cgi?id=422 take into account Mike's comments: revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. added new constant : FILE_DELETED_FLAGS list of supported delete flags check value of deleted string in file_clerk_client and file_clerk. converted deleted to string.lower(deleted) so not to confuse people who specify enstore file --bfid GCMS125018340400000 --deleted NO Added interface functions, to support migration working through file clerk: find_migration_info() set_copied() unset_copied() set_swapped() unset_swapped() set_checked() unset_checked() set_closed() unset_closed() Support functions added: time2timestamp() __find_migration_info() __migration() __set_migration() __unset_migration() These changes have these Enstore servers using dispatching_worker.send_reply_with_long_answer() instead of their local copy of this function. Also, the commented out duplicated code has been removed too. discovered that enstore info --list wrong_volume_format breaks. This commit fixes it followed up on our e-mail exchange this morning - added function invoke_functon that can be overriden by subclasses. Modified file_clerk accordingly file_clerk - minor printout change info_server, info_client: noticed that command like "enstore info --show-file CDMS115209320400002" do not work Fixed. file_clerk: 1) in __tape_list use connection pool 2) spawn tape_list3 in separate thread 3) add two new parameters - max number of threads and maximum number of open database connections (which is passed to edb) edb: 1) add new data member connection pool (from DBUtils) 2) add new parameter - maximum open connection allowed in the pool fix unbound variable 1) fix pool instantiatin call 2) fix run_in_thread call call tape_list3 function in separate thread Removed a commented out section of code. Added a new parameter to __tape_list(), export_format, that should be set to False if called from tape_list2() and True if called from tape_list3(). Fixed some Trace.log() calls to report the correct names for what the function name is. Also, if reply_to_caller_with_long_answer_part1() does not return a control_socket, get_vols3() and get_pvols2() handle it now. Catch KeyErrors from failed csc.get() calls in FileClerkInfoMethods.__init__(). Fixed the code in __init__() that obtains the brand. If reply_to_caller_with_long_answer() is given a reply ticket with an error status, be sure to skip waiting for a connection after the UDP reply, because there will not a client trying to connect. Patched an issue with get_vols3(). The conversion to external formats section was left out. This is a very large patch. 1) The clients now use send_with_long_answer() to generically handle requests where the response is large. Servers, now have reply_to_caller_with_long_answer() to handle the same thing on their side. These functions use an "active" TCP protocol. 2) Better error checking for erroneous values. For example, if a command required a bfid but is supplied a pnfsid the user would get a cryptic value (at best) or no information from the DB query. Now, they would get something like "value is not a bfid". This is done with the extract_value_from_ticket() family of functions. ========== callback.py ==================================================================================== Added function get_unacked_packet_count(). This function returns the number of un-ACKed packets on the sender side. Patched generic_client and dispatching_worker to extend the client/server protocol to handle messages that are too long for UDP. The send() function in generic_client.py was modified and so was dispatching_worker.send_reply(). The function callback.connect_to_callback() handles most of the connection issues to establish the socket connection. This is for the opposite side of the already existing callback.get_callback() function. The connect_to_callback() function is used by generic_client.send(). In dispatching_worker.py, send_reply() may call the new function send_reply_with_long_answer() if reply_to_caller() fails becuase the reply message is too long. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Be more persistent in select()ing for writing to pipes. The Linux implementation seems to only return a pipe as writable if the send queue is empty, which is not what POSIX says should happen. It should return the pipe as writable as long as the queue is not full. Thus, this attempt for a work around. Removed most (but not quite all) traces of string exceptions. Renamed record_recv_error() to log_socket_state(). Use log_socket_state() to log sending/writing errors too. Modified write_raw() to loop until all the bytes to be written are written. It seems to be possible that trying to write something as short as 8 characters can be split up. Now it will retry to send each message fragment in its entirety. Attempt to log the MAC address of the remote machine upon a network related error. This will only work for machines on the same subnet. If we only read some of the bytes we are expecting, keep looping for the rest. ========== encp_user.py ==================================================================================== Major bug 147 addressed. encp.py revision 1.889 introduced a defect that only effected encp when encp_admin.py, encp_user.py or encp_user2.py were used to make a frozen version. The problem was that encp would always return 0, even for failures. ========== udp_server.py ==================================================================================== revert recent changes Removed some debugging output. udp_server.py: 1) Eliminated self.reply_address, self.current_id and self.client_number since they are not thread safe. 2) Added reply_to_caller2(). Differs from reply_to_caller() by taking two arguments, the first is the object to send, the second is a dictionary with the 'r_a' element. 3) Added a self.tsd (Thread Safe Data) member variable. Created the init_thread_specific_data() function to handle the initializing of the thread specific data. dispatching_worker.py: 1) Eliminated use of self.reply_address and other self.xxx member varialbles, since they are not threadsafe. 2) Updated invoke_function() to handle starting a function in another thread/process if the function name is in the self.run_in_parallel list. The default is to run it in the main thread. 3) Extend self.tsd.is_child to include spawned threads. 4) Added __getattr__() and __setattr__() to handle looking at self.xxx or self.tsd.xxx. 5) Removed dispatching_worker.run_in_thread(). dispatching_worker.DispatchingWorker.run_in_thread should be used instead. code review followup (bugzilla ticket 284) 1. Removed enable_raw_queue_print 2. Added comments for disable_reshuffling 3. in send_with_list do not send a reply if deepcopy failed 1. Added fuctions to control rawUDP 1.1. enable_raw_queue_print() - for debugging 1.2. disable_reshuffle() - when reinserting ticket into incoming queue do not change thge original place in the queue - this done for better processing of mover requests 1.3. set_keyword() - address messages in the queue by specified keyword instead of message id - this is for mover requests. Keyword: "mover" 2. in reply_with list() wrap list_copy = copy.deepcopy(list) into try...except see comments in reply_with list() for details. socket.sendto() silently truncates long UDP messages in python 2.4. In 2.6 an exception is raised. This patch catches long messages before they are truncated and raises the exception. Removed Trace.handle_error() calls. Now that dispatching_worker and generic_client can handle long messages, these errors need to be passed up to those layers so that they can do the correct thing. Modifications after code inspection. 1. Changed a raw message format. Not it is : (message, (sender_ip, sender_port)) 2. Use Types If using rwaUDP rawUDP.receiver starts in serve_forever method. This is done to provide spawning receiver thread (process) in a thread. Before it was done when UDPServer onbject was created (in main thread) added monitor Do not check message if used with rawUDP, because it is already checked in rawUDP changed trace statement process NameError and ValueError that could be risen by r_eval for envent_relay messages Turned on selection of rawUDP or rawUDP_p depending on python version Some message processing was moved out of _get_raw_message to rawUDP:put pychecker fixes Use multiprocessing for python 2.6 and later Send response on the port on which the message was recieved. This rule wqas broken in rawUDP and wqas preventing communications with firewalled clients. In process_request: if idn does not exist return None. Before it would cause a traceback with not processed KeyError set receive_timeout in rawUDP this udp_server uses rawUDP.py queue size is now the class attribute ========== edb.py ==================================================================================== fix syntax 1) fix pool instantiatin call 2) fix run_in_thread call call tape_list3 function in separate thread changes to handle modification_time in volume table Revision 1.36 of edb.py has a newly introduced bug. The else statement in str_value() needs to have the value variable wrapped with str(), to avoid concatination errors. If a value to be inserted into the database contains a single quote, special care needs to be done, since the single quote has special meaning to SQL. This patch handles single quotes correctly. Additionally, if PyGreSQL 3.8.1 or later is being used, then the pg.escape_string() function will be used instead of the local hack that only handles single quotes. ========== entv.py ==================================================================================== 1) Catch all exceptions while a thread has acquired a lock. 2) If the display thread hangs waiting for X, which has been observed using gdb, then re-exec() the process. 3) Instead of re-exec()ing when the number of objects increases, only re-exec() if more than 50% of the total system memory is consumed by the entv process. Fixed the --generate-messages-file switch. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Handle an error where the current systems 'known_config_hosts' and the target systems name for itself in its own 'known_config_hosts section are not the same. Catch udp_client.UDPError before errno.errorcode[errno.ETIMEDOUT] in handle_messages() for python 2.6 (and later). ========== migrate.py ==================================================================================== Include recomendations from review of bugzilla #418. Expanded comments in show_status() and better error message in restore() for an impossible situation that should never occur. In get_file_info() and get_volume_info() set the status field when accessing the DB directly. See bugzilla #427. 1) Added a seperate deleted queue to seperate active from deleted files. These two types of tapes go to different tapes, so we don't want one type waiting on the mount/dismount of the other. 2) Moving toward using the clerks or using the DB directly. The USE_CLERKS constant near the beginning of the file controls which to use. Currently it is set to false. 3) Inside pnfs_find() added __find_pnfsid_path(). PNFS will from time to time report ENOENT for files that really do exist. The loops a few times to see if the answer changes. (bugzilla #390) 4) When migrating a backup file of the Migration PNFS database, don't fail the migration because the basename contains "Migration". (bugzilla #410) Patch for bugzilla #370. In pnfs_find() there was one spot where the lock was not being released before the exception was raised. Moved a Trace.handle_error() call in pnfs_find(). Both the source and destination (if copied already) get checked, the error logging was happening after only the first was checked. When reading/scanning if an error occurs a check was added to see if the tape is NOACCESS or NOTALLOWED. If so, reading of additional files is stopped. Files already read to disk should be allowed to be written to tape. Added get_file_info() and get_volume_info(). They aren't used anywhere (yet), but they should prove to be useful in the future. 1) Detect and fix files marked as deleted in the Enstore DB that exist in PNFS. 2) Replaced --scan-volumes with --scan. Now a destination BFID can be give instead of just destination volumes. 3) Fixed the scanning of a multiple copy file to scan the multiple copy instead of the original copy. 1) Corrected an ordering issue with --status and multiple copies. 2) The proceed_number value is honored again. 3) Checks if the deletion lists are not empty after an encp returns. Reports an error if they are not empty and clears them before continuing. Added a return when writing to tape that indicates the write attempt failed due to a non-retriable error. Also, moved the return on one of the returns in the same function, write_file(), to the right one column where it belongs. If migrating a delete file where the original directory no longer existed; an error was occuring. The exeception was IOError; while only OSError was being caught. Added IOError to that except list in use_libraries() to fix it. Added code to detect when the migration path points to a ghost file on a re-run of the migration. When removing the output file before retrying the encp write, if the error remove() returns is ENOENT treat it as a success. Hopefully this patch will address bug #310. This issue is with makedirs() failing in PNFS, because it is succeeding to make a directory. Removed debugging output. Removed some debugging output. 1) Handle migrating a file to multiple tapes using encp's multiple copy functionality. 2) Handle migrating a tape filled with multiple copy files. 3) Fixed a condition where the file was written to the new tape, but not swapped. A re-run of the migration of that file continued to fail because the temporary file in /pnfs/fs/usr/Migration had been deleted. 4) Updated the --status output to include 'O' and 'M' for original and multiple copy, respectively, in the S column. Fixed a bug were the migration ignores extra libraries passed on the command line using --libraries. Use some code from scanfiles.py to better handle joining threads. Coordinate errors from encp to migration better. If encp returns an exit status of 2 the migration should not retry the encp. (This does the same thing that get and sdsscp did for the SDSS migration.) Patch the case were a deleted file is migrated to multiple copies. The issue was in the use of pnfs library information found in the tags versus the use of library information obtained from the user using --library. Some multiple copy files/tapes had migrations attempted before the check to prevent it from happening was added. This patch fixes --restore to be able to unmigrated a multiple copy file that had such a migration attempt made on it. At some point the code may be modified to support the migration of multiple copy files, but for now it is not allowed. 1) Allow for the duplication of a duplicate. 2) Allow for make additional duplicates (i.e. _copy_2) if _copy_1 already exists). 3) Make --make-failed-copies work. When writing, set the delayed dismount to 2 minutes for each multiple copy to be written. This is to avoid unecessary bouncing of tapes in and out of drives. Handle the rare situation where a deleted file now shares the same as as a directory. The remove() fails, but since the file type is wrong, we can conclude that it should not fail the scan. Accidentally left using volume assert on as the default. Now off by default. 1) Moved set euid/egid functionality to file_utils.py. 2) Fixed a bug that would cause a thread to exit early, but not be joined or report any errors. Any future unexpected errors like this will now be logged too. 3) Fixed a queueing bug that was the root cause of #2. This should solve the infamous "hanging" problem. 4) Fixed being able to Ctrl-C a migration process. Resolved a number of deadlocks. Patch to fix --scan-vol after the last patch. Modified to be able to keep multiple encp requests going at once to keep the LM queue full. Fixes a number of issues with respect to multi-threaded/multi-process migration. (There still is a deadlock situation, so this is currently disabled.) Patch to read files with a space in their pathname. Fixed is_migrated() to ignore deleted files that don't have a pnfs_id set even when --with-deleted. Also, patched --status to marked files in this skipped state with an E in the bad files collumn (even though they aren't 'B' bad files). Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. For migration and duplication: 1) Fixed migrating deleted files. For duplication: 1) Honor --use-volume-assert for scanning. This involved reworking the code so duplicate.py used final_scan_volume() and final_scan_file() from migrate.py instead of its own (obsolete) duplicate.py versions. For Migration: 1) Prevent migrate.py --restore from working on duplicated files. Added checks to detect if a tape belongs to a non-existent library. Most likely this will stop the migration/duplication of test or shelved tapes. Fixed a situation where a volume is migrated and scanned, except for a bad file.If on the second pass --skip-bad is used, no files will match (since the sources are deleted after the destination is scanned) and the migration was stopping on the empty volume check. Now the code includes adding migrated (aka copied) files to the list of files even if they have been marked deleted after the destination is scanned; thus avoiding the empty volume check. Modified an error message to give more useful information. Added check to prevent the migration, duplication or clonning of an empty volume. Modified the --restore functionality to handle errors where the swap step is recored as being done, but the layer information was not. Handle EPIPE gracefully when the pipe to less, more, head, etc. is closed. Patched the --status to handle tapes that have been involved in both migration/cloning and duplication as a source and destination volume. Patched a bug where a failed stat() would lead to a traceback. Also, patched a bug when running the migration against an already migrated volume. Two patches to get_migration_type(): 1) If a source tape has not been started, show_status() will pass the dst_vol as None to get_migration_type(). get_migration_type() simply return None immediatly now. 2) Duplicated tapes will now not show up as confused between migration and duplication now that the migration query has been improved. Just some code cleanup and new comments. Patched get_migration_type() for --status to handle more extreme situations. 1) A volume is a partially migrated to volume and it itself has been started to be migrated. (very odd, most likely caused by human error) 2) If a source volume's system inhibit is set to something like 'readonly' or 'full', the --status output can still determine if migration or duplication was used. Added use of --use-volume-assert for scanning. Also, patched the temp_file() function to handle disk files without crashing. Lastly, patched to skip the temp_file() call if the file has already been migrated (since it has a name that may have changed). Added function run_in_parallel() that will either call run_in_thread() or run_in_process() depending on if USE_THREADS is true or false. If --library is used, the correct output media_type is found. Previously, it falsely used the library as the media_type string. Removed most (but not quite all) traces of string exceptions. Turn off all the fancy fork()ing code. There are some interesting syncronous errors that need more invenstigating. Modified a lot of code to use threads and/or processes. Now if USE_PROCESS_PER_FILE_TRANSFER is true, then multiple processes are generated to handle issuing encps; keeping requests in the queue and minimizing any time gaps between transfers. The --status output generation has been improved. The most obvious change being the addition of the 'B' column where a bad file is marked with a 'B'. Patched --restore to assume the correct euid/egid for updating layer 1 and 4. Fixed --status output with the placement of the D field information for files that were failures when writing to the new tape. Removed a debugging change to add "--verbose 4" to the encp command line. Changed the incorrect ST_MODE to ST_SIZE for comparision with the file DB size value. --use-disk-file now seems to work. 1) Patched --restore for more permissions issues. 2) Modified open_log() to convert non-strings to strings to avoid crashes. 3) If USE_PROCESS_PER_FILE_MIGRATION is true, then issue a new process for each file to migrate (which in turn splits into two; for reading and writing); instead of one process for all reads and another for all writes. This is an attempt to keep files in the LM queue to avoid delays. 4) Fixed some volume_assert usage bugs. Disable use of volume_assert for scanning until the movers are updated. If USE_VOLUME_ASSERT is true, use volume assert to scan the destination volumes instead of multiple encps. If the select in get_requests() is interupted with a signal it will give an exception with errno EINTR, which is not really and error, but reason to retry the select. This should get things going again. 1) Added --proceed-number hidden switch. 2) Fixed yet another effective ID switching bug. 3) Added check of Migration directory to make sure the permissions are correct. Include the deleted status information in the --status output for the source and destination bfids. 1) Fixed permission bugs/issues with the deleting the mig_path after the new copy is scanned. 2) Attempting to add code to ingnore SIGTERM, SIGQUIT and SIGINT during the swap step. 3) Added --use-disk-files to take the source from files already on disk. To be use for tapes sent for recovery where the file locations on the returned tape are different from the original. 4) Patched --restore to work on unswapped (in addition to swapped) copied files. Needed for undoing some errors. ========== encp_wrapper.py ==================================================================================== Added error message to catch intermitant problem of missing error message from encp. Access the encp error message value in a thread safe way. 1) Accept the encp command as a list of argv values instead of as a string. Spaces in filenames caused the old implimentation to fail. 2) Set the effective UID and GID back to root after the encp. Minor patch to make cutting-and-pasting easier in the future for making other wrappers. This change also keeps module similar to the new volume_assert wrapper module. ========== udp_client.py ==================================================================================== chnged trace Be more verbose in the unit_test output. Include the current thread name, and callback address in that output. Moved the thread_specific_data to a member of UDPClient() class instead of being a global. Removed use of the local Container() class to hold thread specific data and replaced it with the threading.local() class. Also, modified the unit test to instantiate the UDPclient() class in one thread, but run the send/receive test in another. This is in response to bug #367. Catch a key error trying to delete a sometimes non-existent ID from a list of transaction IDs. Added parameter exponential_to to send() by default exponential_to is set to False if set to True in indefinite retry mode the timeout will grow exponentially Otherwise in indefinite retry mode the timeout will be constant Handle the situation where a message, using repeat_deferred(), is going to be sent, but it has already been accepted by another thread and no longer in the sent-but-not-received queue. Added recv_deferred2(). This returns a tuple of the message and the transaction ID instead of just the message like recv_deferred() returns. Use expnential reply timeout only if initial timeout is set to 0. Which means: try forever. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. 1. Use same timestamp for all messages for the same UDPClient object as was set originally 2. Do not include txn_counter into message id. These changes are done to identify "retried" messages in rawUDP create a new time stamp for each message. Before it was creted once, when UDPClient object was created Make message identifier unique for multithreded processes. Added functions repeat_deferred() and drop_deferred(). They are used when the caller schedules its own resending timeout schedule. Grow receive respose timeout by 2^n, where n - number of retries. The growth stops when n==MAX_EXPONENT MAX_EXPONENT=6 Raise UDPError instead of errno.errorcode[errno.ETIMEDOUT]. This is to avoid raising string exceptions that python 2.6 no longer supports. ========== enstore_display.py ==================================================================================== Sasha reported a bug. The issue is that the state change message was not received, a transfer message was received and when entv went to update the new instantaneous rate it ws failing because the state was not ACTIVE. Some of the interactive features were broken with the last patch. They are working now. 1) Catch all exceptions while a thread has acquired a lock. 2) If the display thread hangs waiting for X, which has been observed using gdb, then re-exec() the process. 3) Instead of re-exec()ing when the number of objects increases, only re-exec() if more than 50% of the total system memory is consumed by the entv process. Instead of using images for the active and positioning icons, draw them instead. ========== put.py ==================================================================================== Same change made to get.py revision 1.138. See bugzilla 337. This is from the encp bug where encp.py would return the correct exit status, but frozen encps would always return success. These changes re-work the the looping of the code. The main change was the addition of wait_for_messages() that can wait for both the LM submision reply and the mover callback. ========== library_manager_client.py ==================================================================================== Fixed bug on line 473 moverlock state added. ========== html_main.py ==================================================================================== Added the ability for the new home of the weekly summary reports to be linked to from top enstore pages. To achieve this, html_main.py now accepts the --pages switch for creation of the generated html pages. ========== histogram.py ==================================================================================== simplify statistical calculations and remove extra members prov ========== volume_clerk.py ==================================================================================== Moved some common for library_manager.py and volume_clerk.py constants to enstore_constants.py Moved hack() to volume_family.py and renamed it to match_volume_families() This was done as followup on library_manager.py code review for LM scalability, bugzilla ticket 435 Replace lines using e_errors.FILE_CLERK_ERROR with e_errors.VOLUME_CLERK_ERROR. This is due to comments from Dmitry in Bugzilla #394. revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. Added interface functions, to support migration working through volume clerk: list_migrated_files() list_duplicated_files() get_migration_history() set_migration_history() set_migration_history_closed() Support functions added: _migration_history() These changes have these Enstore servers using dispatching_worker.send_reply_with_long_answer() instead of their local copy of this function. Also, the commented out duplicated code has been removed too. update modification time on volume touch removed stupid and misleading print from set_remaining_bytes call Patched reply_to_caller_with_long_answer_part1() to have the latest patches that file_clerk.reply_to_caller_with_long_answer_part1() has. Catch KeyErrors from failed csc.get() calls in VolumeClerkInfoMethods.__init__(). Fixed new_library(). An if statement had an extra "not" in it. This is the same type of defect as was fixed in revision 1.434 regarding clr_system_inhibit(). changes to handle modification_time in volume table Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Fixed clr_system_inhibit(). An if statement had an extra "not" in it. put "limit 1" on a query that chacks if volume has non-deleted files Use timeouts when querying the MC for the state of a volume before recycling it. If the media_changer is down, we don't want to wait indefinately. The addvol() function had a copy-paste error. It should have looked for "external_label" in the ticket, but was looking for "vol" since that piece of code was copied from set_comment() and set_comment() looks for "vol". If reply_to_caller_with_long_answer() is given a reply ticket with an error status, be sure to skip waiting for a connection after the UDP reply, because there will not a client trying to connect. Minor patch to fix get_vols3(). This is a very large patch. 1) The clients now use send_with_long_answer() to generically handle requests where the response is large. Servers, now have reply_to_caller_with_long_answer() to handle the same thing on their side. These functions use an "active" TCP protocol. 2) Better error checking for erroneous values. For example, if a command required a bfid but is supplied a pnfsid the user would get a cryptic value (at best) or no information from the DB query. Now, they would get something like "value is not a bfid". This is done with the extract_value_from_ticket() family of functions. ========== volume_family.py ==================================================================================== Added a comment per code review for ticket 435 Moved hack() to volume_family.py and renamed it to match_volume_families() This was done as followup on library_manager.py code review for LM scalability, bugzilla ticket 435 ========== generic_server.py ==================================================================================== Removed most (but not quite all) traces of string exceptions. Corrected some log messages related to the handling on new config messages. Spelling fix for comment. eiter -> ether ========== inquisitor.py ==================================================================================== revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. revert back from combining two exceptions into tuple. Sasha explained that this might have unpredictable behaviour when we switch to python 2.6.2 BUG FIX: Fixed uninitialized variable use. Name threads as "class_name.method_name" remove confusing log message removed the bug intriduced in 1.342 (removed return in case thread with given name is found) this was fatal bug implemented code review recommendations put execution of periodic tasks into separate named threads. Check if there is already running thread with the same name as function and skip execution if such thread is found. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. 1) import callback 2) change "msg" -> "message" on line 598 threaded inquisitor Removed most (but not quite all) traces of string exceptions. ========== e_errors.py ==================================================================================== Instead of having the python version exactly match (2, 6) it must be greater than or equal to (2, 6). Also, fixed the EnstoreErrors string representation to be EnstoreError instead of EncpError. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. New LM state added: MOVERLOCKED. In this state requests do not get expedited to movers. This state is used for testing. It helps to accumulate any number requests. Added new errors: TOO_MANY_FILES TOO_MANY_VOLUMES DATABASE_ERROR INVALID_ACTION ========== mover.py ==================================================================================== Added library_list and current_priority to library manager ticket in disk mover Return correct errors when assert fails.(bugzilla ticket 460) Also changed some trace lavels. Removed locally defined CRC_ERROR and replaced it with e_errors.CRC_ERROR Code review followup for ticket 463 Mover incorrectly alarmed CRC error when encp interrupts transfer (ticket 463) In assert mode mover was failing when encp was gone because self.header_labels was not defined (bugzilla #430) For cern wrapper mover was incorrectly treating interrupted writes (ENCP_GONE). This was causing a tape to get set to NOACCESS. (bugzilla #430). This pacth fixes this problem. Additional changes: changed trace levels for often repeated output to deeper levels. revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. Changes in responce to code review comments: 1. Added comments to log_processes 2. Changed the order of set_system_readonly and Trace.alarm in some places. 1. Change error logging to alarming for cases when tape is set to readonly due to write error. 2. Set tape to NOACCESS when it is set to readonly. This change may be reverted later. 3. log all running processes when write error occurs. Add log_processes method These modifications address problem tickets: 349, 350, 352 Replaced string exceptions with exception class. Python does not support string exceptions beginning with version 2.6 1. Chaged controll socket connect timeout from 15 to 5 and the number of retries from 4 to 3 This should be more than enough. ENCP should already wait for connection. 2. Changed data socket connection to from 60s to 20s. This also should be enough. added current_time to lm ticket fixed crc check drivestat log message new drivestat log message format added log message to record read and write errors reported by drive minor spell fixes added timing for tape read and write operations modified no_work set lm update flag in assert mode back off changes back off changes set reply timout to self.interval in update_lm() Follow up on the code review. 1. Exit from check_connection when control socket is closed (after successful completion of assert or after error) 2. Put repeated code into shell_command more changes for single file mark mode fixed a logic flaw in dismount_volume which was causing an attempt to write file mark in a single file mark mode if the write failure was caused by TAPE or DRIVE. 1. Fixed thread synchronization problem in assert_volume. 2. replaces deprecated popen2 with subprocess do not dismount tape after ASSERT In assert mode: If client dropped connection, stop reading files and dismount volume. Typo fixed Explicitely specify timeout for fcc.tape_list (300, 2) in assert_volume In assert mode: if file clerk did not respond error out gracefully cern wrapper in assert mode was handled incorrecly added separate update interval for HAVE_BOUND defined by update_interval_in_bound key in config dict added lost changes for hostname in assert mode. Some minor improvement in assert_vol Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. replaced a number with constant for max buffer Restrict max_buffer by the allowed in python list size Merged with changes to eliminate string exceptions. 1. fixed uprocessed IndexError exception in read_tape 2. In DiskMover fixed unprocessed OSError exception on tape.open Removed most (but not quite all) traces of string exceptions. Instead of just doing "raise MemoryError" in stream_read(), do "raise MemoryError(message)" where message is a string describing what what happened when the error occured. Hopefully, this will yield more information about the MemoryErrors that occur while trying to migrate the D0 bphysics file family. In connect_client() self.tmp_vf is only set for writes. This patch modifies two if statements to prevent self.tmp_vf from being accessed during reads. The code was broken some time ago to work with "get" as a client. This patch fixes this bug. changed state to state_name: cosmetic change modified assert to return crc check results for every file modified code to read files and check crc in ASSERT mode. This can be used to accelerate scans after migration if mover_port is defined in configuration use it to communicate with library manager ========== encp_user2.py ==================================================================================== Major bug 147 addressed. encp.py revision 1.889 introduced a defect that only effected encp when encp_admin.py, encp_user.py or encp_user2.py were used to make a frozen version. The problem was that encp would always return 0, even for failures. ========== find_pnfs_file.py ==================================================================================== Differentiate between a file not existing at all and a file that has been replaced. Replace os module calls with file_utils.py calls. When reporting the error that a file recorded as deleted in the Enstore DB was found in pnfs, give the filename instead of the pnfsid. Give better error message when permissions prevent successful findinf of the file. Add a condition to report orphaned files correctly. Some were being reported as "missing layer 1". When the caller specifies path_type (fs or non-fs), honor it when looping over mount points found in /etc/mtab. ========== alarm.py ==================================================================================== Removed most (but not quite all) traces of string exceptions. ========== enstore_pg.py ==================================================================================== fix the issue having to do with particular order of columns in "select acc_daily_summary.py accounting_client.py accounting_client.py.~1.21.~ accounting_client.pyc accounting.py accounting.py.~1.16.~ accounting.pyc accounting_query.py accounting_query.pyc accounting_server.py accounting_server.py.~1.29.~ accounting_server.pyc aci.py alarm_client.py alarm_client.pyc alarm.py alarm.py.~1.51.~ alarm.pyc alarm_server.py alarm_server.py.~1.53.~ alarm_server.pyc aml2_dummy.py aml2_log.py aml2.py atomic.py atomic.pyc backup_backup.py backup_client.py backup_client.pyc backup.py backup.pyc bfid_db.py blah.ps bytes_per_day_plotter_module.py bytes_per_day_plotter_module.pyc callback.py callback.py~ callback.py.~1.132.~ callback.pyc cern_wrapper.py change_loc_cookie.py change_s_i.py charset.py charset.pyc checkdb.py checkdbs.py check_pnfs_db.py cleanUDP.py cleanUDP.py.~1.35.~ cleanUDP.pyc config-filec config.o configuration_client.py configuration_client.py.~1.139.~ configuration_client.pyc configuration_server.py configuration_server.py.~1.143.~ configuration_server.pyc core.17621 cpio_odc_wrapper.py create_sg_db.py CVS db_compare.py db_dump.py db.py db_retrieve_backup.py dbs.py dcache_make_queue_plot_page.py dcache_monitor.py delete_at_exit.py delete_at_exit.py.~1.23.~ delete_at_exit.pyc discard_copy.py discipline.py discipline.pyc disk_driver.py disk_driver.pyc dispatching_worker.py dispatching_worker.py.~1.191.~ dispatching_worker.py.bak_before_merge dispatching_worker.pyc dispatching_worker.py.new drivestat2.py drivestat_client.py drivestat_client.pyc drivestat_server.py drive_utilization_plotter_module.py drive_utilization_plotter_module.pyc duplicate.py duplicate.py.~1.26.~ duplicate.pyc duplication_util.py duplication_util.pyc ecron_util.py edb.py edb.py.~1.35.~ edb.pyc e_errors.py e_errors.py.~1.101.~ e_errors.pyc ejournal.py ejournal.pyc encp_admin.py encp_admin.py.~1.1.~ encp_admin.pyc ENCPBIN encp.py encp.py~ encp.py.1.829 encp.py.~1.973.~ encp.py.bak2 encp.py.broken encp.pyc encp_rate_multi_plotter_module.py encp_rate_multi_plotter_module.pyc encp_ticket.py encp_ticket.pyc encp_user2.py encp_user2.py.~1.1.~ encp_user2.pyc encp_user.py encp_user.py.~1.1.~ encp_user.pyc encp_wrapper.py encp_wrapper.py.~1.7.~ encp_wrapper.pyc en_eval.py en_eval.pyc enmail.py enmv_admin.py enmv.py enmv_user.py enstore_admin.py enstore_alarm_cgi.py enstore_alarm_search_cgi.py ENSTORE_BIN ENSTORE_BIN_TEMP enstore_constants.py enstore_constants.py~ enstore_constants.pyc enstore_display.py enstore_display.py.~1.172.~ enstore_display.pyc enstore_display.py.test enstore_erc_functions.py enstore_erc_functions.pyc enstore_file_listing_cgi.py enstore_files.py enstore_files.py~ enstore_files.pyc enstore_functions2.py enstore_functions2.py~ enstore_functions2.py.~1.24.~ enstore_functions2.pyc enstore_functions3.py enstore_functions3.py.~1.7.~ enstore_functions3.pyc #enstore_functions.py# enstore_functions.py enstore_functions.py.~1.56.~ enstore_functions.pyc enstore_html.py enstore_html.py~ enstore_html.pyc enstore_log_file_search_cgi.py enstore_mail.py enstore_mail.pyc enstore_make_generated_page.py enstore_make_generated_page.pyc enstore_make_log_calendar.py enstore_make_plot_page.py enstore_make_plot_page.py~ enstore_make_plot_page.pyc enstore_overall_status.py enstore_pg.py enstore_pg.pyc enstore_plots.py enstore_plots.pyc enstore_plotter_framework.py enstore_plotter_framework.pyc enstore_plotter_module.py enstore_plotter_module.pyc enstore.py enstore.py~ enstore.py.~1.83.~ enstore.pyc enstore_restart.py enstore_restart.pyc enstore_saag_network.py enstore_saag_network.pyc enstore_saag.py enstore_saag.pyc enstore_show_inventory_cgi.py enstore_show_inventory_cgi.pyc enstore_show_inv_summary_cgi.py enstore_start.py enstore_start.py.~1.67.~ enstore_start.pyc enstore_status.py enstore_status.pyc enstore_stop.py enstore_stop.py.~1.32.~ enstore_stop.pyc enstore_system_html.py enstore_system_html.py~ enstore_system_html.pyc enstore_up_down.py enstore_up_down.pyc enstore_user2.py enstore_user_cgi.py enstore_user.py enstore_utils_cgi.py ensync_admin.py ensync.py ensync_user.py ENTV_BIN entv.py entv.py.~1.128.~ entv.pyc entv.py.from_sasha entv.tar esgdb.py esgdb.pyc espion.py estart.py estop.py event_relay_client.py event_relay_client.py.~1.32.~ event_relay_client.pyc event_relay_messages.py event_relay_messages.pyc event_relay.py fake_quota.py fake_quota.pyc fdbdump.py file_clerk_client.py file_clerk_client.py.~1.183.~ file_clerk_client.pyc file_clerk.py file_clerk.py.~1.286.~ file_clerk.pyc file_family_analysis_plotter_module.py file_family_analysis_plotter_module.pyc fileinfo.py #file_utils.py# file_utils.py file_utils.py.~1.14.~ file_utils.pyc find_pnfs_file.py find_pnfs_file.py.~1.11.~ find_pnfs_file.pyc flip_tab_acsls.sh flip_tab_das.sh frozen.o fs.py fs.py~ fs.pyc ftt_driver.py ftt_driver.pyc ftt.py ftt.py.~1.13.~ ftt.pyc generic_alarm.py #generic_client.py# generic_client.py generic_client.py.~1.71.~ generic_client.pyc generic_driver.py generic_driver.pyc generic_server.py generic_server.py.~1.48.~ generic_server.pyc get_all_bytes_counter.py get_cron_title.py get.py get.py.~1.137.~ get.py.2 get.pyc get_total_bytes_counter.py hello histogram.py histogram.pyc hostaddr.py hostaddr.pyc host_config.py host_config.pyc html_cmd.py~ html_cmd.pyc html_main.py idlemovers.py info_client.py info_client.py.~1.38.~ info_client.pyc info_server.py info_server.py~ info_server.py.~1.56.~ info_server.pyc inquisitor_client.py inquisitor_client.pyc inquisitor_plots.py inquisitor_plots.py~ inquisitor_plots.pyc inquisitor.py inquisitor.py~ inquisitor.py.~1.346.~ inquisitor.pyc interface.py inventory.py inventory.py~ inventory.py.~1.170.~ inventory.pyc journal.py label_tape.py library_manager_client.py library_manager_client.py.~1.163.~ library_manager_client.pyc #library_manager.py# library_manager.py library_manager.pyc library_manager.py.prod_updated library_manager.py.sahsa_new library_manager.py.sahsa_original lintit lm_list.py lm_list.pyc lm_que_length.py log_client.py log_client.py.~1.93.~ log_client.pyc log_finish_event.py log_server.py log_server.py.~1.78.~ log_start_event.py log_trans_fail.py log_trans_fail.pyc m2.py M_abc.o M__abcoll.o Makefile Makefile.~1.66.~ make_ingest_rates_html_page.py make_migrated_as_duplicate.py make_original_as_duplicate.py makeplot.py make_pyc_files.py manage_queue.py manage_queue.pyc match_syslog.py M_base64.o M_BaseHTTPServer.o M_bdb.o M_bisect.o M_cmd.o M_codecs.o M_collections.o M_copy.o M_copy_reg.o M_ctypes___endian.o M_ctypes__macholib__dyld.o M_ctypes__macholib__dylib.o M_ctypes__macholib__framework.o M_ctypes__macholib.o M_ctypes.o M_ctypes__util.o M_difflib.o M_dis.o M_distutils__dep_util.o M_distutils__errors.o M_distutils__log.o M_distutils.o M_distutils__spawn.o M_distutils__sysconfig.o M_distutils__text_file.o M_distutils__util.o M_doctest.o M_dummy_threading.o M_dummy_thread.o media_changer_client.py media_changer_client.py.~1.106.~ media_changer_client.pyc media_changer.py media_changer.py~ media_changer.py.~1.281.~ media_changer.pyc media_changer_test.py media_changer_test.py~ media_changer_test.pyc M_email__base64mime.o M_email__charset.o M_email__encoders.o M_email__errors.o M_email__feedparser.o M_email__message.o M_email__mime.o M_email.o M_email___parseaddr.o M_email__parser.o M_email__quoprimime.o M_email__utils.o M_encodings__aliases.o M_encodings.o M_FixTk.o M_fnmatch.o M_formatter.o M_ftplib.o M_functools.o M___future__.o M_genericpath.o M_getopt.o M_getpass.o M_gettext.o M_glob.o M_heapq.o M_httplib.o migrate.py migrate.py~ migrate.py.~1.202.~ migrate.pyc migration_scope.py migration_summary_plotter_module.py migration_summary_plotter_module.py~ migration_summary_plotter_module.py.~1.15.~ migration_summary_plotter_module.pyc M_inspect.o M_keyword.o M_linecache.o M_locale.o M_macurl2path.o M___main__.o M_mimetools.o M_mimetypes.o M_ntpath.o M_nturl2path.o module_trace.py module_trace.py.~1.2.~ monitor_client.py monitor_client.py.~1.91.~ monitor_client.pyc monitored_server.py monitored_server.pyc monitor_server.py M_opcode.o M_optparse.o M_os2emxpath.o M_os.o mounts_plot.py mounts_plotter_module.py mounts_plotter_module.pyc movcmd_mc.py mover_client.py mover_client.pyc mover_constants.py mover_constants.pyc mover-nanny.py mover.py mover.py.~1.1109.~ mover.pyc mover_summary_plotter_module.py M_pdb.o M_pickle.o M_pkgutil.o M_posixpath.o M_pprint.o mpq.py mpq.pyc M_py_compile.o M_pydoc.o M_pydoc_topics.o M_quopri.o M_random.o M_re.o M_repr.o M_rfc822.o M_shlex.o M_site.o M_socket.o M_SocketServer.o M_sre_compile.o M_sre_constants.o M_sre_parse.o M_ssl.o M_stat.o M_StringIO.o M_string.o M_struct.o M_subprocess.o M_tempfile.o M_textwrap.o M__threading_local.o M_threading.o M_Tkconstants.o M_Tkinter.o M_tokenize.o M_token.o M_traceback.o M_tty.o M_types.o multiple_interface.py multiple_interface.pyc M_unittest.o M_urllib.o M_urlparse.o M_UserDict.o M_uu.o M_warnings.o M_webbrowser.o net_directory.py net_driver.py net_driver.pyc null_driver.py null_driver.pyc null_wrapper.py null_wrapper.pyc on-call.py operation.py option.py option.py~ option.py.~1.178.~ option.pyc plotter_main.py plotter_main.py~ plotter_main.pyc plotter.py plotter.py~ plotter.pyc pnfs_agent_client.py pnfs_agent_client.py~ pnfs_agent_client.py.~1.21.~ pnfs_agent_client.pyc pnfs_agent.py pnfs_agent.py.~1.32.~ pnfs_agent.pyc pnfs_backup_plot.py pnfs_backup_plotter_module.py pnfs_backup_plotter_module.pyc pnfsidparser.py #pnfs.py# pnfs.py pnfs.py~ pnfs.py.~1.291.~ pnfs.pyc pnfs.py.saved priority_selector.py priority_selector.pyc put.py put.py.~1.4.~ put.pyc quickquota.py quota.py quota.pyc quotas_plotter_module.py quotas_plotter_module.pyc ratekeeper_client.py ratekeeper_client.pyc ratekeeper_plotter_module.py ratekeeper_plotter_module.pyc ratekeeper.pts ratekeeper.py ratekeeper.py.~1.41.~ ratekeeper.pyc ratemeter.py rate_test.py rawUDP_p.py rawUDP_p.pyc rawUDP.py rawUDP.pyc readonly_volumes.py readonly_volumes.py~ readonly_volumes.pyc recent_file_listing.py recent_file_listing.py.~1.9.~ recent_file_listing.pyc report_volume_quotas.py restoredb.py rm_volmap.py run1.py safe_dict.py scanfiles.py scanfiles.py~ scanfiles.py.~1.149.~ scanfiles.pyc scanfiles.py.SAVED send_alive.py set_lm_noread.py setpath.py setpath.pyc sg_db.py show_file_cgi.py show_volume_cgi.py sljdfsf slots_usage_plotter_module.py slots_usage_plotter_module.pyc stethoscope.py strace_out string_driver.py string_driver.pyc summary_bpd_plotter_module.py summary_burn_rate_plotter_module.py summary_burn_rate_plotter_module.py~ summary_burn_rate_plotter_module.pyc summary_plots.py swap_original_and_copy.py swap_original_and_copy.py.~1.2.~ swap_original_and_copy.pyc tab_flipping_exemption.py tab_flipping_nanny.py take_out_acsls.sh take_out_das.sh tapes_burn_rate_plotter_module.py tapes_burn_rate_plotter_module.py~ tapes_burn_rate_plotter_module.pyc time_fifo.py timeofday.py timeofday.pyc Trace.py Trace.py.~1.81.~ Trace.pyc udp_client.py udp_client.py.~1.170.~ udp_client.pyc udp_cl.py udp_common.py udp_common.py.~1.13.~ udp_common.pyc udp_load_test.py udp_load_test.py.~1.1.~ udp_load_test.pyc #udp_server.py# udp_server.py udp_server.py.~1.58.~ udp_server.pyc udp_server_test_15118 udp_server_test_16059 udp_srvr.py vdbdump.py volume_assert.py volume_assert.py.~1.25.~ volume_assert.pyc volume_assert_wrapper.py volume_assert_wrapper.pyc volume_clerk_client.py volume_clerk_client.py.~1.272.~ volume_clerk_client.pyc volume_clerk.py volume_clerk.py~ volume_clerk.py.~1.445.~ volume_clerk.pyc volume_family.py volume_family.pyc web_server.py web_server.pyc weekly_summary_report.py write_protection_alert.py www_server.py www_server.pyc YANKED yank.py from encp_xfer" and "select acc_daily_summary.py accounting_client.py accounting_client.py.~1.21.~ accounting_client.pyc accounting.py accounting.py.~1.16.~ accounting.pyc accounting_query.py accounting_query.pyc accounting_server.py accounting_server.py.~1.29.~ accounting_server.pyc aci.py alarm_client.py alarm_client.pyc alarm.py alarm.py.~1.51.~ alarm.pyc alarm_server.py alarm_server.py.~1.53.~ alarm_server.pyc aml2_dummy.py aml2_log.py aml2.py atomic.py atomic.pyc backup_backup.py backup_client.py backup_client.pyc backup.py backup.pyc bfid_db.py blah.ps bytes_per_day_plotter_module.py bytes_per_day_plotter_module.pyc callback.py callback.py~ callback.py.~1.132.~ callback.pyc cern_wrapper.py change_loc_cookie.py change_s_i.py charset.py charset.pyc checkdb.py checkdbs.py check_pnfs_db.py cleanUDP.py cleanUDP.py.~1.35.~ cleanUDP.pyc config-filec config.o configuration_client.py configuration_client.py.~1.139.~ configuration_client.pyc configuration_server.py configuration_server.py.~1.143.~ configuration_server.pyc core.17621 cpio_odc_wrapper.py create_sg_db.py CVS db_compare.py db_dump.py db.py db_retrieve_backup.py dbs.py dcache_make_queue_plot_page.py dcache_monitor.py delete_at_exit.py delete_at_exit.py.~1.23.~ delete_at_exit.pyc discard_copy.py discipline.py discipline.pyc disk_driver.py disk_driver.pyc dispatching_worker.py dispatching_worker.py.~1.191.~ dispatching_worker.py.bak_before_merge dispatching_worker.pyc dispatching_worker.py.new drivestat2.py drivestat_client.py drivestat_client.pyc drivestat_server.py drive_utilization_plotter_module.py drive_utilization_plotter_module.pyc duplicate.py duplicate.py.~1.26.~ duplicate.pyc duplication_util.py duplication_util.pyc ecron_util.py edb.py edb.py.~1.35.~ edb.pyc e_errors.py e_errors.py.~1.101.~ e_errors.pyc ejournal.py ejournal.pyc encp_admin.py encp_admin.py.~1.1.~ encp_admin.pyc ENCPBIN encp.py encp.py~ encp.py.1.829 encp.py.~1.973.~ encp.py.bak2 encp.py.broken encp.pyc encp_rate_multi_plotter_module.py encp_rate_multi_plotter_module.pyc encp_ticket.py encp_ticket.pyc encp_user2.py encp_user2.py.~1.1.~ encp_user2.pyc encp_user.py encp_user.py.~1.1.~ encp_user.pyc encp_wrapper.py encp_wrapper.py.~1.7.~ encp_wrapper.pyc en_eval.py en_eval.pyc enmail.py enmv_admin.py enmv.py enmv_user.py enstore_admin.py enstore_alarm_cgi.py enstore_alarm_search_cgi.py ENSTORE_BIN ENSTORE_BIN_TEMP enstore_constants.py enstore_constants.py~ enstore_constants.pyc enstore_display.py enstore_display.py.~1.172.~ enstore_display.pyc enstore_display.py.test enstore_erc_functions.py enstore_erc_functions.pyc enstore_file_listing_cgi.py enstore_files.py enstore_files.py~ enstore_files.pyc enstore_functions2.py enstore_functions2.py~ enstore_functions2.py.~1.24.~ enstore_functions2.pyc enstore_functions3.py enstore_functions3.py.~1.7.~ enstore_functions3.pyc #enstore_functions.py# enstore_functions.py enstore_functions.py.~1.56.~ enstore_functions.pyc enstore_html.py enstore_html.py~ enstore_html.pyc enstore_log_file_search_cgi.py enstore_mail.py enstore_mail.pyc enstore_make_generated_page.py enstore_make_generated_page.pyc enstore_make_log_calendar.py enstore_make_plot_page.py enstore_make_plot_page.py~ enstore_make_plot_page.pyc enstore_overall_status.py enstore_pg.py enstore_pg.pyc enstore_plots.py enstore_plots.pyc enstore_plotter_framework.py enstore_plotter_framework.pyc enstore_plotter_module.py enstore_plotter_module.pyc enstore.py enstore.py~ enstore.py.~1.83.~ enstore.pyc enstore_restart.py enstore_restart.pyc enstore_saag_network.py enstore_saag_network.pyc enstore_saag.py enstore_saag.pyc enstore_show_inventory_cgi.py enstore_show_inventory_cgi.pyc enstore_show_inv_summary_cgi.py enstore_start.py enstore_start.py.~1.67.~ enstore_start.pyc enstore_status.py enstore_status.pyc enstore_stop.py enstore_stop.py.~1.32.~ enstore_stop.pyc enstore_system_html.py enstore_system_html.py~ enstore_system_html.pyc enstore_up_down.py enstore_up_down.pyc enstore_user2.py enstore_user_cgi.py enstore_user.py enstore_utils_cgi.py ensync_admin.py ensync.py ensync_user.py ENTV_BIN entv.py entv.py.~1.128.~ entv.pyc entv.py.from_sasha entv.tar esgdb.py esgdb.pyc espion.py estart.py estop.py event_relay_client.py event_relay_client.py.~1.32.~ event_relay_client.pyc event_relay_messages.py event_relay_messages.pyc event_relay.py fake_quota.py fake_quota.pyc fdbdump.py file_clerk_client.py file_clerk_client.py.~1.183.~ file_clerk_client.pyc file_clerk.py file_clerk.py.~1.286.~ file_clerk.pyc file_family_analysis_plotter_module.py file_family_analysis_plotter_module.pyc fileinfo.py #file_utils.py# file_utils.py file_utils.py.~1.14.~ file_utils.pyc find_pnfs_file.py find_pnfs_file.py.~1.11.~ find_pnfs_file.pyc flip_tab_acsls.sh flip_tab_das.sh frozen.o fs.py fs.py~ fs.pyc ftt_driver.py ftt_driver.pyc ftt.py ftt.py.~1.13.~ ftt.pyc generic_alarm.py #generic_client.py# generic_client.py generic_client.py.~1.71.~ generic_client.pyc generic_driver.py generic_driver.pyc generic_server.py generic_server.py.~1.48.~ generic_server.pyc get_all_bytes_counter.py get_cron_title.py get.py get.py.~1.137.~ get.py.2 get.pyc get_total_bytes_counter.py hello histogram.py histogram.pyc hostaddr.py hostaddr.pyc host_config.py host_config.pyc html_cmd.py~ html_cmd.pyc html_main.py idlemovers.py info_client.py info_client.py.~1.38.~ info_client.pyc info_server.py info_server.py~ info_server.py.~1.56.~ info_server.pyc inquisitor_client.py inquisitor_client.pyc inquisitor_plots.py inquisitor_plots.py~ inquisitor_plots.pyc inquisitor.py inquisitor.py~ inquisitor.py.~1.346.~ inquisitor.pyc interface.py inventory.py inventory.py~ inventory.py.~1.170.~ inventory.pyc journal.py label_tape.py library_manager_client.py library_manager_client.py.~1.163.~ library_manager_client.pyc #library_manager.py# library_manager.py library_manager.pyc library_manager.py.prod_updated library_manager.py.sahsa_new library_manager.py.sahsa_original lintit lm_list.py lm_list.pyc lm_que_length.py log_client.py log_client.py.~1.93.~ log_client.pyc log_finish_event.py log_server.py log_server.py.~1.78.~ log_start_event.py log_trans_fail.py log_trans_fail.pyc m2.py M_abc.o M__abcoll.o Makefile Makefile.~1.66.~ make_ingest_rates_html_page.py make_migrated_as_duplicate.py make_original_as_duplicate.py makeplot.py make_pyc_files.py manage_queue.py manage_queue.pyc match_syslog.py M_base64.o M_BaseHTTPServer.o M_bdb.o M_bisect.o M_cmd.o M_codecs.o M_collections.o M_copy.o M_copy_reg.o M_ctypes___endian.o M_ctypes__macholib__dyld.o M_ctypes__macholib__dylib.o M_ctypes__macholib__framework.o M_ctypes__macholib.o M_ctypes.o M_ctypes__util.o M_difflib.o M_dis.o M_distutils__dep_util.o M_distutils__errors.o M_distutils__log.o M_distutils.o M_distutils__spawn.o M_distutils__sysconfig.o M_distutils__text_file.o M_distutils__util.o M_doctest.o M_dummy_threading.o M_dummy_thread.o media_changer_client.py media_changer_client.py.~1.106.~ media_changer_client.pyc media_changer.py media_changer.py~ media_changer.py.~1.281.~ media_changer.pyc media_changer_test.py media_changer_test.py~ media_changer_test.pyc M_email__base64mime.o M_email__charset.o M_email__encoders.o M_email__errors.o M_email__feedparser.o M_email__message.o M_email__mime.o M_email.o M_email___parseaddr.o M_email__parser.o M_email__quoprimime.o M_email__utils.o M_encodings__aliases.o M_encodings.o M_FixTk.o M_fnmatch.o M_formatter.o M_ftplib.o M_functools.o M___future__.o M_genericpath.o M_getopt.o M_getpass.o M_gettext.o M_glob.o M_heapq.o M_httplib.o migrate.py migrate.py~ migrate.py.~1.202.~ migrate.pyc migration_scope.py migration_summary_plotter_module.py migration_summary_plotter_module.py~ migration_summary_plotter_module.py.~1.15.~ migration_summary_plotter_module.pyc M_inspect.o M_keyword.o M_linecache.o M_locale.o M_macurl2path.o M___main__.o M_mimetools.o M_mimetypes.o M_ntpath.o M_nturl2path.o module_trace.py module_trace.py.~1.2.~ monitor_client.py monitor_client.py.~1.91.~ monitor_client.pyc monitored_server.py monitored_server.pyc monitor_server.py M_opcode.o M_optparse.o M_os2emxpath.o M_os.o mounts_plot.py mounts_plotter_module.py mounts_plotter_module.pyc movcmd_mc.py mover_client.py mover_client.pyc mover_constants.py mover_constants.pyc mover-nanny.py mover.py mover.py.~1.1109.~ mover.pyc mover_summary_plotter_module.py M_pdb.o M_pickle.o M_pkgutil.o M_posixpath.o M_pprint.o mpq.py mpq.pyc M_py_compile.o M_pydoc.o M_pydoc_topics.o M_quopri.o M_random.o M_re.o M_repr.o M_rfc822.o M_shlex.o M_site.o M_socket.o M_SocketServer.o M_sre_compile.o M_sre_constants.o M_sre_parse.o M_ssl.o M_stat.o M_StringIO.o M_string.o M_struct.o M_subprocess.o M_tempfile.o M_textwrap.o M__threading_local.o M_threading.o M_Tkconstants.o M_Tkinter.o M_tokenize.o M_token.o M_traceback.o M_tty.o M_types.o multiple_interface.py multiple_interface.pyc M_unittest.o M_urllib.o M_urlparse.o M_UserDict.o M_uu.o M_warnings.o M_webbrowser.o net_directory.py net_driver.py net_driver.pyc null_driver.py null_driver.pyc null_wrapper.py null_wrapper.pyc on-call.py operation.py option.py option.py~ option.py.~1.178.~ option.pyc plotter_main.py plotter_main.py~ plotter_main.pyc plotter.py plotter.py~ plotter.pyc pnfs_agent_client.py pnfs_agent_client.py~ pnfs_agent_client.py.~1.21.~ pnfs_agent_client.pyc pnfs_agent.py pnfs_agent.py.~1.32.~ pnfs_agent.pyc pnfs_backup_plot.py pnfs_backup_plotter_module.py pnfs_backup_plotter_module.pyc pnfsidparser.py #pnfs.py# pnfs.py pnfs.py~ pnfs.py.~1.291.~ pnfs.pyc pnfs.py.saved priority_selector.py priority_selector.pyc put.py put.py.~1.4.~ put.pyc quickquota.py quota.py quota.pyc quotas_plotter_module.py quotas_plotter_module.pyc ratekeeper_client.py ratekeeper_client.pyc ratekeeper_plotter_module.py ratekeeper_plotter_module.pyc ratekeeper.pts ratekeeper.py ratekeeper.py.~1.41.~ ratekeeper.pyc ratemeter.py rate_test.py rawUDP_p.py rawUDP_p.pyc rawUDP.py rawUDP.pyc readonly_volumes.py readonly_volumes.py~ readonly_volumes.pyc recent_file_listing.py recent_file_listing.py.~1.9.~ recent_file_listing.pyc report_volume_quotas.py restoredb.py rm_volmap.py run1.py safe_dict.py scanfiles.py scanfiles.py~ scanfiles.py.~1.149.~ scanfiles.pyc scanfiles.py.SAVED send_alive.py set_lm_noread.py setpath.py setpath.pyc sg_db.py show_file_cgi.py show_volume_cgi.py sljdfsf slots_usage_plotter_module.py slots_usage_plotter_module.pyc stethoscope.py strace_out string_driver.py string_driver.pyc summary_bpd_plotter_module.py summary_burn_rate_plotter_module.py summary_burn_rate_plotter_module.py~ summary_burn_rate_plotter_module.pyc summary_plots.py swap_original_and_copy.py swap_original_and_copy.py.~1.2.~ swap_original_and_copy.pyc tab_flipping_exemption.py tab_flipping_nanny.py take_out_acsls.sh take_out_das.sh tapes_burn_rate_plotter_module.py tapes_burn_rate_plotter_module.py~ tapes_burn_rate_plotter_module.pyc time_fifo.py timeofday.py timeofday.pyc Trace.py Trace.py.~1.81.~ Trace.pyc udp_client.py udp_client.py.~1.170.~ udp_client.pyc udp_cl.py udp_common.py udp_common.py.~1.13.~ udp_common.pyc udp_load_test.py udp_load_test.py.~1.1.~ udp_load_test.pyc #udp_server.py# udp_server.py udp_server.py.~1.58.~ udp_server.pyc udp_server_test_15118 udp_server_test_16059 udp_srvr.py vdbdump.py volume_assert.py volume_assert.py.~1.25.~ volume_assert.pyc volume_assert_wrapper.py volume_assert_wrapper.pyc volume_clerk_client.py volume_clerk_client.py.~1.272.~ volume_clerk_client.pyc volume_clerk.py volume_clerk.py~ volume_clerk.py.~1.445.~ volume_clerk.pyc volume_family.py volume_family.pyc web_server.py web_server.pyc weekly_summary_report.py write_protection_alert.py www_server.py www_server.pyc YANKED yank.py from encp_error" by using dictionaries instead of bare data. replace "select *" with "select date,node ...." skipping library cuz inquisitor code chokes on extra columns. NB: inquisitor needs fixing as well ========== ftt.py ==================================================================================== beginning with swig 1.3.21 the list is returned instead of tuple this workaround fixes the incompatibility issue ========== Trace.py ==================================================================================== The locks have been commented out. Added locks to prevent one thread from starting to print a message, get interupted by another thread printing a differnet message and then the rest of the first message is printed. Only one should happend at a time now. Code review followup changes. bigzilla reference ticket 378 Do not truncate message in trace(), when it is sent to stdout. Output the time in default_log_func(). Bugzilla #374. Removed most (but not quite all) traces of string exceptions. Added log_stack_trace() that sends the current stack trace to the DEBUGLOG. If the default severity is overridden, it is possible to sent it to the normal log file instead. If KeyboardInterrupt or SystemExit is caught, re-raise them. ========== pnfs_agent_client.py ==================================================================================== Added some ifs to detect errors in some functions that were missing them. Added some Trace.log() calls to get_stat and get_pnfsstat() to furthur understand a reported traceback. Both changes are related to bugzilla #469. Use the default timeout and retry values while waiting for responses from the pnfs_agent. ========== alarm_server.py ==================================================================================== revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. ========== checkdb.py ==================================================================================== change call to file_utils.rmdir to account for new signature (now this function is void and just throws exception if something goes wrong) added header 1) added function rmdir to file_utils that effectively executes "rm -rf path" 2) modified checkdb.py so that it cleans up after itself not leaving old stuff behind. This change makes it immune to postgresql version changes ========== enstore_utils_cgi.py ==================================================================================== remove dependency on regex and regex_syntax ========== configuration_server.py ==================================================================================== Changes made for bugzilla #367 comment #16. New function __get_config_value(). _get_domains() handles if the 'domains' section of the configuration is not a dictionary. Some comments are corrected to match the code. revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. Set default do_copies to 1. This is for copy.copy() to be used instead of a copy.deepcopy() or a direct reference to configdict. The previous patch introduced a bug where dump2() would still work, but dump() failed with "Message too long". Improved the speed of some functions by modifying how they tried to access the configdict. Added some locks for thread safety. Added support to adjust the internal copy level of the configdict on the fly using the "enstore conf --copy 0|1|2" command. Work done to improve the response times of lookup(), dump() and dump2(). Some caching is involved of these responses. For dump() and dump2() don't supply a thread name. Allow the total maximum threads per Enstore processes imposed by dispatching_worker. This is to hopefully address the issues in bugzilla 403. The main thing for speeding it up is that _get_system_name() caches its return value. Other thread syncronization changes have also been done, since there have been errors. These patches are related to getting the configuration server switched to using dispatching_worker's new send_reply_with_long_answer() function. Backward compatiblity with old clients and servers is still there. Use the new dispatching_worker.send_reply() function. ========== media_changer.py ==================================================================================== Add in a select() of the pipe from the rsh process in STK_MediaLoader.timed_command(). This should allow for timing out of the process if the rsh doesn't close the its side of the pipe. Previously, the process would block in read(), and if the timeout expired, the process would stay blocked if the rsh didn't close the pipe. (Bugzilla #455.) revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. Return the location information for STK robots for the --show-volume command. For STK media_changer in listSlots(), when converting strings to integers, only one conversion was catching TypeError and ValueError. Most were just catching TypeError. If a value error occured (when instead of the expected integer value the MC got something like "Volume VOG123 ejected" instead) the request was not getting removed from the work_query_list and eventually would result in a denial-of-service state once the queue was full of old/failed requests. Define --eject and --insert to be no-ops, with an error, for STK, IBM and MTX media_changers. Added a check in the STK listSlots() function to prevent the reporting of robots that are not configured for use by this Enstore system. Missed saving the insertion of a comment before the previous commit. Fixed a bug where the max_work value isn't honored. It was caused "load" and "unload" in a list of work types instead of "mount" and "dismount", which are the correct values. Also, added more DEBUGLOG messages. Scrubbed the code. Merge in the media_changer_test.py modifications for forcing errors with the buttons. This is only allowed if 'test' is set to true in the configuration for the Manual_MediaLoader. ========== enstore_stop.py ==================================================================================== Fixed enstore_start.this_host() to report all hostname and IP addresses for the node. The use_alias variables are no longer needed since all hostname and IP information is used every time. Converted udp_client.UDPError and callback.TCPError to be replaced by e_errors.EnstoreError. Removed most (but not quite all) traces of string exceptions. Catch udp_client.UDPError before errno.errorcode[errno.ETIMEDOUT]. ========== enstore_functions2.py ==================================================================================== Use socket.getfqdn() instead of socket.gethostbyname() in fullpath() and fullpath2(). Added is_migration_file_family(), is_duplication_file_family() and is_migration_related_file_family() functions. ========== info_server.py ==================================================================================== remove these warnings: nfo_server.py:272: Local variable (msg) not used info_server.py:322: Local variable (msg) not used info_server.py:390: Local variable (msg) not used The commands --find-original, --find-the-original, --find-duplicates, --find-all-copies and --find-copies commands are now available to users through the info server. Admins can now use them using the "enstore file" or "enstore info" commands (though the "enstore file" variants points to the info server). In addition to this the enstore info --show-copies now works like it was supposed to. correct previous patch fix for: Line 371 assumes that there will only be one match possible for each location cookie, external label combination. This is not true. I offer one random example, there are many more: bug fix: Mike reported that enstore info --file TEST23:0000_000000000_0000600 stopped working. This patch fixes it. revert recent changes Various patches to bring the servers into alignment with dispatching_worker.py 1.189 and upd_server.py 1.56. These changes have these Enstore servers using dispatching_worker.send_reply_with_long_answer() instead of their local copy of this function. Also, the commented out duplicated code has been removed too. remove print file_clerk - minor printout change info_server, info_client: noticed that command like "enstore info --show-file CDMS115209320400002" do not work Fixed. When a new configuration file message is recieved, this patched reinit() will allow for the sending of the heartbeat messages to work again. This is a very large patch. 1) The clients now use send_with_long_answer() to generically handle requests where the response is large. Servers, now have reply_to_caller_with_long_answer() to handle the same thing on their side. These functions use an "active" TCP protocol. 2) Better error checking for erroneous values. For example, if a command required a bfid but is supplied a pnfsid the user would get a cryptic value (at best) or no information from the DB query. Now, they would get something like "value is not a bfid". This is done with the extract_value_from_ticket() family of functions. Corrected some log messages related to the handling on new config messages. ========== Interfaces.c ==================================================================================== Allow the __arp() function to build on MacOS X. (Bugzilla #503) Added the ability to report MAC addresses currently located in the arp table. ========== EXfer.c ==================================================================================== Patch 1.139 introduced a resource leak. At that time, static thread objects were made into dynamic thread objects to support the EXfer module being used in multiple migration threads. This patch adds calling the appropriate destroy functions. This is for bugzilla #396. Modified the way that the network and disk rates are calculated for threaded transfers. Previously, disk drive rates were based on the wall clock time for regular files; getrusage() times plus time in select() and read()/write() were used for sockets. Now all file types are the wall clock time minus the time waiting for the other thread. Modified do_select() to use the right timeout after and EINTR. Also, change some signal handling in thread_collect(). Handle EINTR errors from select() by recalling select(). Allow for do_read_write() or do_read_write_threaded() to be able to be called from multiple threads. This involved additional locks and signal handling. The use of global variables has, almost all, been eliminated. This is the first step in being able to run do_read_write() or do_read_write_threaded() from multiple threads. ========== enstore.table ==================================================================================== Added $ENSTORE_DIR/tools to $PATH. (Bugzilla 495.) Change the ENSTORE_CONFIG_HOST values from *srv2.fnal.gov to conf-*.fnal.gov. add $ENSTORE_DIR to PYTHONPATH, so package DBUtils can be exported For the few who use enstore with UPS, make migrate and duplicate symbolic links from $ENSTORE_DIR/bin/. added $ENSTORE_DIR/modules/PyGreSQL to PYTHONPATH