-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:45 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg323-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg323-server oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:47 (1709786447) start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m02s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800ab2c6800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:45.000000000 -0500 Modify: 2024-03-06 23:41:45.000000000 -0500 Change: 2024-03-06 23:41:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server unloading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local modules unloaded. oleg323-server: oleg323-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg323-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg323-server: oleg323-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg323-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg323-server: oleg323-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg323-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server PASS 35b (137s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:07 (1709786587) start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server 000000 2 000008 oleg323-server: 8+0 records in oleg323-server: 8+0 records out oleg323-server: 4096 bytes (4.1 kB) copied, 0.000715989 s, 5.7 MB/s start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server unloading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 PASS 38 (96s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg323-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg323-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg323-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 oleg323-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg323-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:46 (1709786686) Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg323-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg323-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg323-server umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server unloading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local modules unloaded. oleg323-server: oleg323-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg323-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg323-server: oleg323-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg323-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg323-server: oleg323-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg323-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg323-server: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg323-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server PASS 51 (155s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:24 (1709786844) start mds service on oleg323-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 341 (time 1709786872.60 total 10.16 last 33.56) - open/close 807 (time 1709786882.61 total 20.17 last 46.56) - open/close 1137 (time 1709786893.09 total 30.66 last 31.46) - open/close 1333 (time 1709786903.10 total 40.66 last 19.59) - open/close 1589 (time 1709786913.79 total 51.35 last 23.94) - open/close 1732 (time 1709786924.32 total 61.88 last 13.59) - open/close 1966 (time 1709786934.60 total 72.17 last 22.75) - open/close 2155 (time 1709786947.55 total 85.11 last 14.60) - open/close 2317 (time 1709786958.30 total 95.86 last 15.07) - open/close 2531 (time 1709786968.65 total 106.21 last 20.67) - open/close 2713 (time 1709786979.30 total 116.87 last 17.09) - open/close 2908 (time 1709786989.31 total 126.87 last 19.49) - open/close 3132 (time 1709786999.31 total 136.88 last 22.38) - open/close 3336 (time 1709787011.27 total 148.83 last 17.06) - open/close 3595 (time 1709787022.20 total 159.77 last 23.69) - open/close 3825 (time 1709787032.21 total 169.78 last 22.98) - open/close 3940 (time 1709787042.22 total 179.78 last 11.49) - open/close 4150 (time 1709787053.13 total 190.69 last 19.25) - open/close 4397 (time 1709787063.56 total 201.12 last 23.67) - open/close 4580 (time 1709787073.57 total 211.14 last 18.28) - open/close 4729 (time 1709787084.34 total 221.91 last 13.84) - open/close 4930 (time 1709787094.35 total 231.91 last 20.08) - open/close 5076 (time 1709787104.97 total 242.53 last 13.75) - open/close 5318 (time 1709787114.97 total 252.54 last 24.19) - open/close 5523 (time 1709787125.43 total 262.99 last 19.61) - open/close 5770 (time 1709787135.91 total 273.47 last 23.56) - open/close 6029 (time 1709787145.91 total 283.48 last 25.89) - open/close 6186 (time 1709787157.80 total 295.37 last 13.20) - open/close 6349 (time 1709787167.80 total 305.37 last 16.30) - open/close 6479 (time 1709787177.83 total 315.40 last 12.96) - open/close 6568 (time 1709787187.87 total 325.44 last 8.86) - open/close 6708 (time 1709787197.97 total 335.53 last 13.87) - open/close 6869 (time 1709787209.20 total 346.76 last 14.34) - open/close 7048 (time 1709787219.66 total 357.22 last 17.11) - open/close 7165 (time 1709787229.73 total 367.29 last 11.62) - open/close 7333 (time 1709787240.45 total 378.01 last 15.67) - open/close 7564 (time 1709787250.45 total 388.01 last 23.09) - open/close 7714 (time 1709787260.67 total 398.23 last 14.68) - open/close 7895 (time 1709787270.99 total 408.55 last 17.54) - open/close 8023 (time 1709787280.99 total 418.55 last 12.80) - open/close 8177 (time 1709787291.47 total 429.04 last 14.69) - open/close 8419 (time 1709787301.68 total 439.25 last 23.70) - open/close 8696 (time 1709787311.69 total 449.26 last 27.68) - open/close 8817 (time 1709787322.66 total 460.22 last 11.03) - open/close 8993 (time 1709787333.20 total 470.77 last 16.69) - open/close 9169 (time 1709787344.18 total 481.74 last 16.04) - open/close 9376 (time 1709787354.25 total 491.81 last 20.56) - open/close 9528 (time 1709787364.75 total 502.31 last 14.47) - open/close 9681 (time 1709787375.61 total 513.18 last 14.09) - open/close 9932 (time 1709787385.61 total 523.18 last 25.10) - open/close 10000 (time 1709787388.57 total 526.14 last 22.97) - open/close 10131 (time 1709787399.10 total 536.66 last 12.45) - open/close 10314 (time 1709787409.86 total 547.42 last 17.00) - open/close 10505 (time 1709787426.37 total 563.94 last 11.57) - open/close 10929 (time 1709787436.38 total 573.94 last 42.37) - open/close 11385 (time 1709787446.39 total 583.96 last 45.54) - open/close 11864 (time 1709787456.42 total 593.98 last 47.79) - open/close 12244 (time 1709787466.43 total 603.99 last 37.95) - open/close 12615 (time 1709787476.46 total 614.02 last 36.99) - open/close 13017 (time 1709787486.48 total 624.05 last 40.11) - open/close 13384 (time 1709787496.49 total 634.06 last 36.67) - open/close 13754 (time 1709787506.50 total 644.07 last 36.95) - open/close 14141 (time 1709787516.51 total 654.07 last 38.68) - open/close 14444 (time 1709787526.54 total 664.11 last 30.20) - open/close 14816 (time 1709787536.59 total 674.15 last 37.04) - open/close 15170 (time 1709787546.61 total 684.17 last 35.33) - open/close 15482 (time 1709787556.62 total 694.19 last 31.15) - open/close 15791 (time 1709787566.63 total 704.19 last 30.88) - open/close 16090 (time 1709787576.63 total 714.20 last 29.88) - open/close 16410 (time 1709787586.67 total 724.23 last 31.88) - open/close 16693 (time 1709787596.69 total 734.25 last 28.26) - open/close 16973 (time 1709787606.71 total 744.27 last 27.94) - open/close 17263 (time 1709787616.74 total 754.30 last 28.92) - open/close 17552 (time 1709787626.75 total 764.31 last 28.86) - open/close 17886 (time 1709787636.77 total 774.34 last 33.31) - open/close 18165 (time 1709787646.82 total 784.39 last 27.77) - open/close 18481 (time 1709787656.85 total 794.41 last 31.52) - open/close 18785 (time 1709787666.87 total 804.43 last 30.33) - open/close 19053 (time 1709787676.92 total 814.48 last 26.67) - open/close 19316 (time 1709787686.94 total 824.50 last 26.26) - open/close 19566 (time 1709787696.96 total 834.52 last 24.94) - open/close 19808 (time 1709787706.96 total 844.53 last 24.19) - open/close 20000 (time 1709787715.11 total 852.68 last 23.56) - open/close 20290 (time 1709787725.12 total 862.69 last 28.97) - open/close 20561 (time 1709787735.14 total 872.70 last 27.06) - open/close 20832 (time 1709787745.14 total 882.70 last 27.09) - open/close 21066 (time 1709787755.15 total 892.71 last 23.38) - open/close 21341 (time 1709787765.17 total 902.73 last 27.44) - open/close 21625 (time 1709787775.25 total 912.81 last 28.17) - open/close 21871 (time 1709787785.28 total 922.84 last 24.53) - open/close 21935 (time 1709787798.74 total 936.30 last 4.75) - open/close 22229 (time 1709787808.74 total 946.30 last 29.40) - open/close 22498 (time 1709787818.77 total 956.34 last 26.81) - open/close 22774 (time 1709787828.79 total 966.35 last 27.56) - open/close 23068 (time 1709787838.80 total 976.36 last 29.36) - open/close 23335 (time 1709787848.80 total 986.37 last 26.70) - open/close 23625 (time 1709787858.84 total 996.40 last 28.90) - open/close 23906 (time 1709787868.86 total 1006.43 last 28.03) - open/close 24148 (time 1709787878.89 total 1016.45 last 24.14) - open/close 24419 (time 1709787888.91 total 1026.47 last 27.05) - open/close 24691 (time 1709787898.95 total 1036.51 last 27.09) - open/close 24974 (time 1709787908.96 total 1046.52 last 28.28) - open/close 25240 (time 1709787918.99 total 1056.55 last 26.51) - open/close 25487 (time 1709787929.02 total 1066.58 last 24.63) - open/close 25733 (time 1709787939.04 total 1076.60 last 24.56) - open/close 25984 (time 1709787949.05 total 1086.62 last 25.06) - open/close 26240 (time 1709787959.06 total 1096.62 last 25.60) - open/close 26504 (time 1709787969.07 total 1106.63 last 26.36) - open/close 26769 (time 1709787979.09 total 1116.65 last 26.45) - open/close 27046 (time 1709787989.10 total 1126.67 last 27.66) - open/close 27326 (time 1709787999.12 total 1136.68 last 27.96) - open/close 27598 (time 1709788009.13 total 1146.70 last 27.16) - open/close 27875 (time 1709788019.18 total 1156.74 last 27.58) - open/close 28106 (time 1709788029.20 total 1166.77 last 23.04) - open/close 28366 (time 1709788039.21 total 1176.77 last 25.98) - open/close 28617 (time 1709788049.24 total 1186.81 last 25.01) - open/close 28865 (time 1709788059.28 total 1196.85 last 24.71) - open/close 29125 (time 1709788069.30 total 1206.86 last 25.96) - open/close 29430 (time 1709788079.31 total 1216.87 last 30.47) - open/close 29765 (time 1709788089.32 total 1226.88 last 33.47) - open/close 30000 (time 1709788096.21 total 1233.77 last 34.12) - open/close 30329 (time 1709788106.21 total 1243.78 last 32.88) - open/close 30659 (time 1709788116.23 total 1253.79 last 32.95) - open/close 31002 (time 1709788126.26 total 1263.82 last 34.19) - open/close 31341 (time 1709788136.27 total 1273.83 last 33.87) - open/close 31680 (time 1709788146.30 total 1283.86 last 33.81) - open/close 31989 (time 1709788156.31 total 1293.87 last 30.87) - open/close 32300 (time 1709788166.31 total 1303.87 last 31.08) - open/close 32615 (time 1709788176.33 total 1313.89 last 31.44) - open/close 32928 (time 1709788186.33 total 1323.90 last 31.29) - open/close 33221 (time 1709788196.34 total 1333.91 last 29.27) - open/close 33502 (time 1709788206.36 total 1343.92 last 28.06) - open/close 33792 (time 1709788216.37 total 1353.93 last 28.96) - open/close 34085 (time 1709788226.41 total 1363.97 last 29.18) - open/close 34368 (time 1709788236.44 total 1374.00 last 28.23) - open/close 34650 (time 1709788246.45 total 1384.02 last 28.14) - open/close 34899 (time 1709788256.47 total 1394.03 last 24.86) - open/close 35177 (time 1709788266.50 total 1404.06 last 27.72) - open/close 35457 (time 1709788276.52 total 1414.09 last 27.93) - open/close 35722 (time 1709788286.52 total 1424.09 last 26.49) - open/close 35983 (time 1709788296.54 total 1434.11 last 26.05) - open/close 36241 (time 1709788306.57 total 1444.13 last 25.74) - open/close 36502 (time 1709788316.58 total 1454.15 last 26.06) - open/close 36771 (time 1709788326.59 total 1464.15 last 26.89) - open/close 37042 (time 1709788336.59 total 1474.15 last 27.08) - open/close 37308 (time 1709788346.61 total 1484.17 last 26.56) - open/close 37587 (time 1709788356.61 total 1494.17 last 27.89) - open/close 37848 (time 1709788366.64 total 1504.20 last 26.01) - open/close 38105 (time 1709788376.67 total 1514.23 last 25.63) - open/close 38359 (time 1709788386.71 total 1524.27 last 25.31) - open/close 38613 (time 1709788396.74 total 1534.30 last 25.32) - open/close 38869 (time 1709788406.75 total 1544.32 last 25.56) - open/close 39128 (time 1709788416.77 total 1554.33 last 25.86) - open/close 39404 (time 1709788426.79 total 1564.36 last 27.52) - open/close 39684 (time 1709788436.80 total 1574.36 last 27.98) - open/close 39949 (time 1709788446.82 total 1584.39 last 26.44) - open/close 40000 (time 1709788448.97 total 1586.54 last 23.72) - open/close 40266 (time 1709788459.00 total 1596.57 last 26.52) - open/close 40519 (time 1709788469.04 total 1606.60 last 25.21) - open/close 40795 (time 1709788479.05 total 1616.61 last 27.58) - open/close 41039 (time 1709788489.08 total 1626.64 last 24.32) - open/close 41275 (time 1709788499.11 total 1636.68 last 23.52) - open/close 41535 (time 1709788509.12 total 1646.68 last 25.99) - open/close 41801 (time 1709788519.13 total 1656.69 last 26.58) - open/close 42026 (time 1709788538.95 total 1676.51 last 11.35) - open/close 42284 (time 1709788548.97 total 1686.53 last 25.75) - open/close 42535 (time 1709788558.99 total 1696.55 last 25.05) - open/close 42796 (time 1709788569.01 total 1706.58 last 26.03) - open/close 43065 (time 1709788579.04 total 1716.60 last 26.84) - open/close 43326 (time 1709788589.07 total 1726.63 last 26.02) - open/close 43588 (time 1709788599.10 total 1736.66 last 26.13) - open/close 43856 (time 1709788609.11 total 1746.68 last 26.75) - open/close 44118 (time 1709788619.13 total 1756.70 last 26.15) - open/close 44378 (time 1709788629.14 total 1766.70 last 25.98) - open/close 44618 (time 1709788639.15 total 1776.71 last 23.99) - open/close 44829 (time 1709788649.16 total 1786.73 last 21.06) - open/close 45037 (time 1709788659.18 total 1796.75 last 20.76) - open/close 45253 (time 1709788669.19 total 1806.75 last 21.59) - open/close 45488 (time 1709788679.20 total 1816.76 last 23.47) - open/close 45709 (time 1709788689.24 total 1826.80 last 22.01) - open/close 45934 (time 1709788699.25 total 1836.81 last 22.48) - open/close 46162 (time 1709788709.26 total 1846.82 last 22.78) - open/close 46384 (time 1709788719.26 total 1856.82 last 22.19) - open/close 46616 (time 1709788729.27 total 1866.83 last 23.18) - open/close 46849 (time 1709788739.30 total 1876.86 last 23.23) - open/close 47085 (time 1709788749.34 total 1886.90 last 23.51) - open/close 47303 (time 1709788759.36 total 1896.93 last 21.75) - open/close 47507 (time 1709788769.40 total 1906.96 last 20.33) - open/close 47705 (time 1709788779.44 total 1917.00 last 19.72) - open/close 47915 (time 1709788789.47 total 1927.04 last 20.93) - open/close 48122 (time 1709788799.50 total 1937.06 last 20.65) - open/close 48333 (time 1709788809.53 total 1947.09 last 21.04) - open/close 48546 (time 1709788819.53 total 1957.10 last 21.28) - open/close 48744 (time 1709788829.54 total 1967.10 last 19.80) - open/close 48963 (time 1709788839.54 total 1977.10 last 21.90) - open/close 49178 (time 1709788849.57 total 1987.13 last 21.43) - open/close 49390 (time 1709788859.61 total 1997.17 last 21.11) - open/close 49596 (time 1709788869.65 total 2007.21 last 20.53) - open/close 49806 (time 1709788879.68 total 2017.25 last 20.92) total: 50000 open/close in 2026.19 seconds: 24.68 ops/second - unlinked 0 (time 1709788889 ; total 0 ; last 0) - unlinked 10000 (time 1709789423 ; total 534 ; last 534) - unlinked 20000 (time 1709789871 ; total 982 ; last 448) - unlinked 30000 (time 1709790188 ; total 1299 ; last 317) - unlinked 40000 (time 1709790339 ; total 1450 ; last 151) total: 50000 unlinks in 1495 seconds: 33.444817 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server unloading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 PASS 101a (3552s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg323-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg323-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg323-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 oleg323-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg323-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:46:37 (1709790397) start mds service on oleg323-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock oleg323-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server