-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:44:09 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg353-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg353-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg353-server oleg353-server: oleg353-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg353-server: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg353-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:45:19 (1709786719) start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h45m34s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b58e6000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:46:19.000000000 -0500 Modify: 2024-03-06 23:46:19.000000000 -0500 Change: 2024-03-06 23:46:19.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server unloading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. oleg353-server: oleg353-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg353-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg353-server: oleg353-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg353-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg353-server: oleg353-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg353-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg353-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: oleg353-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg353-server: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg353-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server PASS 35b (134s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:47:36 (1709786856) start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server 000000 2 000008 oleg353-server: 8+0 records in oleg353-server: 8+0 records out oleg353-server: 4096 bytes (4.1 kB) copied, 0.00140382 s, 2.9 MB/s start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server unloading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 PASS 38 (90s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg353-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg353-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg353-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg353-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:49:10 (1709786950) Stopping clients: oleg353-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: oleg353-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg353-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg353-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg353-server umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server unloading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. oleg353-server: oleg353-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg353-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg353-server: oleg353-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg353-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg353-server: oleg353-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg353-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg353-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: oleg353-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg353-server: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg353-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server PASS 51 (199s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:33 (1709787153) start mds service on oleg353-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 138 (time 1709787189.27 total 10.93 last 12.62) - open/close 363 (time 1709787199.28 total 20.94 last 22.48) - open/close 569 (time 1709787209.28 total 30.94 last 20.60) - open/close 809 (time 1709787219.29 total 40.95 last 23.98) - open/close 1122 (time 1709787229.53 total 51.19 last 30.57) - open/close 1277 (time 1709787239.73 total 61.39 last 15.20) - open/close 1489 (time 1709787249.74 total 71.40 last 21.18) - open/close 1616 (time 1709787259.85 total 81.52 last 12.55) - open/close 1748 (time 1709787269.87 total 91.53 last 13.19) - open/close 1905 (time 1709787280.34 total 102.00 last 14.99) - open/close 2139 (time 1709787290.64 total 112.31 last 22.72) - open/close 2326 (time 1709787300.65 total 122.32 last 18.68) - open/close 2488 (time 1709787311.10 total 132.76 last 15.51) - open/close 2775 (time 1709787321.11 total 142.77 last 28.66) - open/close 2955 (time 1709787331.12 total 152.79 last 17.98) - open/close 3141 (time 1709787341.88 total 163.54 last 17.29) - open/close 3333 (time 1709787351.89 total 173.55 last 19.19) - open/close 3494 (time 1709787361.96 total 183.62 last 15.99) - open/close 3717 (time 1709787371.96 total 193.62 last 22.30) - open/close 3934 (time 1709787381.98 total 203.65 last 21.65) - open/close 4260 (time 1709787392.31 total 213.97 last 31.57) - open/close 4442 (time 1709787402.32 total 223.98 last 18.18) - open/close 4635 (time 1709787412.32 total 233.99 last 19.30) - open/close 4763 (time 1709787422.78 total 244.45 last 12.24) - open/close 4931 (time 1709787432.82 total 254.49 last 16.73) - open/close 5096 (time 1709787443.39 total 265.05 last 15.61) - open/close 5298 (time 1709787454.42 total 276.09 last 18.31) - open/close 5438 (time 1709787464.70 total 286.37 last 13.62) - open/close 5617 (time 1709787475.41 total 297.07 last 16.72) - open/close 5730 (time 1709787485.73 total 307.39 last 10.95) - open/close 5874 (time 1709787496.48 total 318.14 last 13.39) - open/close 6055 (time 1709787507.29 total 328.96 last 16.74) - open/close 6223 (time 1709787517.30 total 338.96 last 16.79) - open/close 6386 (time 1709787527.32 total 348.98 last 16.27) - open/close 6655 (time 1709787537.33 total 359.00 last 26.86) - open/close 6760 (time 1709787547.55 total 369.21 last 10.28) - open/close 6960 (time 1709787557.55 total 379.22 last 19.99) - open/close 7152 (time 1709787567.60 total 389.27 last 19.10) - open/close 7323 (time 1709787578.33 total 399.99 last 15.95) - open/close 7489 (time 1709787589.19 total 410.86 last 15.28) - open/close 7684 (time 1709787599.41 total 421.08 last 19.08) - open/close 7873 (time 1709787609.42 total 431.08 last 18.90) - open/close 8057 (time 1709787620.44 total 442.10 last 16.69) - open/close 8191 (time 1709787631.27 total 452.93 last 12.37) - open/close 8367 (time 1709787641.99 total 463.66 last 16.41) - open/close 8511 (time 1709787652.45 total 474.12 last 13.77) - open/close 8767 (time 1709787662.57 total 484.23 last 25.31) - open/close 8946 (time 1709787672.57 total 494.23 last 17.89) - open/close 9143 (time 1709787683.44 total 505.10 last 18.12) - open/close 9267 (time 1709787693.45 total 515.11 last 12.39) - open/close 9365 (time 1709787703.47 total 525.14 last 9.77) - open/close 9635 (time 1709787713.86 total 535.53 last 25.99) - open/close 9818 (time 1709787724.09 total 545.76 last 17.89) - open/close 10000 (time 1709787733.40 total 555.06 last 19.56) - open/close 10148 (time 1709787743.41 total 565.08 last 14.77) - open/close 10309 (time 1709787753.78 total 575.44 last 15.53) - open/close 10488 (time 1709787765.79 total 587.45 last 14.90) - open/close 10611 (time 1709787776.81 total 598.47 last 11.16) - open/close 10768 (time 1709787787.75 total 609.42 last 14.35) - open/close 11114 (time 1709787797.77 total 619.43 last 34.54) - open/close 11308 (time 1709787811.84 total 633.50 last 13.79) - open/close 11817 (time 1709787821.85 total 643.51 last 50.84) - open/close 12341 (time 1709787831.87 total 653.53 last 52.31) - open/close 12885 (time 1709787841.89 total 663.55 last 54.27) - open/close 13386 (time 1709787851.91 total 673.58 last 49.98) - open/close 13879 (time 1709787861.93 total 683.60 last 49.22) - open/close 14357 (time 1709787871.95 total 693.61 last 47.73) - open/close 14777 (time 1709787881.95 total 703.62 last 41.98) - open/close 15213 (time 1709787891.96 total 713.63 last 43.55) - open/close 15667 (time 1709787901.98 total 723.64 last 45.33) - open/close 16117 (time 1709787911.98 total 733.65 last 44.97) - open/close 16541 (time 1709787921.99 total 743.65 last 42.39) - open/close 16955 (time 1709787932.00 total 753.66 last 41.37) - open/close 17372 (time 1709787942.01 total 763.67 last 41.66) - open/close 17753 (time 1709787952.02 total 773.68 last 38.05) - open/close 18165 (time 1709787962.02 total 783.68 last 41.20) - open/close 18564 (time 1709787972.04 total 793.70 last 39.84) - open/close 18935 (time 1709787982.04 total 803.71 last 37.07) - open/close 19370 (time 1709787992.05 total 813.72 last 43.46) - open/close 19788 (time 1709788002.06 total 823.72 last 41.77) - open/close 20000 (time 1709788007.50 total 829.16 last 38.98) - open/close 20397 (time 1709788017.51 total 839.18 last 39.64) - open/close 20772 (time 1709788027.53 total 849.19 last 37.44) - open/close 21139 (time 1709788037.54 total 859.20 last 36.66) - open/close 21484 (time 1709788047.55 total 869.22 last 34.46) - open/close 21829 (time 1709788057.55 total 879.22 last 34.49) - open/close 22174 (time 1709788067.56 total 889.23 last 34.47) - open/close 22580 (time 1709788077.58 total 899.24 last 40.53) - open/close 23046 (time 1709788087.59 total 909.26 last 46.55) - open/close 23513 (time 1709788097.62 total 919.28 last 46.56) - open/close 23964 (time 1709788107.64 total 929.30 last 45.02) - open/close 24408 (time 1709788117.64 total 939.31 last 44.38) - open/close 24848 (time 1709788127.65 total 949.32 last 43.95) - open/close 25299 (time 1709788137.67 total 959.33 last 45.03) - open/close 25743 (time 1709788147.69 total 969.35 last 44.31) - open/close 26149 (time 1709788157.69 total 979.36 last 40.59) - open/close 26573 (time 1709788167.72 total 989.39 last 42.27) - open/close 26990 (time 1709788177.72 total 999.39 last 41.69) - open/close 27413 (time 1709788187.73 total 1009.39 last 42.28) - open/close 27817 (time 1709788197.74 total 1019.40 last 40.36) - open/close 28205 (time 1709788207.74 total 1029.40 last 38.79) - open/close 28603 (time 1709788217.76 total 1039.42 last 39.73) - open/close 29021 (time 1709788227.78 total 1049.44 last 41.72) - open/close 29417 (time 1709788237.80 total 1059.47 last 39.51) - open/close 29783 (time 1709788247.82 total 1069.49 last 36.52) - open/close 30000 (time 1709788254.45 total 1076.11 last 32.74) - open/close 30359 (time 1709788264.46 total 1086.13 last 35.86) - open/close 30710 (time 1709788274.48 total 1096.15 last 35.02) - open/close 31054 (time 1709788284.50 total 1106.16 last 34.36) - open/close 31375 (time 1709788294.52 total 1116.18 last 32.04) - open/close 31730 (time 1709788304.54 total 1126.20 last 35.44) - open/close 32070 (time 1709788314.56 total 1136.23 last 33.91) - open/close 32404 (time 1709788324.57 total 1146.23 last 33.38) - open/close 32776 (time 1709788334.57 total 1156.24 last 37.18) - open/close 33123 (time 1709788344.59 total 1166.25 last 34.66) - open/close 33479 (time 1709788354.61 total 1176.27 last 35.53) - open/close 33805 (time 1709788364.61 total 1186.28 last 32.58) - open/close 34125 (time 1709788374.62 total 1196.29 last 31.96) - open/close 34438 (time 1709788384.63 total 1206.29 last 31.28) - open/close 34771 (time 1709788394.66 total 1216.32 last 33.20) - open/close 35105 (time 1709788404.67 total 1226.34 last 33.36) - open/close 35422 (time 1709788414.69 total 1236.36 last 31.63) - open/close 35767 (time 1709788424.73 total 1246.39 last 34.39) - open/close 36072 (time 1709788434.73 total 1256.40 last 30.48) - open/close 36380 (time 1709788444.74 total 1266.41 last 30.77) - open/close 36683 (time 1709788454.75 total 1276.42 last 30.27) - open/close 36990 (time 1709788464.79 total 1286.45 last 30.58) - open/close 37296 (time 1709788474.80 total 1296.46 last 30.58) - open/close 37600 (time 1709788484.80 total 1306.47 last 30.38) - open/close 37895 (time 1709788494.81 total 1316.47 last 29.48) - open/close 38186 (time 1709788504.81 total 1326.48 last 29.08) - open/close 38486 (time 1709788514.84 total 1336.51 last 29.91) - open/close 38787 (time 1709788524.86 total 1346.52 last 30.06) - open/close 39082 (time 1709788534.89 total 1356.55 last 29.40) - open/close 39384 (time 1709788544.91 total 1366.57 last 30.14) - open/close 39680 (time 1709788554.91 total 1376.57 last 29.60) - open/close 39971 (time 1709788564.93 total 1386.60 last 29.04) - open/close 40274 (time 1709788574.94 total 1396.60 last 30.28) - open/close 40579 (time 1709788584.96 total 1406.63 last 30.42) - open/close 40879 (time 1709788594.97 total 1416.64 last 29.97) - open/close 41182 (time 1709788604.98 total 1426.64 last 30.28) - open/close 41475 (time 1709788615.00 total 1436.67 last 29.22) - open/close 41783 (time 1709788625.02 total 1446.68 last 30.75) - open/close 42078 (time 1709788635.05 total 1456.71 last 29.42) - open/close 42379 (time 1709788645.06 total 1466.72 last 30.07) - open/close 42677 (time 1709788655.06 total 1476.73 last 29.79) - open/close 42973 (time 1709788665.08 total 1486.75 last 29.54) - open/close 43249 (time 1709788675.09 total 1496.76 last 27.57) - open/close 43519 (time 1709788685.11 total 1506.78 last 26.95) - open/close 43788 (time 1709788695.13 total 1516.80 last 26.84) - open/close 44059 (time 1709788705.17 total 1526.83 last 27.00) - open/close 44327 (time 1709788715.19 total 1536.86 last 26.74) - open/close 44601 (time 1709788725.21 total 1546.87 last 27.35) - open/close 44885 (time 1709788735.21 total 1556.88 last 28.39) - open/close 45168 (time 1709788745.21 total 1566.88 last 28.30) - open/close 45440 (time 1709788755.24 total 1576.91 last 27.13) - open/close 45698 (time 1709788765.26 total 1586.93 last 25.74) - open/close 45941 (time 1709788775.30 total 1596.96 last 24.21) - open/close 46190 (time 1709788785.30 total 1606.97 last 24.89) - open/close 46439 (time 1709788795.31 total 1616.98 last 24.88) - open/close 46678 (time 1709788805.34 total 1627.01 last 23.82) - open/close 46920 (time 1709788815.36 total 1637.02 last 24.17) - open/close 47163 (time 1709788825.40 total 1647.06 last 24.20) - open/close 47414 (time 1709788835.42 total 1657.08 last 25.05) - open/close 47661 (time 1709788845.42 total 1667.09 last 24.68) - open/close 47901 (time 1709788855.44 total 1677.10 last 23.97) - open/close 48142 (time 1709788865.46 total 1687.12 last 24.06) - open/close 48385 (time 1709788875.48 total 1697.15 last 24.24) - open/close 48625 (time 1709788885.51 total 1707.17 last 23.93) - open/close 48885 (time 1709788895.52 total 1717.19 last 25.97) - open/close 49131 (time 1709788905.56 total 1727.22 last 24.51) - open/close 49375 (time 1709788915.57 total 1737.23 last 24.38) - open/close 49632 (time 1709788925.57 total 1747.23 last 25.70) - open/close 49870 (time 1709788935.60 total 1757.27 last 23.72) total: 50000 open/close in 1762.86 seconds: 28.36 ops/second - unlinked 0 (time 1709788942 ; total 0 ; last 0) - unlinked 10000 (time 1709789436 ; total 494 ; last 494) - unlinked 20000 (time 1709789871 ; total 929 ; last 435) - unlinked 30000 (time 1709790185 ; total 1243 ; last 314) - unlinked 40000 (time 1709790338 ; total 1396 ; last 153) total: 50000 unlinks in 1441 seconds: 34.698128 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server unloading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 PASS 101a (3244s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg353-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg353-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg353-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg353-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:46:39 (1709790399) start mds service on oleg353-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock oleg353-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server start ost1 service on oleg353-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aa01b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa01b000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server unloading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 PASS 101b (83s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg353-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg353-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg353-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg353-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 oleg353-server: oleg353-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3855 sec ========== 00:48:25 (1709790505) === conf-sanity: start cleanup 00:48:26 (1709790506) === === conf-sanity: finish cleanup 00:48:26 (1709790506) ===