-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:38:10 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg203-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg203-server oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:39:34 (1709786374) start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h39m50s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800a9f0d800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:40:35.000000000 -0500 Modify: 2024-03-06 23:40:35.000000000 -0500 Change: 2024-03-06 23:40:35.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. oleg203-server: oleg203-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg203-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg203-server: oleg203-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg203-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg203-server: oleg203-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg203-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server PASS 35b (115s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:31 (1709786491) start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server 000000 2 000008 oleg203-server: 8+0 records in oleg203-server: 8+0 records out oleg203-server: 4096 bytes (4.1 kB) copied, 0.000632946 s, 6.5 MB/s start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 PASS 38 (67s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg203-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg203-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg203-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg203-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:40 (1709786560) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg203-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec stop ost2 service on oleg203-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg203-server umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. oleg203-server: oleg203-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg203-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg203-server: oleg203-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg203-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg203-server: oleg203-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg203-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server PASS 51 (159s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:22 (1709786722) start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 321 (time 1709786751.86 total 10.80 last 29.71) - open/close 745 (time 1709786764.51 total 23.45 last 33.52) - open/close 879 (time 1709786774.80 total 33.74 last 13.02) - open/close 1214 (time 1709786784.81 total 43.75 last 33.48) - open/close 1589 (time 1709786794.82 total 53.76 last 37.46) - open/close 1830 (time 1709786805.69 total 64.63 last 22.17) - open/close 2018 (time 1709786815.95 total 74.89 last 18.32) - open/close 2250 (time 1709786825.96 total 84.90 last 23.18) - open/close 2461 (time 1709786835.97 total 94.91 last 21.09) - open/close 2589 (time 1709786846.85 total 105.79 last 11.77) - open/close 2854 (time 1709786857.54 total 116.48 last 24.79) - open/close 3099 (time 1709786867.69 total 126.64 last 24.12) - open/close 3263 (time 1709786878.71 total 137.65 last 14.89) - open/close 3489 (time 1709786889.56 total 148.50 last 20.83) - open/close 3727 (time 1709786899.57 total 158.51 last 23.79) - open/close 3994 (time 1709786909.87 total 168.81 last 25.92) - open/close 4146 (time 1709786920.86 total 179.80 last 13.83) - open/close 4294 (time 1709786930.91 total 189.86 last 14.71) - open/close 4516 (time 1709786940.92 total 199.86 last 22.19) - open/close 4659 (time 1709786951.06 total 210.00 last 14.10) - open/close 4778 (time 1709786961.94 total 220.88 last 10.94) - open/close 5060 (time 1709786972.32 total 231.26 last 27.18) - open/close 5284 (time 1709786982.63 total 241.57 last 21.71) - open/close 5598 (time 1709786993.40 total 252.34 last 29.17) - open/close 5758 (time 1709787005.04 total 263.98 last 13.75) - open/close 5930 (time 1709787015.05 total 273.99 last 17.18) - open/close 6151 (time 1709787025.06 total 284.00 last 22.07) - open/close 6305 (time 1709787035.93 total 294.87 last 14.17) - open/close 6528 (time 1709787046.76 total 305.70 last 20.60) - open/close 6732 (time 1709787057.49 total 316.43 last 19.00) - open/close 6877 (time 1709787067.49 total 326.44 last 14.50) - open/close 7072 (time 1709787078.41 total 337.35 last 17.86) - open/close 7218 (time 1709787088.42 total 347.36 last 14.60) - open/close 7517 (time 1709787098.80 total 357.74 last 28.80) - open/close 7657 (time 1709787108.80 total 367.74 last 14.00) - open/close 7746 (time 1709787119.59 total 378.53 last 8.25) - open/close 7905 (time 1709787129.60 total 388.54 last 15.90) - open/close 8096 (time 1709787139.71 total 398.65 last 18.89) - open/close 8247 (time 1709787149.80 total 408.74 last 14.95) - open/close 8384 (time 1709787159.82 total 418.76 last 13.68) - open/close 8552 (time 1709787170.48 total 429.42 last 15.75) - open/close 8812 (time 1709787180.49 total 439.43 last 25.98) - open/close 9083 (time 1709787190.51 total 449.45 last 27.04) - open/close 9227 (time 1709787200.75 total 459.69 last 14.07) - open/close 9424 (time 1709787210.75 total 469.70 last 19.68) - open/close 9610 (time 1709787221.61 total 480.55 last 17.13) - open/close 10000 (time 1709787231.59 total 490.53 last 39.09) - open/close 10576 (time 1709787241.60 total 500.54 last 57.54) - open/close 10586 (time 1709787252.85 total 511.79 last 0.89) - open/close 11081 (time 1709787262.90 total 521.84 last 49.27) - open/close 11578 (time 1709787272.90 total 531.84 last 49.69) - open/close 12103 (time 1709787282.90 total 541.84 last 52.50) - open/close 12590 (time 1709787292.91 total 551.85 last 48.66) - open/close 13106 (time 1709787302.92 total 561.87 last 51.51) - open/close 13581 (time 1709787312.95 total 571.89 last 47.37) - open/close 14038 (time 1709787322.96 total 581.91 last 45.64) - open/close 14553 (time 1709787332.97 total 591.91 last 51.45) - open/close 15024 (time 1709787343.01 total 601.95 last 46.92) - open/close 15462 (time 1709787353.05 total 611.99 last 43.63) - open/close 15981 (time 1709787363.06 total 622.00 last 51.85) - open/close 16405 (time 1709787373.09 total 632.03 last 42.29) - open/close 16782 (time 1709787383.09 total 642.03 last 37.68) - open/close 17199 (time 1709787393.10 total 652.04 last 41.68) - open/close 17538 (time 1709787403.11 total 662.05 last 33.84) - open/close 17913 (time 1709787413.14 total 672.08 last 37.39) - open/close 18291 (time 1709787423.15 total 682.09 last 37.76) - open/close 18665 (time 1709787433.18 total 692.12 last 37.30) - open/close 19061 (time 1709787443.18 total 702.12 last 39.59) - open/close 19444 (time 1709787453.19 total 712.13 last 38.28) - open/close 19768 (time 1709787463.20 total 722.15 last 32.35) - open/close 20000 (time 1709787470.38 total 729.32 last 32.33) - open/close 20352 (time 1709787480.40 total 739.34 last 35.12) - open/close 20749 (time 1709787490.41 total 749.35 last 39.67) - open/close 20815 (time 1709787503.69 total 762.64 last 4.97) - open/close 21152 (time 1709787513.72 total 772.66 last 33.62) - open/close 21470 (time 1709787523.74 total 782.68 last 31.72) - open/close 21744 (time 1709787533.80 total 792.74 last 27.26) - open/close 22060 (time 1709787543.83 total 802.77 last 31.49) - open/close 22385 (time 1709787553.83 total 812.77 last 32.49) - open/close 22701 (time 1709787563.86 total 822.80 last 31.53) - open/close 23026 (time 1709787573.89 total 832.83 last 32.39) - open/close 23352 (time 1709787583.89 total 842.83 last 32.59) - open/close 23659 (time 1709787593.89 total 852.84 last 30.69) - open/close 23956 (time 1709787603.90 total 862.84 last 29.68) - open/close 24287 (time 1709787613.91 total 872.85 last 33.07) - open/close 24597 (time 1709787623.92 total 882.86 last 30.98) - open/close 24667 (time 1709787637.17 total 896.11 last 5.28) - open/close 24675 (time 1709787648.50 total 907.45 last 0.71) - open/close 24972 (time 1709787658.52 total 917.46 last 29.66) - open/close 25303 (time 1709787668.55 total 927.49 last 33.01) - open/close 25596 (time 1709787678.58 total 937.52 last 29.21) - open/close 25908 (time 1709787688.59 total 947.53 last 31.16) - open/close 26198 (time 1709787698.60 total 957.54 last 28.97) - open/close 26489 (time 1709787708.64 total 967.58 last 28.97) - open/close 26744 (time 1709787718.64 total 977.58 last 25.50) - open/close 26999 (time 1709787728.65 total 987.59 last 25.49) - open/close 27276 (time 1709787738.65 total 997.59 last 27.70) - open/close 27545 (time 1709787748.67 total 1007.61 last 26.85) - open/close 27805 (time 1709787758.71 total 1017.66 last 25.88) - open/close 28073 (time 1709787768.72 total 1027.66 last 26.78) - open/close 28361 (time 1709787778.73 total 1037.67 last 28.79) - open/close 28667 (time 1709787788.75 total 1047.69 last 30.53) - open/close 28740 (time 1709787802.59 total 1061.53 last 5.27) - open/close 29034 (time 1709787812.62 total 1071.56 last 29.32) - open/close 29290 (time 1709787822.62 total 1081.57 last 25.59) - open/close 29599 (time 1709787832.63 total 1091.57 last 30.87) - open/close 29872 (time 1709787842.64 total 1101.58 last 27.29) - open/close 30000 (time 1709787847.83 total 1106.77 last 24.66) - open/close 30240 (time 1709787857.84 total 1116.79 last 23.96) - open/close 30478 (time 1709787867.86 total 1126.80 last 23.77) - open/close 30646 (time 1709787885.20 total 1144.14 last 9.69) - open/close 30894 (time 1709787895.26 total 1154.20 last 24.65) - open/close 31136 (time 1709787905.27 total 1164.21 last 24.18) - open/close 31385 (time 1709787915.28 total 1174.22 last 24.87) - open/close 31641 (time 1709787925.28 total 1184.22 last 25.59) - open/close 31909 (time 1709787935.30 total 1194.24 last 26.75) - open/close 32189 (time 1709787945.31 total 1204.25 last 27.97) - open/close 32507 (time 1709787955.33 total 1214.27 last 31.75) - open/close 32820 (time 1709787965.34 total 1224.28 last 31.26) - open/close 33104 (time 1709787975.36 total 1234.30 last 28.35) - open/close 33262 (time 1709787992.27 total 1251.21 last 9.34) - open/close 33596 (time 1709788002.29 total 1261.23 last 33.35) - open/close 33941 (time 1709788012.31 total 1271.25 last 34.41) - open/close 34301 (time 1709788022.32 total 1281.26 last 35.97) - open/close 34669 (time 1709788032.33 total 1291.27 last 36.77) - open/close 35042 (time 1709788042.34 total 1301.28 last 37.25) - open/close 35412 (time 1709788052.34 total 1311.29 last 37.00) - open/close 35779 (time 1709788062.36 total 1321.30 last 36.64) - open/close 36119 (time 1709788072.38 total 1331.32 last 33.94) - open/close 36493 (time 1709788082.39 total 1341.33 last 37.35) - open/close 36858 (time 1709788092.42 total 1351.36 last 36.41) - open/close 37199 (time 1709788102.43 total 1361.37 last 34.05) - open/close 37552 (time 1709788112.45 total 1371.39 last 35.24) - open/close 37893 (time 1709788122.47 total 1381.41 last 34.03) - open/close 38227 (time 1709788132.49 total 1391.43 last 33.33) - open/close 38570 (time 1709788142.51 total 1401.45 last 34.23) - open/close 38909 (time 1709788152.52 total 1411.46 last 33.88) - open/close 39260 (time 1709788162.53 total 1421.47 last 35.04) - open/close 39609 (time 1709788172.54 total 1431.49 last 34.86) - open/close 39947 (time 1709788182.57 total 1441.51 last 33.73) - open/close 40288 (time 1709788192.58 total 1451.52 last 34.05) - open/close 40631 (time 1709788202.59 total 1461.53 last 34.28) - open/close 40967 (time 1709788212.59 total 1471.53 last 33.59) - open/close 41313 (time 1709788222.60 total 1481.54 last 34.58) - open/close 41647 (time 1709788232.63 total 1491.57 last 33.30) - open/close 41970 (time 1709788242.63 total 1501.57 last 32.28) - open/close 42304 (time 1709788252.65 total 1511.59 last 33.36) - open/close 42633 (time 1709788262.66 total 1521.60 last 32.86) - open/close 42950 (time 1709788272.69 total 1531.63 last 31.61) - open/close 43265 (time 1709788282.72 total 1541.66 last 31.41) - open/close 43566 (time 1709788292.74 total 1551.69 last 30.01) - open/close 43883 (time 1709788302.75 total 1561.70 last 31.67) - open/close 44193 (time 1709788312.76 total 1571.70 last 30.99) - open/close 44500 (time 1709788322.78 total 1581.72 last 30.64) - open/close 44802 (time 1709788332.80 total 1591.74 last 30.13) - open/close 45103 (time 1709788342.82 total 1601.76 last 30.04) - open/close 45400 (time 1709788352.84 total 1611.78 last 29.65) - open/close 45692 (time 1709788362.86 total 1621.80 last 29.14) - open/close 45985 (time 1709788372.89 total 1631.83 last 29.21) - open/close 46274 (time 1709788382.92 total 1641.86 last 28.82) - open/close 46550 (time 1709788392.94 total 1651.88 last 27.54) - open/close 46850 (time 1709788402.95 total 1661.89 last 29.96) - open/close 47141 (time 1709788412.97 total 1671.91 last 29.04) - open/close 47429 (time 1709788422.98 total 1681.92 last 28.78) - open/close 47720 (time 1709788432.99 total 1691.93 last 29.06) - open/close 47996 (time 1709788443.02 total 1701.96 last 27.54) - open/close 48274 (time 1709788453.05 total 1711.99 last 27.70) - open/close 48555 (time 1709788463.08 total 1722.02 last 28.02) - open/close 48833 (time 1709788473.09 total 1732.03 last 27.78) - open/close 49129 (time 1709788483.10 total 1742.04 last 29.57) - open/close 49403 (time 1709788493.11 total 1752.05 last 27.36) - open/close 49684 (time 1709788503.14 total 1762.08 last 28.03) - open/close 49969 (time 1709788513.16 total 1772.10 last 28.44) total: 50000 open/close in 1773.14 seconds: 28.20 ops/second - unlinked 0 (time 1709788515 ; total 0 ; last 0) - unlinked 10000 (time 1709788933 ; total 418 ; last 418) - unlinked 20000 (time 1709789312 ; total 797 ; last 379) - unlinked 30000 (time 1709789597 ; total 1082 ; last 285) - unlinked 40000 (time 1709789749 ; total 1234 ; last 152) total: 50000 unlinks in 1287 seconds: 38.850040 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 PASS 101a (3094s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg203-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg203-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg203-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg203-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:36:58 (1709789818) start mds service on oleg203-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9ecb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9ecb800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg203-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg203-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3634 sec ========== 00:38:45 (1709789925) === conf-sanity: start cleanup 00:38:46 (1709789926) === === conf-sanity: finish cleanup 00:38:46 (1709789926) ===