-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:38:55 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg251-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:08 (1709786408) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m23s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b5c07000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:06.000000000 -0500 Modify: 2024-03-06 23:41:06.000000000 -0500 Change: 2024-03-06 23:41:06.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server PASS 35b (114s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:04 (1709786524) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server 000000 2 000008 oleg251-server: 8+0 records in oleg251-server: 8+0 records out oleg251-server: 4096 bytes (4.1 kB) copied, 0.00183359 s, 2.2 MB/s start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 PASS 38 (74s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg251-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:43:21 (1709786601) Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg251-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg251-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server PASS 51 (166s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:10 (1709786770) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 305 (time 1709786802.61 total 10.03 last 30.40) - open/close 545 (time 1709786823.33 total 30.75 last 11.58) - open/close 752 (time 1709786833.67 total 41.09 last 20.01) - open/close 905 (time 1709786843.86 total 51.28 last 15.02) - open/close 1243 (time 1709786853.86 total 61.28 last 33.79) - open/close 1407 (time 1709786863.87 total 71.29 last 16.39) - open/close 1756 (time 1709786874.64 total 82.06 last 32.40) - open/close 1982 (time 1709786884.66 total 92.09 last 22.55) - open/close 2122 (time 1709786894.89 total 102.31 last 13.69) - open/close 2428 (time 1709786904.89 total 112.31 last 30.59) - open/close 2687 (time 1709786914.90 total 122.33 last 25.87) - open/close 2829 (time 1709786924.91 total 132.33 last 14.19) - open/close 3030 (time 1709786936.57 total 143.99 last 17.25) - open/close 3225 (time 1709786946.57 total 153.99 last 19.50) - open/close 3407 (time 1709786956.57 total 163.99 last 18.20) - open/close 3539 (time 1709786966.59 total 174.01 last 13.17) - open/close 3742 (time 1709786976.87 total 184.30 last 19.74) - open/close 4010 (time 1709786986.99 total 194.41 last 26.49) - open/close 4246 (time 1709786997.99 total 205.41 last 21.46) - open/close 4414 (time 1709787008.53 total 215.96 last 15.93) - open/close 4579 (time 1709787018.56 total 225.98 last 16.46) - open/close 4700 (time 1709787028.71 total 236.14 last 11.91) - open/close 4867 (time 1709787038.72 total 246.14 last 16.69) - open/close 5084 (time 1709787049.50 total 256.92 last 20.14) - open/close 5297 (time 1709787059.51 total 266.93 last 21.28) - open/close 5435 (time 1709787070.52 total 277.94 last 12.53) - open/close 5624 (time 1709787080.93 total 288.35 last 18.16) - open/close 5808 (time 1709787091.37 total 298.79 last 17.62) - open/close 5989 (time 1709787101.38 total 308.80 last 18.08) - open/close 6242 (time 1709787111.51 total 318.93 last 24.99) - open/close 6416 (time 1709787121.72 total 329.14 last 17.04) - open/close 6507 (time 1709787131.74 total 339.16 last 9.08) - open/close 6671 (time 1709787142.61 total 350.03 last 15.08) - open/close 6939 (time 1709787153.50 total 360.92 last 24.63) - open/close 7137 (time 1709787164.00 total 371.42 last 18.86) - open/close 7321 (time 1709787174.76 total 382.18 last 17.09) - open/close 7482 (time 1709787185.07 total 392.49 last 15.62) - open/close 7582 (time 1709787195.07 total 402.50 last 9.99) - open/close 7758 (time 1709787205.08 total 412.50 last 17.59) - open/close 8001 (time 1709787215.95 total 423.37 last 22.36) - open/close 8301 (time 1709787225.95 total 433.37 last 29.99) - open/close 8406 (time 1709787236.41 total 443.83 last 10.04) - open/close 8514 (time 1709787247.29 total 454.71 last 9.93) - open/close 8611 (time 1709787264.95 total 472.37 last 5.49) - open/close 8788 (time 1709787275.75 total 483.17 last 16.38) - open/close 8899 (time 1709787286.17 total 493.59 last 10.65) - open/close 9108 (time 1709787296.67 total 504.09 last 19.91) - open/close 9311 (time 1709787307.31 total 514.73 last 19.07) - open/close 9365 (time 1709787319.80 total 527.22 last 4.32) - open/close 9899 (time 1709787329.81 total 537.23 last 53.38) - open/close 10350 (time 1709787339.82 total 547.24 last 45.04) - open/close 10771 (time 1709787349.82 total 557.24 last 42.09) - open/close 11289 (time 1709787359.83 total 567.25 last 51.77) - open/close 11760 (time 1709787369.83 total 577.25 last 47.08) - open/close 12098 (time 1709787389.12 total 596.55 last 17.52) - open/close 12485 (time 1709787399.15 total 606.57 last 38.61) - open/close 12818 (time 1709787409.17 total 616.59 last 33.23) - open/close 13210 (time 1709787419.19 total 626.61 last 39.13) - open/close 13602 (time 1709787429.20 total 636.62 last 39.13) - open/close 13966 (time 1709787439.22 total 646.64 last 36.35) - open/close 14364 (time 1709787449.22 total 656.65 last 39.78) - open/close 14719 (time 1709787459.27 total 666.69 last 35.36) - open/close 15056 (time 1709787469.27 total 676.69 last 33.69) - open/close 15409 (time 1709787479.28 total 686.70 last 35.28) - open/close 15786 (time 1709787489.29 total 696.71 last 37.66) - open/close 16129 (time 1709787499.31 total 706.73 last 34.21) - open/close 16468 (time 1709787509.33 total 716.75 last 33.84) - open/close 16817 (time 1709787519.35 total 726.77 last 34.83) - open/close 17105 (time 1709787529.40 total 736.82 last 28.66) - open/close 17369 (time 1709787539.41 total 746.84 last 26.36) - open/close 17702 (time 1709787549.43 total 756.85 last 33.26) - open/close 18017 (time 1709787559.43 total 766.85 last 31.49) - open/close 18356 (time 1709787569.43 total 776.86 last 33.88) - open/close 18693 (time 1709787579.46 total 786.88 last 33.62) - open/close 18996 (time 1709787589.48 total 796.90 last 30.24) - open/close 19311 (time 1709787599.49 total 806.91 last 31.48) - open/close 19643 (time 1709787609.52 total 816.94 last 33.07) - open/close 19897 (time 1709787619.55 total 826.97 last 25.34) - open/close 20000 (time 1709787623.08 total 830.50 last 29.17) - open/close 20312 (time 1709787633.10 total 840.53 last 31.12) - open/close 20621 (time 1709787643.13 total 850.55 last 30.83) - open/close 20888 (time 1709787653.13 total 860.55 last 26.69) - open/close 21208 (time 1709787663.13 total 870.56 last 31.98) - open/close 21496 (time 1709787673.14 total 880.56 last 28.79) - open/close 21801 (time 1709787683.14 total 890.56 last 30.48) - open/close 22104 (time 1709787693.16 total 900.58 last 30.25) - open/close 22363 (time 1709787703.20 total 910.62 last 25.80) - open/close 22610 (time 1709787713.21 total 920.63 last 24.67) - open/close 22857 (time 1709787723.23 total 930.65 last 24.64) - open/close 23105 (time 1709787733.29 total 940.71 last 24.66) - open/close 23375 (time 1709787743.29 total 950.71 last 27.00) - open/close 23631 (time 1709787753.31 total 960.73 last 25.56) - open/close 23885 (time 1709787763.33 total 970.75 last 25.33) - open/close 24158 (time 1709787773.36 total 980.78 last 27.24) - open/close 24406 (time 1709787783.39 total 990.81 last 24.72) - open/close 24662 (time 1709787793.40 total 1000.82 last 25.57) - open/close 24930 (time 1709787803.44 total 1010.86 last 26.69) - open/close 25212 (time 1709787813.47 total 1020.89 last 28.11) - open/close 25470 (time 1709787823.49 total 1030.91 last 25.77) - open/close 25745 (time 1709787833.50 total 1040.92 last 27.46) - open/close 25995 (time 1709787843.53 total 1050.95 last 24.93) - open/close 26219 (time 1709787853.53 total 1060.95 last 22.39) - open/close 26426 (time 1709787863.64 total 1071.06 last 20.48) - open/close 26691 (time 1709787873.65 total 1081.07 last 26.47) - open/close 26930 (time 1709787883.68 total 1091.11 last 23.83) - open/close 27164 (time 1709787893.70 total 1101.12 last 23.37) - open/close 27372 (time 1709787903.75 total 1111.17 last 20.69) - open/close 27414 (time 1709787916.63 total 1124.05 last 3.26) - open/close 27637 (time 1709787926.70 total 1134.12 last 22.16) - open/close 27886 (time 1709787936.72 total 1144.14 last 24.84) - open/close 28160 (time 1709787946.74 total 1154.16 last 27.34) - open/close 28439 (time 1709787956.75 total 1164.17 last 27.88) - open/close 28734 (time 1709787966.77 total 1174.19 last 29.45) - open/close 29000 (time 1709787976.78 total 1184.20 last 26.56) - open/close 29266 (time 1709787986.79 total 1194.21 last 26.57) - open/close 29551 (time 1709787996.81 total 1204.23 last 28.46) - open/close 29877 (time 1709788006.83 total 1214.25 last 32.51) - open/close 30000 (time 1709788010.57 total 1217.99 last 32.96) - open/close 30346 (time 1709788020.59 total 1228.01 last 34.51) - open/close 30708 (time 1709788030.60 total 1238.02 last 36.16) - open/close 31067 (time 1709788040.61 total 1248.03 last 35.89) - open/close 31409 (time 1709788050.62 total 1258.04 last 34.14) - open/close 31765 (time 1709788060.64 total 1268.06 last 35.53) - open/close 32088 (time 1709788070.65 total 1278.07 last 32.28) - open/close 32403 (time 1709788080.66 total 1288.08 last 31.47) - open/close 32741 (time 1709788090.66 total 1298.08 last 33.80) - open/close 33057 (time 1709788100.66 total 1308.08 last 31.59) - open/close 33383 (time 1709788110.66 total 1318.08 last 32.60) - open/close 33711 (time 1709788120.67 total 1328.09 last 32.76) - open/close 34031 (time 1709788130.68 total 1338.10 last 31.99) - open/close 34351 (time 1709788140.69 total 1348.11 last 31.96) - open/close 34668 (time 1709788150.72 total 1358.14 last 31.62) - open/close 34992 (time 1709788160.75 total 1368.17 last 32.30) - open/close 35300 (time 1709788170.76 total 1378.18 last 30.77) - open/close 35629 (time 1709788180.78 total 1388.20 last 32.81) - open/close 35943 (time 1709788190.80 total 1398.22 last 31.33) - open/close 36252 (time 1709788200.82 total 1408.24 last 30.85) - open/close 36571 (time 1709788210.84 total 1418.26 last 31.84) - open/close 36893 (time 1709788220.87 total 1428.29 last 32.08) - open/close 37194 (time 1709788230.88 total 1438.30 last 30.09) - open/close 37499 (time 1709788240.90 total 1448.32 last 30.43) - open/close 37789 (time 1709788250.91 total 1458.33 last 28.97) - open/close 38074 (time 1709788260.94 total 1468.36 last 28.42) - open/close 38355 (time 1709788270.94 total 1478.36 last 28.09) - open/close 38631 (time 1709788280.94 total 1488.37 last 27.60) - open/close 38889 (time 1709788290.95 total 1498.38 last 25.77) - open/close 39123 (time 1709788310.61 total 1518.03 last 11.91) - open/close 39399 (time 1709788320.62 total 1528.04 last 27.59) - open/close 39671 (time 1709788330.62 total 1538.04 last 27.19) - open/close 39930 (time 1709788340.62 total 1548.04 last 25.89) - open/close 40000 (time 1709788343.19 total 1550.61 last 27.26) - open/close 40264 (time 1709788353.21 total 1560.64 last 26.34) - open/close 40528 (time 1709788363.22 total 1570.64 last 26.39) - open/close 40790 (time 1709788373.26 total 1580.68 last 26.09) - open/close 41047 (time 1709788383.27 total 1590.69 last 25.67) - open/close 41292 (time 1709788393.29 total 1600.71 last 24.46) - open/close 41556 (time 1709788403.29 total 1610.71 last 26.40) - open/close 41826 (time 1709788413.29 total 1620.71 last 27.00) - open/close 42090 (time 1709788423.30 total 1630.72 last 26.37) - open/close 42365 (time 1709788433.33 total 1640.75 last 27.40) - open/close 42626 (time 1709788443.34 total 1650.76 last 26.09) - open/close 42896 (time 1709788453.36 total 1660.78 last 26.95) - open/close 43148 (time 1709788463.39 total 1670.81 last 25.11) - open/close 43408 (time 1709788473.42 total 1680.84 last 25.93) - open/close 43680 (time 1709788483.44 total 1690.86 last 27.15) - open/close 43948 (time 1709788493.46 total 1700.88 last 26.73) - open/close 44219 (time 1709788503.49 total 1710.91 last 27.02) - open/close 44501 (time 1709788513.52 total 1720.94 last 28.12) - open/close 44776 (time 1709788523.54 total 1730.96 last 27.45) - open/close 45043 (time 1709788533.56 total 1740.98 last 26.64) - open/close 45294 (time 1709788543.58 total 1751.00 last 25.06) - open/close 45565 (time 1709788553.59 total 1761.01 last 27.08) - open/close 45810 (time 1709788563.59 total 1771.01 last 24.50) - open/close 46072 (time 1709788573.61 total 1781.03 last 26.15) - open/close 46334 (time 1709788583.65 total 1791.07 last 26.09) - open/close 46585 (time 1709788593.67 total 1801.09 last 25.06) - open/close 46849 (time 1709788603.68 total 1811.10 last 26.38) - open/close 47113 (time 1709788613.71 total 1821.13 last 26.30) - open/close 47360 (time 1709788623.73 total 1831.15 last 24.66) - open/close 47607 (time 1709788633.74 total 1841.16 last 24.69) - open/close 47859 (time 1709788643.74 total 1851.16 last 25.19) - open/close 48109 (time 1709788653.77 total 1861.19 last 24.94) - open/close 48370 (time 1709788663.79 total 1871.21 last 26.04) - open/close 48626 (time 1709788673.80 total 1881.23 last 25.56) - open/close 48884 (time 1709788683.83 total 1891.25 last 25.74) - open/close 49137 (time 1709788693.84 total 1901.26 last 25.27) - open/close 49397 (time 1709788703.85 total 1911.27 last 25.97) - open/close 49652 (time 1709788713.87 total 1921.29 last 25.46) - open/close 49914 (time 1709788723.88 total 1931.30 last 26.17) total: 50000 open/close in 1934.62 seconds: 25.84 ops/second - unlinked 0 (time 1709788728 ; total 0 ; last 0) - unlinked 10000 (time 1709789218 ; total 490 ; last 490) - unlinked 20000 (time 1709789608 ; total 880 ; last 390) - unlinked 30000 (time 1709789876 ; total 1148 ; last 268) - unlinked 40000 (time 1709790006 ; total 1278 ; last 130) total: 50000 unlinks in 1317 seconds: 37.965073 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 PASS 101a (3287s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg251-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:40:59 (1709790059) start mds service on oleg251-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b67b5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b67b5800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 PASS 101b (82s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3825 sec ========== 00:42:40 (1709790160) === conf-sanity: start cleanup 00:42:41 (1709790161) === === conf-sanity: finish cleanup 00:42:41 (1709790161) ===