-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:35:48 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg143-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg143-server oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:37:03 (1709786223) start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h37m21s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b5871800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b5871800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:38:04.000000000 -0500 Modify: 2024-03-06 23:38:04.000000000 -0500 Change: 2024-03-06 23:38:04.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 63 lines, 63 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server PASS 35b (142s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:39:28 (1709786368) start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server 000000 2 000008 oleg143-server: 8+0 records in oleg143-server: 8+0 records out oleg143-server: 4096 bytes (4.1 kB) copied, 0.000823305 s, 5.0 MB/s start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server stop mds service on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 PASS 38 (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg143-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg143-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg143-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg143-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:40:58 (1709786458) Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg143-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec stop ost2 service on oleg143-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg143-server umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local modules unloaded. oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg143-server: oleg143-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg143-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg143-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg143-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 oleg143-server: oleg143-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg143-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server PASS 51 (181s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:44:02 (1709786642) start mds service on oleg143-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock oleg143-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1709786685.41 total 10.93 last 16.56) - open/close 395 (time 1709786695.41 total 20.94 last 21.39) - open/close 614 (time 1709786706.20 total 31.73 last 20.30) - open/close 828 (time 1709786718.31 total 43.84 last 17.67) - open/close 932 (time 1709786729.19 total 54.71 last 9.56) - open/close 1135 (time 1709786739.42 total 64.94 last 19.84) - open/close 1387 (time 1709786749.43 total 74.95 last 25.17) - open/close 1660 (time 1709786759.43 total 84.96 last 27.28) - open/close 1855 (time 1709786770.31 total 95.83 last 17.93) - open/close 2108 (time 1709786780.40 total 105.92 last 25.08) - open/close 2347 (time 1709786790.40 total 115.92 last 23.89) - open/close 2461 (time 1709786801.21 total 126.73 last 10.55) - open/close 2755 (time 1709786811.93 total 137.46 last 27.42) - open/close 2983 (time 1709786821.94 total 147.46 last 22.78) - open/close 3132 (time 1709786832.14 total 157.66 last 14.61) - open/close 3256 (time 1709786843.07 total 168.60 last 11.34) - open/close 3406 (time 1709786853.10 total 178.62 last 14.96) - open/close 3534 (time 1709786863.50 total 189.03 last 12.30) - open/close 3763 (time 1709786873.51 total 199.04 last 22.88) - open/close 4009 (time 1709786883.52 total 209.05 last 24.58) - open/close 4183 (time 1709786893.53 total 219.06 last 17.38) - open/close 4538 (time 1709786904.17 total 229.69 last 33.38) - open/close 4769 (time 1709786914.17 total 239.69 last 23.10) - open/close 5001 (time 1709786924.18 total 249.70 last 23.18) - open/close 5121 (time 1709786935.19 total 260.72 last 10.90) - open/close 5232 (time 1709786945.23 total 270.76 last 11.05) - open/close 5357 (time 1709786955.62 total 281.15 last 12.03) - open/close 5583 (time 1709786966.19 total 291.71 last 21.40) - open/close 5818 (time 1709786976.57 total 302.09 last 22.64) - open/close 6086 (time 1709786986.58 total 312.10 last 26.77) - open/close 6356 (time 1709786996.86 total 322.38 last 26.27) - open/close 6649 (time 1709787007.22 total 332.75 last 28.26) - open/close 6931 (time 1709787017.23 total 342.76 last 28.17) - open/close 7160 (time 1709787027.74 total 353.26 last 21.80) - open/close 7405 (time 1709787037.74 total 363.27 last 24.49) - open/close 7591 (time 1709787048.22 total 373.74 last 17.75) - open/close 7672 (time 1709787058.72 total 384.25 last 7.71) - open/close 7797 (time 1709787068.73 total 394.26 last 12.49) - open/close 7983 (time 1709787081.69 total 407.21 last 14.36) - open/close 8187 (time 1709787091.99 total 417.51 last 19.80) - open/close 8339 (time 1709787102.49 total 428.02 last 14.47) - open/close 8540 (time 1709787113.37 total 438.89 last 18.48) - open/close 8694 (time 1709787123.38 total 448.91 last 15.38) - open/close 8899 (time 1709787133.40 total 458.93 last 20.46) - open/close 9013 (time 1709787143.40 total 468.93 last 11.40) - open/close 9179 (time 1709787153.44 total 478.96 last 16.54) - open/close 9278 (time 1709787163.92 total 489.44 last 9.45) - open/close 9391 (time 1709787174.62 total 500.14 last 10.56) - open/close 9767 (time 1709787184.63 total 510.15 last 37.58) - open/close 10000 (time 1709787189.51 total 515.03 last 47.72) - open/close 10436 (time 1709787199.51 total 525.04 last 43.58) - open/close 10861 (time 1709787209.52 total 535.05 last 42.45) - open/close 11277 (time 1709787219.53 total 545.05 last 41.59) - open/close 11430 (time 1709787234.80 total 560.33 last 10.02) - open/close 11703 (time 1709787252.12 total 577.64 last 15.76) - open/close 12157 (time 1709787262.14 total 587.67 last 45.29) - open/close 12341 (time 1709787277.18 total 602.70 last 12.24) - open/close 12795 (time 1709787287.20 total 612.73 last 45.29) - open/close 13238 (time 1709787297.21 total 622.73 last 44.28) - open/close 13657 (time 1709787307.22 total 632.75 last 41.83) - open/close 14044 (time 1709787317.22 total 642.75 last 38.69) - open/close 14424 (time 1709787327.23 total 652.76 last 37.98) - open/close 14802 (time 1709787337.24 total 662.76 last 37.77) - open/close 15103 (time 1709787347.27 total 672.80 last 29.99) - open/close 15484 (time 1709787357.27 total 682.80 last 38.10) - open/close 15840 (time 1709787367.30 total 692.82 last 35.52) - open/close 16201 (time 1709787377.30 total 702.82 last 36.10) - open/close 16487 (time 1709787387.30 total 712.82 last 28.60) - open/close 16812 (time 1709787397.34 total 722.87 last 32.35) - open/close 17117 (time 1709787407.36 total 732.88 last 30.47) - open/close 17451 (time 1709787417.37 total 742.90 last 33.34) - open/close 17810 (time 1709787427.39 total 752.92 last 35.83) - open/close 18128 (time 1709787437.44 total 762.97 last 31.64) - open/close 18416 (time 1709787447.46 total 772.99 last 28.74) - open/close 18734 (time 1709787457.47 total 782.99 last 31.79) - open/close 19026 (time 1709787467.50 total 793.03 last 29.09) - open/close 19301 (time 1709787477.56 total 803.08 last 27.36) - open/close 19591 (time 1709787487.57 total 813.09 last 28.97) - open/close 19907 (time 1709787497.60 total 823.13 last 31.50) - open/close 20000 (time 1709787501.10 total 826.62 last 26.60) - open/close 20290 (time 1709787511.13 total 836.66 last 28.90) - open/close 20577 (time 1709787521.17 total 846.70 last 28.58) - open/close 20887 (time 1709787531.21 total 856.74 last 30.89) - open/close 21170 (time 1709787541.22 total 866.75 last 28.26) - open/close 21425 (time 1709787551.24 total 876.76 last 25.47) - open/close 21648 (time 1709787561.26 total 886.79 last 22.25) - open/close 21878 (time 1709787571.28 total 896.81 last 22.95) - open/close 22145 (time 1709787581.29 total 906.82 last 26.67) - open/close 22412 (time 1709787591.32 total 916.85 last 26.61) - open/close 22668 (time 1709787601.35 total 926.87 last 25.54) - open/close 22967 (time 1709787611.38 total 936.90 last 29.81) - open/close 23225 (time 1709787621.43 total 946.96 last 25.66) - open/close 23463 (time 1709787631.43 total 956.96 last 23.80) - open/close 23749 (time 1709787641.44 total 966.97 last 28.56) - open/close 24010 (time 1709787651.45 total 976.97 last 26.10) - open/close 24260 (time 1709787661.49 total 987.01 last 24.90) - open/close 24504 (time 1709787671.49 total 997.02 last 24.39) - open/close 24792 (time 1709787681.50 total 1007.02 last 28.77) - open/close 25040 (time 1709787691.50 total 1017.03 last 24.80) - open/close 25309 (time 1709787701.52 total 1027.05 last 26.84) - open/close 25589 (time 1709787711.55 total 1037.07 last 27.92) - open/close 25875 (time 1709787721.57 total 1047.10 last 28.53) - open/close 26178 (time 1709787731.60 total 1057.12 last 30.22) - open/close 26444 (time 1709787741.62 total 1067.15 last 26.53) - open/close 26693 (time 1709787751.67 total 1077.19 last 24.80) - open/close 26920 (time 1709787761.67 total 1087.20 last 22.69) - open/close 27170 (time 1709787771.68 total 1097.21 last 24.97) - open/close 27424 (time 1709787781.72 total 1107.24 last 25.32) - open/close 27660 (time 1709787791.72 total 1117.25 last 23.58) - open/close 27903 (time 1709787801.80 total 1127.33 last 24.11) - open/close 28166 (time 1709787811.83 total 1137.36 last 26.22) - open/close 28434 (time 1709787821.88 total 1147.40 last 26.69) - open/close 28650 (time 1709787831.89 total 1157.42 last 21.57) - open/close 28867 (time 1709787841.90 total 1167.43 last 21.67) - open/close 29095 (time 1709787851.92 total 1177.44 last 22.77) - open/close 29344 (time 1709787861.92 total 1187.45 last 24.90) - open/close 29591 (time 1709787871.92 total 1197.45 last 24.69) - open/close 29825 (time 1709787881.93 total 1207.46 last 23.38) - open/close 30000 (time 1709787889.57 total 1215.09 last 22.91) - open/close 30231 (time 1709787899.59 total 1225.12 last 23.04) - open/close 30463 (time 1709787909.62 total 1235.14 last 23.15) - open/close 30696 (time 1709787919.65 total 1245.17 last 23.22) - open/close 30913 (time 1709787929.69 total 1255.22 last 21.60) - open/close 31135 (time 1709787939.72 total 1265.24 last 22.15) - open/close 31361 (time 1709787949.72 total 1275.25 last 22.60) - open/close 31635 (time 1709787959.73 total 1285.25 last 27.38) - open/close 31877 (time 1709787969.76 total 1295.29 last 24.11) - open/close 32137 (time 1709787979.77 total 1305.29 last 25.98) - open/close 32413 (time 1709787989.80 total 1315.32 last 27.52) - open/close 32685 (time 1709787999.82 total 1325.35 last 27.13) - open/close 32926 (time 1709788009.84 total 1335.36 last 24.07) - open/close 33182 (time 1709788019.86 total 1345.38 last 25.55) - open/close 33433 (time 1709788029.87 total 1355.40 last 25.06) - open/close 33688 (time 1709788039.88 total 1365.41 last 25.47) - open/close 33935 (time 1709788049.91 total 1375.44 last 24.62) - open/close 34186 (time 1709788059.92 total 1385.44 last 25.10) - open/close 34438 (time 1709788069.93 total 1395.46 last 25.16) - open/close 34705 (time 1709788079.93 total 1405.46 last 26.70) - open/close 34959 (time 1709788089.96 total 1415.48 last 25.34) - open/close 35208 (time 1709788099.97 total 1425.50 last 24.86) - open/close 35444 (time 1709788109.99 total 1435.51 last 23.56) - open/close 35687 (time 1709788120.01 total 1445.54 last 24.24) - open/close 35921 (time 1709788130.02 total 1455.55 last 23.38) - open/close 36177 (time 1709788140.06 total 1465.59 last 25.49) - open/close 36413 (time 1709788150.09 total 1475.62 last 23.54) - open/close 36651 (time 1709788160.12 total 1485.64 last 23.73) - open/close 36887 (time 1709788170.14 total 1495.67 last 23.54) - open/close 37122 (time 1709788180.15 total 1505.67 last 23.49) - open/close 37356 (time 1709788190.17 total 1515.69 last 23.35) - open/close 37592 (time 1709788200.17 total 1525.69 last 23.59) - open/close 37846 (time 1709788210.17 total 1535.70 last 25.39) - open/close 38096 (time 1709788220.17 total 1545.70 last 24.99) - open/close 38343 (time 1709788230.18 total 1555.71 last 24.69) - open/close 38588 (time 1709788240.19 total 1565.72 last 24.47) - open/close 38826 (time 1709788250.22 total 1575.75 last 23.73) - open/close 39031 (time 1709788260.24 total 1585.76 last 20.46) - open/close 39260 (time 1709788270.28 total 1595.81 last 22.80) - open/close 39426 (time 1709788288.49 total 1614.02 last 9.12) - open/close 39664 (time 1709788298.53 total 1624.05 last 23.71) - open/close 39897 (time 1709788308.55 total 1634.08 last 23.25) - open/close 40000 (time 1709788313.13 total 1638.65 last 22.51) - open/close 40233 (time 1709788323.16 total 1648.68 last 23.23) - open/close 40462 (time 1709788333.19 total 1658.72 last 22.82) - open/close 40686 (time 1709788343.20 total 1668.72 last 22.39) - open/close 40914 (time 1709788353.20 total 1678.73 last 22.79) - open/close 41148 (time 1709788363.22 total 1688.75 last 23.36) - open/close 41376 (time 1709788373.26 total 1698.79 last 22.71) - open/close 41608 (time 1709788383.28 total 1708.80 last 23.16) - open/close 41833 (time 1709788393.28 total 1718.81 last 22.49) - open/close 42060 (time 1709788403.28 total 1728.81 last 22.70) - open/close 42292 (time 1709788413.29 total 1738.82 last 23.19) - open/close 42506 (time 1709788423.33 total 1748.85 last 21.32) - open/close 42723 (time 1709788433.33 total 1758.86 last 21.68) - open/close 42942 (time 1709788443.36 total 1768.88 last 21.85) - open/close 43153 (time 1709788453.38 total 1778.91 last 21.05) - open/close 43364 (time 1709788463.41 total 1788.94 last 21.04) - open/close 43578 (time 1709788473.42 total 1798.95 last 21.37) - open/close 43787 (time 1709788483.47 total 1808.99 last 20.81) - open/close 44003 (time 1709788493.48 total 1819.01 last 21.57) - open/close 44226 (time 1709788503.49 total 1829.02 last 22.28) - open/close 44448 (time 1709788513.51 total 1839.04 last 22.15) - open/close 44672 (time 1709788523.52 total 1849.05 last 22.38) - open/close 44890 (time 1709788533.53 total 1859.06 last 21.77) - open/close 45113 (time 1709788543.56 total 1869.08 last 22.25) - open/close 45339 (time 1709788553.60 total 1879.13 last 22.50) - open/close 45562 (time 1709788563.64 total 1889.16 last 22.22) - open/close 45788 (time 1709788573.68 total 1899.20 last 22.51) - open/close 45995 (time 1709788583.70 total 1909.23 last 20.64) - open/close 46208 (time 1709788593.73 total 1919.25 last 21.25) - open/close 46420 (time 1709788603.75 total 1929.27 last 21.16) - open/close 46638 (time 1709788613.75 total 1939.28 last 21.79) - open/close 46868 (time 1709788623.77 total 1949.29 last 22.96) - open/close 47095 (time 1709788633.82 total 1959.35 last 22.58) - open/close 47309 (time 1709788643.86 total 1969.38 last 21.33) - open/close 47516 (time 1709788653.87 total 1979.40 last 20.67) - open/close 47724 (time 1709788663.87 total 1989.40 last 20.80) - open/close 47927 (time 1709788673.91 total 1999.44 last 20.21) - open/close 48109 (time 1709788683.97 total 2009.49 last 18.10) - open/close 48297 (time 1709788693.98 total 2019.50 last 18.78) - open/close 48496 (time 1709788704.02 total 2029.54 last 19.82) - open/close 48693 (time 1709788714.06 total 2039.59 last 19.61) - open/close 48886 (time 1709788724.08 total 2049.61 last 19.26) - open/close 49079 (time 1709788734.08 total 2059.61 last 19.30) - open/close 49271 (time 1709788744.10 total 2069.63 last 19.16) - open/close 49452 (time 1709788754.14 total 2079.66 last 18.04) - open/close 49632 (time 1709788764.16 total 2089.68 last 17.96) - open/close 49819 (time 1709788774.17 total 2099.70 last 18.68) total: 50000 open/close in 2109.01 seconds: 23.71 ops/second