-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:41:12 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg255-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:42:30 (1709786550) start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h42m52s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa423800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800aa423800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:43:36.000000000 -0500 Modify: 2024-03-06 23:43:36.000000000 -0500 Change: 2024-03-06 23:43:36.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 64 lines, 64 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server PASS 35b (173s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:45:26 (1709786726) start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server 000000 2 000008 oleg255-server: 8+0 records in oleg255-server: 8+0 records out oleg255-server: 4096 bytes (4.1 kB) copied, 0.000684665 s, 6.0 MB/s start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server stop mds service on oleg255-server unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 PASS 38 (176s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg255-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg255-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg255-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg255-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:25 (1709786905) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg255-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg255-server: oleg255-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg255-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server PASS 51 (247s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:37 (1709787157) start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 200 (time 1709787202.44 total 10.00 last 19.99) - open/close 445 (time 1709787212.95 total 20.52 last 23.31) - open/close 592 (time 1709787223.97 total 31.54 last 13.34) - open/close 777 (time 1709787234.74 total 42.31 last 17.17) - open/close 999 (time 1709787245.15 total 52.72 last 21.34) - open/close 1261 (time 1709787255.16 total 62.73 last 26.17) - open/close 1466 (time 1709787265.33 total 72.90 last 20.16) - open/close 1645 (time 1709787276.26 total 83.83 last 16.37) - open/close 1748 (time 1709787286.86 total 94.43 last 9.72) - open/close 1937 (time 1709787297.00 total 104.57 last 18.64) - open/close 2188 (time 1709787307.01 total 114.58 last 25.08) - open/close 2446 (time 1709787317.54 total 125.11 last 24.51) - open/close 2533 (time 1709787327.98 total 135.55 last 8.33) - open/close 2653 (time 1709787338.00 total 145.57 last 11.97) - open/close 2846 (time 1709787348.83 total 156.40 last 17.83) - open/close 3034 (time 1709787359.44 total 167.01 last 17.72) - open/close 3206 (time 1709787370.13 total 177.70 last 16.09) - open/close 3412 (time 1709787380.98 total 188.55 last 18.98) - open/close 3576 (time 1709787391.00 total 198.57 last 16.37) - open/close 3811 (time 1709787401.00 total 208.57 last 23.50) - open/close 3925 (time 1709787411.49 total 219.06 last 10.87) - open/close 4050 (time 1709787421.71 total 229.28 last 12.24) - open/close 4278 (time 1709787431.73 total 239.30 last 22.75) - open/close 4365 (time 1709787449.71 total 257.28 last 4.84) - open/close 4563 (time 1709787460.52 total 268.09 last 18.31) - open/close 4687 (time 1709787470.97 total 278.54 last 11.87) - open/close 4845 (time 1709787480.98 total 288.55 last 15.78) - open/close 5020 (time 1709787491.04 total 298.61 last 17.40) - open/close 5142 (time 1709787501.31 total 308.88 last 11.88) - open/close 5314 (time 1709787511.86 total 319.43 last 16.30) - open/close 5500 (time 1709787521.96 total 329.53 last 18.42) - open/close 5637 (time 1709787531.98 total 339.55 last 13.67) - open/close 5804 (time 1709787541.99 total 349.56 last 16.69) - open/close 6040 (time 1709787552.92 total 360.49 last 21.59) - open/close 6174 (time 1709787562.94 total 370.51 last 13.37) - open/close 6415 (time 1709787572.94 total 380.51 last 24.09) - open/close 6616 (time 1709787582.96 total 390.52 last 20.08) - open/close 6763 (time 1709787593.90 total 401.47 last 13.43) - open/close 6911 (time 1709787605.01 total 412.58 last 13.32) - open/close 7118 (time 1709787615.68 total 423.25 last 19.40) - open/close 7262 (time 1709787625.70 total 433.27 last 14.37) - open/close 7535 (time 1709787636.68 total 444.24 last 24.88) - open/close 7748 (time 1709787647.70 total 455.27 last 19.32) - open/close 7856 (time 1709787658.02 total 465.59 last 10.46) - open/close 8059 (time 1709787668.07 total 475.64 last 20.19) - open/close 8274 (time 1709787678.67 total 486.24 last 20.28) - open/close 8465 (time 1709787689.04 total 496.60 last 18.43) - open/close 8607 (time 1709787699.73 total 507.30 last 13.28) - open/close 8736 (time 1709787709.76 total 517.33 last 12.86) - open/close 8856 (time 1709787719.79 total 527.36 last 11.97) - open/close 8999 (time 1709787730.78 total 538.35 last 13.01) - open/close 9108 (time 1709787740.79 total 548.35 last 10.90) - open/close 9223 (time 1709787750.81 total 558.38 last 11.48) - open/close 9453 (time 1709787760.96 total 568.53 last 22.65) - open/close 9584 (time 1709787770.96 total 578.53 last 13.10) - open/close 9760 (time 1709787780.99 total 588.56 last 17.55) - open/close 10000 (time 1709787790.23 total 597.80 last 25.96) - open/close 10596 (time 1709787800.24 total 607.80 last 59.59) - open/close 11148 (time 1709787810.26 total 617.83 last 55.08) - open/close 11646 (time 1709787820.27 total 627.84 last 49.74) - open/close 11826 (time 1709787835.16 total 642.72 last 12.09) - open/close 12313 (time 1709787845.16 total 652.73 last 48.66) - open/close 12759 (time 1709787855.16 total 662.73 last 44.60) - open/close 13198 (time 1709787865.17 total 672.74 last 43.87) - open/close 13714 (time 1709787875.18 total 682.75 last 51.53) - open/close 14184 (time 1709787885.19 total 692.76 last 46.97) - open/close 14600 (time 1709787895.20 total 702.77 last 41.57) - open/close 15005 (time 1709787905.21 total 712.78 last 40.47) - open/close 15417 (time 1709787915.21 total 722.78 last 41.18) - open/close 15855 (time 1709787925.21 total 732.78 last 43.79) - open/close 16284 (time 1709787935.29 total 742.86 last 42.56) - open/close 16723 (time 1709787945.31 total 752.88 last 43.84) - open/close 17218 (time 1709787955.32 total 762.89 last 49.44) - open/close 17715 (time 1709787965.34 total 772.91 last 49.60) - open/close 18156 (time 1709787975.34 total 782.91 last 44.10) - open/close 18592 (time 1709787985.36 total 792.93 last 43.51) - open/close 19041 (time 1709787995.37 total 802.94 last 44.86) - open/close 19552 (time 1709788005.38 total 812.95 last 51.05) - open/close 20000 (time 1709788014.04 total 821.60 last 51.74) - open/close 20544 (time 1709788024.05 total 831.62 last 54.34) - open/close 21080 (time 1709788034.07 total 841.63 last 53.50) - open/close 21621 (time 1709788044.07 total 851.64 last 54.07) - open/close 22154 (time 1709788054.08 total 861.65 last 53.25) - open/close 22654 (time 1709788064.09 total 871.65 last 49.98) - open/close 23128 (time 1709788074.09 total 881.66 last 47.38) - open/close 23667 (time 1709788084.11 total 891.67 last 53.81) - open/close 24152 (time 1709788094.11 total 901.68 last 48.48) - open/close 24627 (time 1709788104.11 total 911.68 last 47.48) - open/close 25110 (time 1709788114.13 total 921.69 last 48.24) - open/close 25566 (time 1709788124.15 total 931.72 last 45.51) - open/close 26039 (time 1709788134.15 total 941.72 last 47.29) - open/close 26514 (time 1709788144.16 total 951.73 last 47.45) - open/close 26977 (time 1709788154.18 total 961.75 last 46.21) - open/close 27455 (time 1709788164.19 total 971.75 last 47.76) - open/close 27926 (time 1709788174.19 total 981.76 last 47.07) - open/close 28369 (time 1709788184.20 total 991.77 last 44.27) - open/close 28838 (time 1709788194.21 total 1001.78 last 46.85) - open/close 29276 (time 1709788204.21 total 1011.78 last 43.79) - open/close 29732 (time 1709788214.22 total 1021.78 last 45.59) - open/close 30000 (time 1709788220.09 total 1027.66 last 45.63) - open/close 30437 (time 1709788230.11 total 1037.68 last 43.60) - open/close 30856 (time 1709788240.11 total 1047.68 last 41.90) - open/close 31289 (time 1709788250.13 total 1057.70 last 43.23) - open/close 31722 (time 1709788260.14 total 1067.71 last 43.23) - open/close 32109 (time 1709788270.15 total 1077.72 last 38.69) - open/close 32523 (time 1709788280.15 total 1087.72 last 41.39) - open/close 32893 (time 1709788290.16 total 1097.73 last 36.95) - open/close 33288 (time 1709788300.17 total 1107.74 last 39.49) - open/close 33676 (time 1709788310.17 total 1117.74 last 38.77) - open/close 34060 (time 1709788320.19 total 1127.76 last 38.33) - open/close 34435 (time 1709788330.21 total 1137.78 last 37.44) - open/close 34808 (time 1709788340.22 total 1147.79 last 37.24) - open/close 35173 (time 1709788350.23 total 1157.80 last 36.46) - open/close 35533 (time 1709788360.24 total 1167.81 last 35.97) - open/close 35893 (time 1709788370.25 total 1177.81 last 35.99) - open/close 36241 (time 1709788380.25 total 1187.82 last 34.78) - open/close 36574 (time 1709788390.28 total 1197.85 last 33.22) - open/close 36933 (time 1709788400.29 total 1207.86 last 35.85) - open/close 37291 (time 1709788410.29 total 1217.86 last 35.79) - open/close 37647 (time 1709788420.32 total 1227.89 last 35.49) - open/close 37993 (time 1709788430.34 total 1237.91 last 34.55) - open/close 38330 (time 1709788440.35 total 1247.91 last 33.67) - open/close 38669 (time 1709788450.35 total 1257.91 last 33.90) - open/close 39011 (time 1709788460.37 total 1267.94 last 34.11) - open/close 39342 (time 1709788470.39 total 1277.95 last 33.06) - open/close 39702 (time 1709788480.39 total 1287.96 last 35.97) - open/close 40000 (time 1709788489.50 total 1297.06 last 32.74) - open/close 40336 (time 1709788499.51 total 1307.08 last 33.54) - open/close 40679 (time 1709788509.52 total 1317.09 last 34.27) - open/close 41021 (time 1709788519.53 total 1327.10 last 34.17) - open/close 41352 (time 1709788529.56 total 1337.12 last 33.02) - open/close 41685 (time 1709788539.57 total 1347.14 last 33.26) - open/close 42008 (time 1709788549.58 total 1357.14 last 32.28) - open/close 42341 (time 1709788559.59 total 1367.15 last 33.26) - open/close 42676 (time 1709788569.61 total 1377.17 last 33.43) - open/close 43006 (time 1709788579.63 total 1387.20 last 32.91) - open/close 43341 (time 1709788589.64 total 1397.21 last 33.47) - open/close 43682 (time 1709788599.66 total 1407.22 last 34.05) - open/close 44010 (time 1709788609.69 total 1417.25 last 32.70) - open/close 44343 (time 1709788619.71 total 1427.28 last 33.22) - open/close 44662 (time 1709788629.71 total 1437.28 last 31.89) - open/close 44965 (time 1709788639.73 total 1447.30 last 30.24) - open/close 45269 (time 1709788649.76 total 1457.32 last 30.33) - open/close 45569 (time 1709788659.78 total 1467.35 last 29.92) - open/close 45872 (time 1709788669.80 total 1477.36 last 30.26) - open/close 46179 (time 1709788679.82 total 1487.39 last 30.64) - open/close 46472 (time 1709788689.83 total 1497.40 last 29.25) - open/close 46770 (time 1709788699.83 total 1507.40 last 29.80) - open/close 47061 (time 1709788709.86 total 1517.43 last 29.02) - open/close 47365 (time 1709788719.89 total 1527.46 last 30.32) - open/close 47662 (time 1709788729.89 total 1537.46 last 29.69) - open/close 47961 (time 1709788739.90 total 1547.47 last 29.86) - open/close 48246 (time 1709788749.94 total 1557.51 last 28.40) - open/close 48524 (time 1709788759.97 total 1567.54 last 27.72) - open/close 48810 (time 1709788769.97 total 1577.54 last 28.58) - open/close 49086 (time 1709788779.98 total 1587.54 last 27.60) - open/close 49362 (time 1709788790.01 total 1597.58 last 27.51) - open/close 49652 (time 1709788800.02 total 1607.58 last 28.98) - open/close 49928 (time 1709788810.04 total 1617.61 last 27.53) total: 50000 open/close in 1620.19 seconds: 30.86 ops/second - unlinked 0 (time 1709788813 ; total 0 ; last 0) - unlinked 10000 (time 1709789270 ; total 457 ; last 457) - unlinked 20000 (time 1709789659 ; total 846 ; last 389) - unlinked 30000 (time 1709789925 ; total 1112 ; last 266) - unlinked 40000 (time 1709790039 ; total 1226 ; last 114) total: 50000 unlinks in 1267 seconds: 39.463299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 PASS 101a (2938s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg255-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg255-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg255-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg255-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:41:36 (1709790096) start mds service on oleg255-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec oleg255-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5847800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5847800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 PASS 101b (89s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg255-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg255-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3733 sec ========== 00:43:26 (1709790206) === conf-sanity: start cleanup 00:43:27 (1709790207) === === conf-sanity: finish cleanup 00:43:27 (1709790207) ===