-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:37:41 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg440-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg440-server oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:38:50 (1709786330) start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h39m08s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800a9fba800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800a9fba800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:39:51.000000000 -0500 Modify: 2024-03-06 23:39:51.000000000 -0500 Change: 2024-03-06 23:39:51.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 66 lines, 66 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server PASS 35b (135s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:07 (1709786467) start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server 000000 2 000008 oleg440-server: 8+0 records in oleg440-server: 8+0 records out oleg440-server: 4096 bytes (4.1 kB) copied, 0.000443225 s, 9.2 MB/s start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server stop mds service on oleg440-server unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 PASS 38 (101s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg440-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg440-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg440-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg440-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:50 (1709786570) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg440-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg440-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg440-server umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg440-server: oleg440-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg440-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server PASS 51 (201s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:13 (1709786773) start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 232 (time 1709786814.30 total 10.01 last 23.19) - open/close 539 (time 1709786825.12 total 20.83 last 28.37) - open/close 832 (time 1709786836.13 total 31.83 last 26.63) - open/close 1189 (time 1709786846.93 total 42.63 last 33.05) - open/close 1602 (time 1709786857.60 total 53.31 last 38.69) - open/close 1881 (time 1709786868.16 total 63.86 last 26.43) - open/close 2124 (time 1709786878.39 total 74.09 last 23.75) - open/close 2397 (time 1709786888.40 total 84.11 last 27.27) - open/close 2586 (time 1709786898.41 total 94.11 last 18.88) - open/close 2769 (time 1709786908.41 total 104.12 last 18.30) - open/close 3045 (time 1709786919.11 total 114.81 last 25.80) - open/close 3274 (time 1709786931.53 total 127.23 last 18.44) - open/close 3424 (time 1709786941.54 total 137.24 last 14.99) - open/close 3694 (time 1709786952.35 total 148.05 last 24.97) - open/close 3783 (time 1709786962.78 total 158.48 last 8.53) - open/close 4089 (time 1709786973.46 total 169.16 last 28.65) - open/close 4308 (time 1709786984.08 total 179.78 last 20.62) - open/close 4656 (time 1709786994.69 total 190.39 last 32.81) - open/close 4828 (time 1709787005.22 total 200.92 last 16.33) - open/close 4959 (time 1709787016.20 total 211.90 last 11.92) - open/close 5129 (time 1709787026.20 total 221.90 last 17.00) - open/close 5390 (time 1709787036.21 total 231.91 last 26.08) - open/close 5561 (time 1709787046.22 total 241.92 last 17.09) - open/close 5752 (time 1709787056.22 total 251.92 last 19.09) - open/close 5939 (time 1709787067.27 total 262.97 last 16.92) - open/close 6162 (time 1709787077.29 total 272.99 last 22.26) - open/close 6406 (time 1709787087.29 total 282.99 last 24.39) - open/close 6555 (time 1709787097.69 total 293.39 last 14.34) - open/close 6752 (time 1709787107.99 total 303.69 last 19.12) - open/close 6950 (time 1709787118.87 total 314.57 last 18.20) - open/close 7145 (time 1709787128.88 total 324.58 last 19.49) - open/close 7328 (time 1709787138.92 total 334.62 last 18.22) - open/close 7639 (time 1709787149.68 total 345.38 last 28.90) - open/close 7872 (time 1709787159.69 total 355.39 last 23.28) - open/close 8176 (time 1709787170.04 total 365.74 last 29.38) - open/close 8346 (time 1709787180.83 total 376.54 last 15.74) - open/close 8609 (time 1709787192.96 total 388.66 last 21.69) - open/close 8723 (time 1709787203.28 total 398.99 last 11.04) - open/close 8958 (time 1709787214.34 total 410.04 last 21.25) - open/close 9196 (time 1709787225.14 total 420.85 last 22.03) - open/close 9385 (time 1709787235.29 total 430.99 last 18.63) - open/close 9560 (time 1709787245.29 total 441.00 last 17.49) - open/close 9728 (time 1709787263.76 total 459.46 last 9.10) - open/close 9900 (time 1709787273.76 total 469.47 last 17.20) - open/close 10461 (time 1709787283.77 total 479.47 last 56.08) - open/close 11006 (time 1709787293.77 total 489.48 last 54.47) - open/close 11574 (time 1709787303.79 total 499.49 last 56.71) - open/close 12123 (time 1709787313.84 total 509.54 last 54.65) - open/close 12589 (time 1709787332.89 total 528.59 last 24.46) - open/close 13152 (time 1709787342.89 total 538.60 last 56.26) - open/close 13648 (time 1709787352.90 total 548.61 last 49.55) - open/close 14154 (time 1709787362.91 total 558.61 last 50.59) - open/close 14607 (time 1709787372.95 total 568.65 last 45.10) - open/close 15106 (time 1709787382.96 total 578.66 last 49.86) - open/close 15539 (time 1709787392.97 total 588.67 last 43.26) - open/close 16030 (time 1709787402.97 total 598.67 last 49.08) - open/close 16439 (time 1709787412.99 total 608.69 last 40.82) - open/close 16812 (time 1709787423.00 total 618.70 last 37.28) - open/close 17197 (time 1709787433.03 total 628.73 last 38.39) - open/close 17588 (time 1709787443.06 total 638.76 last 38.96) - open/close 17949 (time 1709787453.06 total 648.77 last 36.10) - open/close 18300 (time 1709787463.08 total 658.78 last 35.04) - open/close 18664 (time 1709787473.09 total 668.79 last 36.37) - open/close 19019 (time 1709787483.10 total 678.80 last 35.47) - open/close 19376 (time 1709787493.17 total 688.87 last 35.44) - open/close 19713 (time 1709787503.18 total 698.88 last 33.68) - open/close 20000 (time 1709787512.69 total 708.39 last 30.18) - open/close 20336 (time 1709787522.70 total 718.40 last 33.56) - open/close 20697 (time 1709787532.79 total 728.49 last 35.79) - open/close 21056 (time 1709787542.79 total 738.49 last 35.88) - open/close 21433 (time 1709787552.81 total 748.52 last 37.62) - open/close 21801 (time 1709787562.82 total 758.52 last 36.79) - open/close 22216 (time 1709787572.83 total 768.53 last 41.44) - open/close 22577 (time 1709787582.85 total 778.55 last 36.03) - open/close 23013 (time 1709787592.86 total 788.56 last 43.55) - open/close 23435 (time 1709787602.89 total 798.59 last 42.10) - open/close 23843 (time 1709787612.91 total 808.61 last 40.72) - open/close 24236 (time 1709787622.93 total 818.64 last 39.19) - open/close 24272 (time 1709787634.94 total 830.65 last 3.00) - open/close 24653 (time 1709787644.95 total 840.66 last 38.06) - open/close 25057 (time 1709787654.96 total 850.66 last 40.39) - open/close 25451 (time 1709787664.97 total 860.67 last 39.34) - open/close 25844 (time 1709787675.01 total 870.72 last 39.14) - open/close 26202 (time 1709787685.03 total 880.73 last 35.76) - open/close 26608 (time 1709787695.03 total 890.73 last 40.57) - open/close 26959 (time 1709787715.80 total 911.51 last 16.90) - open/close 27275 (time 1709787725.83 total 921.53 last 31.53) - open/close 27651 (time 1709787735.84 total 931.54 last 37.55) - open/close 28059 (time 1709787745.86 total 941.56 last 40.74) - open/close 28410 (time 1709787755.86 total 951.57 last 35.07) - open/close 28706 (time 1709787765.88 total 961.58 last 29.56) - open/close 28981 (time 1709787775.88 total 971.58 last 27.50) - open/close 29259 (time 1709787785.88 total 981.58 last 27.80) - open/close 29546 (time 1709787795.93 total 991.64 last 28.55) - open/close 29868 (time 1709787805.95 total 1001.65 last 32.15) - open/close 30000 (time 1709787809.65 total 1005.35 last 35.70) - open/close 30311 (time 1709787819.66 total 1015.37 last 31.05) - open/close 30608 (time 1709787829.67 total 1025.37 last 29.69) - open/close 30949 (time 1709787839.68 total 1035.38 last 34.06) - open/close 31314 (time 1709787849.70 total 1045.40 last 36.42) - open/close 31703 (time 1709787859.71 total 1055.42 last 38.86) - open/close 32069 (time 1709787869.74 total 1065.44 last 36.51) - open/close 32449 (time 1709787879.76 total 1075.46 last 37.91) - open/close 32836 (time 1709787889.77 total 1085.48 last 38.66) - open/close 33210 (time 1709787899.79 total 1095.49 last 37.34) - open/close 33547 (time 1709787909.80 total 1105.50 last 33.68) - open/close 33904 (time 1709787919.83 total 1115.53 last 35.60) - open/close 34241 (time 1709787929.83 total 1125.53 last 33.69) - open/close 34602 (time 1709787939.86 total 1135.56 last 36.00) - open/close 34959 (time 1709787949.87 total 1145.57 last 35.65) - open/close 35346 (time 1709787959.88 total 1155.58 last 38.66) - open/close 35687 (time 1709787969.90 total 1165.60 last 34.05) - open/close 36017 (time 1709787979.90 total 1175.61 last 32.97) - open/close 36380 (time 1709787989.90 total 1185.61 last 36.30) - open/close 36729 (time 1709787999.91 total 1195.62 last 34.87) - open/close 36984 (time 1709788009.92 total 1205.62 last 25.48) - open/close 37230 (time 1709788019.94 total 1215.65 last 24.55) - open/close 37555 (time 1709788029.95 total 1225.65 last 32.48) - open/close 37903 (time 1709788039.97 total 1235.67 last 34.73) - open/close 38242 (time 1709788049.99 total 1245.69 last 33.84) - open/close 38560 (time 1709788059.99 total 1255.69 last 31.79) - open/close 38887 (time 1709788070.01 total 1265.71 last 32.65) - open/close 39213 (time 1709788080.04 total 1275.74 last 32.48) - open/close 39552 (time 1709788090.06 total 1285.76 last 33.83) - open/close 39873 (time 1709788100.09 total 1295.79 last 32.02) - open/close 40000 (time 1709788104.23 total 1299.93 last 30.67) - open/close 40303 (time 1709788114.24 total 1309.94 last 30.26) - open/close 40639 (time 1709788124.26 total 1319.96 last 33.53) - open/close 40924 (time 1709788134.28 total 1329.98 last 28.44) - open/close 41213 (time 1709788144.28 total 1339.98 last 28.89) - open/close 41494 (time 1709788154.31 total 1350.01 last 28.01) - open/close 41731 (time 1709788164.34 total 1360.04 last 23.64) - open/close 41982 (time 1709788174.36 total 1370.06 last 25.05) - open/close 42293 (time 1709788184.38 total 1380.08 last 31.04) - open/close 42589 (time 1709788194.38 total 1390.08 last 29.60) - open/close 42856 (time 1709788204.39 total 1400.09 last 26.69) - open/close 43156 (time 1709788214.41 total 1410.12 last 29.91) - open/close 43451 (time 1709788224.43 total 1420.13 last 29.46) - open/close 43747 (time 1709788234.44 total 1430.14 last 29.58) - open/close 44008 (time 1709788244.44 total 1440.14 last 26.10) - open/close 44304 (time 1709788254.44 total 1450.14 last 29.59) - open/close 44616 (time 1709788264.47 total 1460.17 last 31.09) - open/close 44907 (time 1709788274.48 total 1470.18 last 29.07) - open/close 45170 (time 1709788284.52 total 1480.22 last 26.20) - open/close 45466 (time 1709788294.55 total 1490.25 last 29.53) - open/close 45747 (time 1709788304.58 total 1500.28 last 28.02) - open/close 46021 (time 1709788314.59 total 1510.30 last 27.35) - open/close 46125 (time 1709788329.68 total 1525.38 last 6.89) - open/close 46415 (time 1709788339.71 total 1535.41 last 28.93) - open/close 46694 (time 1709788349.71 total 1545.41 last 27.89) - open/close 46953 (time 1709788359.73 total 1555.44 last 25.84) - open/close 47244 (time 1709788369.73 total 1565.44 last 29.10) - open/close 47528 (time 1709788379.74 total 1575.44 last 28.38) - open/close 47813 (time 1709788389.76 total 1585.46 last 28.45) - open/close 48104 (time 1709788399.78 total 1595.48 last 29.04) - open/close 48380 (time 1709788409.78 total 1605.49 last 27.60) - open/close 48625 (time 1709788419.81 total 1615.51 last 24.44) - open/close 48876 (time 1709788429.84 total 1625.54 last 25.03) - open/close 49127 (time 1709788439.84 total 1635.55 last 25.08) - open/close 49371 (time 1709788449.87 total 1645.57 last 24.35) - open/close 49621 (time 1709788459.88 total 1655.58 last 24.97) - open/close 49883 (time 1709788469.89 total 1665.59 last 26.18) total: 50000 open/close in 1669.95 seconds: 29.94 ops/second - unlinked 0 (time 1709788475 ; total 0 ; last 0) - unlinked 10000 (time 1709788911 ; total 436 ; last 436) - unlinked 20000 (time 1709789289 ; total 814 ; last 378) - unlinked 30000 (time 1709789554 ; total 1079 ; last 265) - unlinked 40000 (time 1709789698 ; total 1223 ; last 144) total: 50000 unlinks in 1277 seconds: 39.154266 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 PASS 101a (3000s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg440-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg440-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg440-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg440-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:36:15 (1709789775) start mds service on oleg440-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6229800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6229800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 PASS 101b (99s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg440-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg440-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3639 sec ========== 00:38:20 (1709789900) === conf-sanity: start cleanup 00:38:21 (1709789901) === === conf-sanity: finish cleanup 00:38:21 (1709789901) ===