-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:46:01 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg434-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg434-server oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:47:35 (1709786855) start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h48m09s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b5841800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b5841800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:48:58.000000000 -0500 Modify: 2024-03-06 23:48:58.000000000 -0500 Change: 2024-03-06 23:48:58.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 69 lines, 69 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server PASS 35b (211s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:51:09 (1709787069) start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server 000000 2 000008 oleg434-server: 8+0 records in oleg434-server: 8+0 records out oleg434-server: 4096 bytes (4.1 kB) copied, 0.00065705 s, 6.2 MB/s start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server stop mds service on oleg434-server unloading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 PASS 38 (153s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg434-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg434-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg434-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg434-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:53:46 (1709787226) Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg434-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg434-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg434-server umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg434-server: oleg434-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg434-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server PASS 51 (262s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:58:13 (1709787493) start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1709787545.10 total 10.00 last 20.20) - open/close 290 (time 1709787555.30 total 20.20 last 8.63) - open/close 643 (time 1709787565.30 total 30.20 last 35.28) - open/close 904 (time 1709787575.31 total 40.21 last 26.09) - open/close 1153 (time 1709787585.31 total 50.21 last 24.89) - open/close 1393 (time 1709787595.32 total 60.22 last 23.99) - open/close 1667 (time 1709787605.32 total 70.22 last 27.38) - open/close 1878 (time 1709787615.62 total 80.52 last 20.48) - open/close 2151 (time 1709787625.77 total 90.67 last 26.91) - open/close 2416 (time 1709787636.18 total 101.08 last 25.45) - open/close 2708 (time 1709787647.00 total 111.90 last 27.00) - open/close 2907 (time 1709787657.21 total 122.11 last 19.49) - open/close 3172 (time 1709787667.42 total 132.32 last 25.95) - open/close 3439 (time 1709787678.42 total 143.32 last 24.28) - open/close 3768 (time 1709787688.42 total 153.32 last 32.88) - open/close 3980 (time 1709787698.47 total 163.37 last 21.10) - open/close 4170 (time 1709787708.48 total 173.38 last 18.99) - open/close 4406 (time 1709787719.34 total 184.24 last 21.73) - open/close 4673 (time 1709787730.06 total 194.96 last 24.90) - open/close 4913 (time 1709787740.23 total 205.13 last 23.60) - open/close 5263 (time 1709787750.53 total 215.43 last 33.98) - open/close 5393 (time 1709787761.36 total 226.26 last 12.00) - open/close 5536 (time 1709787771.78 total 236.68 last 13.72) - open/close 5765 (time 1709787781.91 total 246.81 last 22.60) - open/close 6000 (time 1709787792.93 total 257.83 last 21.33) - open/close 6170 (time 1709787803.06 total 267.96 last 16.77) - open/close 6444 (time 1709787813.08 total 277.98 last 27.36) - open/close 6573 (time 1709787823.09 total 287.99 last 12.89) - open/close 6815 (time 1709787833.10 total 298.00 last 24.18) - open/close 6955 (time 1709787843.12 total 308.02 last 13.96) - open/close 7114 (time 1709787853.45 total 318.35 last 15.40) - open/close 7271 (time 1709787863.45 total 328.35 last 15.69) - open/close 7525 (time 1709787875.74 total 340.64 last 20.67) - open/close 7866 (time 1709787885.96 total 350.86 last 33.35) - open/close 8082 (time 1709787896.72 total 361.63 last 20.07) - open/close 8433 (time 1709787906.73 total 371.63 last 35.08) - open/close 8788 (time 1709787916.85 total 381.75 last 35.08) - open/close 8925 (time 1709787926.98 total 391.89 last 13.52) - open/close 9095 (time 1709787937.56 total 402.46 last 16.07) - open/close 9441 (time 1709787948.57 total 413.47 last 31.44) - open/close 9567 (time 1709787959.39 total 424.29 last 11.64) - open/close 9781 (time 1709787969.61 total 434.51 last 20.94) - open/close 10000 (time 1709787975.86 total 440.76 last 35.02) - open/close 10616 (time 1709787985.87 total 450.77 last 61.55) - open/close 11303 (time 1709787995.88 total 460.78 last 68.62) - open/close 11995 (time 1709788005.88 total 470.78 last 69.19) - open/close 12646 (time 1709788015.90 total 480.80 last 65.02) - open/close 13269 (time 1709788025.91 total 490.81 last 62.22) - open/close 13865 (time 1709788035.91 total 500.81 last 59.56) - open/close 14420 (time 1709788045.91 total 510.81 last 55.50) - open/close 14960 (time 1709788055.92 total 520.83 last 53.94) - open/close 15501 (time 1709788065.94 total 530.84 last 54.02) - open/close 16040 (time 1709788075.95 total 540.85 last 53.87) - open/close 16550 (time 1709788085.96 total 550.86 last 50.96) - open/close 17078 (time 1709788095.96 total 560.86 last 52.78) - open/close 17591 (time 1709788105.97 total 570.87 last 51.22) - open/close 18093 (time 1709788115.99 total 580.89 last 50.13) - open/close 18590 (time 1709788126.00 total 590.90 last 49.62) - open/close 19020 (time 1709788136.03 total 600.93 last 42.88) - open/close 19482 (time 1709788146.05 total 610.95 last 46.10) - open/close 19912 (time 1709788156.06 total 620.96 last 42.97) - open/close 20000 (time 1709788158.09 total 623.00 last 43.21) - open/close 20392 (time 1709788168.11 total 633.01 last 39.14) - open/close 20830 (time 1709788178.13 total 643.03 last 43.71) - open/close 21265 (time 1709788188.15 total 653.05 last 43.43) - open/close 21679 (time 1709788198.16 total 663.06 last 41.33) - open/close 22083 (time 1709788208.17 total 673.07 last 40.38) - open/close 22490 (time 1709788218.19 total 683.09 last 40.61) - open/close 22887 (time 1709788228.21 total 693.11 last 39.63) - open/close 23272 (time 1709788238.24 total 703.14 last 38.39) - open/close 23646 (time 1709788248.24 total 713.14 last 37.39) - open/close 24053 (time 1709788258.25 total 723.15 last 40.66) - open/close 24451 (time 1709788268.27 total 733.17 last 39.73) - open/close 24829 (time 1709788278.28 total 743.18 last 37.74) - open/close 25229 (time 1709788288.30 total 753.20 last 39.93) - open/close 25608 (time 1709788298.30 total 763.20 last 37.89) - open/close 25979 (time 1709788308.32 total 773.22 last 37.04) - open/close 26332 (time 1709788318.32 total 783.23 last 35.29) - open/close 26622 (time 1709788337.26 total 802.16 last 15.31) - open/close 26961 (time 1709788347.28 total 812.18 last 33.83) - open/close 27298 (time 1709788357.29 total 822.19 last 33.67) - open/close 27635 (time 1709788367.32 total 832.22 last 33.60) - open/close 27971 (time 1709788377.34 total 842.24 last 33.52) - open/close 28310 (time 1709788387.35 total 852.25 last 33.89) - open/close 28646 (time 1709788397.36 total 862.26 last 33.58) - open/close 28980 (time 1709788407.38 total 872.28 last 33.33) - open/close 29309 (time 1709788417.42 total 882.32 last 32.77) - open/close 29636 (time 1709788427.43 total 892.33 last 32.66) - open/close 29968 (time 1709788437.45 total 902.35 last 33.11) - open/close 30293 (time 1709788447.48 total 912.38 last 32.41) - open/close 30612 (time 1709788457.49 total 922.39 last 31.88) - open/close 30936 (time 1709788467.49 total 932.39 last 32.40) - open/close 31262 (time 1709788477.52 total 942.42 last 32.51) - open/close 31583 (time 1709788487.54 total 952.44 last 32.02) - open/close 31905 (time 1709788497.55 total 962.45 last 32.18) - open/close 32246 (time 1709788507.56 total 972.46 last 34.04) - open/close 32563 (time 1709788517.57 total 982.47 last 31.68) - open/close 32857 (time 1709788527.60 total 992.50 last 29.30) - open/close 33167 (time 1709788537.63 total 1002.53 last 30.90) - open/close 33475 (time 1709788547.64 total 1012.54 last 30.76) - open/close 33786 (time 1709788557.65 total 1022.55 last 31.09) - open/close 34091 (time 1709788567.67 total 1032.57 last 30.44) - open/close 34398 (time 1709788577.68 total 1042.58 last 30.66) - open/close 34701 (time 1709788587.68 total 1052.58 last 30.30) - open/close 35017 (time 1709788597.69 total 1062.59 last 31.57) - open/close 35327 (time 1709788607.71 total 1072.62 last 30.93) - open/close 35631 (time 1709788617.72 total 1082.62 last 30.39) - open/close 35948 (time 1709788627.73 total 1092.63 last 31.67) - open/close 36266 (time 1709788637.74 total 1102.64 last 31.77) - open/close 36581 (time 1709788647.75 total 1112.65 last 31.45) - open/close 36891 (time 1709788657.78 total 1122.68 last 30.90) - open/close 37195 (time 1709788667.78 total 1132.68 last 30.40) - open/close 37499 (time 1709788677.81 total 1142.71 last 30.32) - open/close 37813 (time 1709788687.84 total 1152.74 last 31.32) - open/close 38123 (time 1709788697.85 total 1162.75 last 30.96) - open/close 38422 (time 1709788707.87 total 1172.77 last 29.83) - open/close 38698 (time 1709788717.88 total 1182.78 last 27.57) - open/close 38977 (time 1709788727.92 total 1192.82 last 27.80) - open/close 39258 (time 1709788737.93 total 1202.83 last 28.07) - open/close 39541 (time 1709788747.93 total 1212.83 last 28.30) - open/close 39830 (time 1709788757.95 total 1222.86 last 28.82) - open/close 40000 (time 1709788764.10 total 1229.00 last 27.67) - open/close 40288 (time 1709788774.13 total 1239.03 last 28.71) - open/close 40551 (time 1709788784.15 total 1249.06 last 26.23) - open/close 40820 (time 1709788794.16 total 1259.06 last 26.88) - open/close 41079 (time 1709788804.19 total 1269.09 last 25.82) - open/close 41336 (time 1709788814.24 total 1279.14 last 25.59) - open/close 41592 (time 1709788824.26 total 1289.16 last 25.53) - open/close 41856 (time 1709788834.29 total 1299.19 last 26.32) - open/close 42109 (time 1709788844.32 total 1309.22 last 25.23) - open/close 42364 (time 1709788854.35 total 1319.25 last 25.43) - open/close 42625 (time 1709788864.39 total 1329.29 last 25.99) - open/close 42887 (time 1709788874.43 total 1339.33 last 26.10) - open/close 43141 (time 1709788884.45 total 1349.35 last 25.34) - open/close 43387 (time 1709788894.49 total 1359.39 last 24.51) - open/close 43635 (time 1709788904.51 total 1369.41 last 24.74) - open/close 43889 (time 1709788914.55 total 1379.45 last 25.30) - open/close 44127 (time 1709788924.57 total 1389.47 last 23.76) - open/close 44368 (time 1709788934.59 total 1399.49 last 24.04) - open/close 44608 (time 1709788944.61 total 1409.51 last 23.96) - open/close 44847 (time 1709788954.61 total 1419.51 last 23.88) - open/close 45085 (time 1709788964.65 total 1429.55 last 23.73) - open/close 45317 (time 1709788974.66 total 1439.56 last 23.17) - open/close 45549 (time 1709788984.68 total 1449.58 last 23.15) - open/close 45780 (time 1709788994.71 total 1459.61 last 23.04) - open/close 46015 (time 1709789004.75 total 1469.65 last 23.41) - open/close 46241 (time 1709789014.78 total 1479.68 last 22.53) - open/close 46468 (time 1709789024.80 total 1489.70 last 22.65) - open/close 46696 (time 1709789034.82 total 1499.72 last 22.77) - open/close 46937 (time 1709789044.84 total 1509.74 last 24.05) - open/close 47184 (time 1709789054.86 total 1519.76 last 24.64) - open/close 47427 (time 1709789064.88 total 1529.78 last 24.24) - open/close 47674 (time 1709789074.91 total 1539.81 last 24.62) - open/close 47919 (time 1709789084.92 total 1549.82 last 24.48) - open/close 48143 (time 1709789094.95 total 1559.85 last 22.35) - open/close 48366 (time 1709789104.95 total 1569.85 last 22.29) - open/close 48593 (time 1709789114.99 total 1579.89 last 22.61) - open/close 48832 (time 1709789125.03 total 1589.93 last 23.80) - open/close 49071 (time 1709789135.05 total 1599.95 last 23.86) - open/close 49300 (time 1709789145.08 total 1609.98 last 22.83) - open/close 49538 (time 1709789155.09 total 1619.99 last 23.78) - open/close 49777 (time 1709789165.12 total 1630.02 last 23.83) total: 50000 open/close in 1639.13 seconds: 30.50 ops/second - unlinked 0 (time 1709789175 ; total 0 ; last 0) - unlinked 10000 (time 1709789690 ; total 515 ; last 515) - unlinked 20000 (time 1709790077 ; total 902 ; last 387) - unlinked 30000 (time 1709790278 ; total 1103 ; last 201) - unlinked 40000 (time 1709790339 ; total 1164 ; last 61) total: 50000 unlinks in 1185 seconds: 42.194092 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 PASS 101a (2885s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg434-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg434-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg434-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg434-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:46:19 (1709790379) start mds service on oleg434-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec oleg434-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6062000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6062000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 PASS 101b (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg434-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg434-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3735 sec ========== 00:48:17 (1709790497) === conf-sanity: start cleanup 00:48:17 (1709790497) === === conf-sanity: finish cleanup 00:48:17 (1709790497) ===