-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:41:13 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg315-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg315-server oleg315-server: oleg315-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:42:50 (1709786570) start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h43m26s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012bd86800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012bd86800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:44:14.000000000 -0500 Modify: 2024-03-06 23:44:14.000000000 -0500 Change: 2024-03-06 23:44:14.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 66 lines, 66 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local modules unloaded. oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 oleg315-server: oleg315-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server PASS 35b (207s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:46:19 (1709786779) start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server 000000 2 000008 oleg315-server: 8+0 records in oleg315-server: 8+0 records out oleg315-server: 4096 bytes (4.1 kB) copied, 0.00773807 s, 529 kB/s start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server stop mds service on oleg315-server unloading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 PASS 38 (132s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg315-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg315-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg315-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 oleg315-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg315-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:34 (1709786914) Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 oleg315-server: oleg315-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg315-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg315-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg315-server umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local modules unloaded. oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg315-server: oleg315-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg315-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 oleg315-server: oleg315-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server PASS 51 (256s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:54 (1709787174) start mds service on oleg315-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1709787223.32 total 10.90 last 22.58) - open/close 423 (time 1709787233.97 total 21.54 last 16.63) - open/close 671 (time 1709787243.97 total 31.54 last 24.79) - open/close 834 (time 1709787254.18 total 41.75 last 15.96) - open/close 1101 (time 1709787264.84 total 52.42 last 25.04) - open/close 1339 (time 1709787275.82 total 63.39 last 21.69) - open/close 1603 (time 1709787285.89 total 73.46 last 26.21) - open/close 1863 (time 1709787295.90 total 83.47 last 25.99) - open/close 2002 (time 1709787306.15 total 93.72 last 13.55) - open/close 2178 (time 1709787316.87 total 104.44 last 16.42) - open/close 2358 (time 1709787328.67 total 116.24 last 15.25) - open/close 2613 (time 1709787339.20 total 126.77 last 24.21) - open/close 2960 (time 1709787349.21 total 136.78 last 34.66) - open/close 3130 (time 1709787360.09 total 147.66 last 15.63) - open/close 3303 (time 1709787370.47 total 158.04 last 16.67) - open/close 3481 (time 1709787381.10 total 168.68 last 16.74) - open/close 3683 (time 1709787391.11 total 178.69 last 20.18) - open/close 3841 (time 1709787401.93 total 189.50 last 14.61) - open/close 4036 (time 1709787411.96 total 199.53 last 19.44) - open/close 4105 (time 1709787423.33 total 210.90 last 6.07) - open/close 4346 (time 1709787433.33 total 220.91 last 24.10) - open/close 4530 (time 1709787443.80 total 231.37 last 17.59) - open/close 4685 (time 1709787454.48 total 242.05 last 14.51) - open/close 4812 (time 1709787465.23 total 252.81 last 11.81) - open/close 5013 (time 1709787475.24 total 262.81 last 20.08) - open/close 5171 (time 1709787485.43 total 273.00 last 15.51) - open/close 5380 (time 1709787495.44 total 283.02 last 20.87) - open/close 5506 (time 1709787510.00 total 297.57 last 8.66) - open/close 5619 (time 1709787520.02 total 307.59 last 11.28) - open/close 5818 (time 1709787530.26 total 317.83 last 19.43) - open/close 6003 (time 1709787540.83 total 328.41 last 17.49) - open/close 6126 (time 1709787551.32 total 338.89 last 11.73) - open/close 6307 (time 1709787561.39 total 348.96 last 17.97) - open/close 6497 (time 1709787571.40 total 358.97 last 18.99) - open/close 6706 (time 1709787582.12 total 369.69 last 19.50) - open/close 6863 (time 1709787592.14 total 379.71 last 15.66) - open/close 7071 (time 1709787602.63 total 390.20 last 19.84) - open/close 7157 (time 1709787612.63 total 400.21 last 8.59) - open/close 7393 (time 1709787623.25 total 410.82 last 22.23) - open/close 7550 (time 1709787633.47 total 421.04 last 15.36) - open/close 7753 (time 1709787644.02 total 431.59 last 19.24) - open/close 7931 (time 1709787654.88 total 442.45 last 16.38) - open/close 8118 (time 1709787664.89 total 452.46 last 18.69) - open/close 8359 (time 1709787675.68 total 463.25 last 22.33) - open/close 8471 (time 1709787685.70 total 473.28 last 11.17) - open/close 8732 (time 1709787695.93 total 483.50 last 25.53) - open/close 8887 (time 1709787705.94 total 493.51 last 15.48) - open/close 9038 (time 1709787715.94 total 503.51 last 15.10) - open/close 9212 (time 1709787726.60 total 514.17 last 16.32) - open/close 9395 (time 1709787736.99 total 524.56 last 17.62) - open/close 9540 (time 1709787747.26 total 534.83 last 14.12) - open/close 9711 (time 1709787757.98 total 545.56 last 15.94) - open/close 9928 (time 1709787767.98 total 555.56 last 21.70) - open/close 10000 (time 1709787770.83 total 558.41 last 25.27) - open/close 10199 (time 1709787780.87 total 568.44 last 19.84) - open/close 10321 (time 1709787790.87 total 578.44 last 12.20) - open/close 10465 (time 1709787800.87 total 588.45 last 14.39) - open/close 10691 (time 1709787810.90 total 598.47 last 22.54) - open/close 10804 (time 1709787820.91 total 608.49 last 11.28) - open/close 11037 (time 1709787830.92 total 618.49 last 23.29) - open/close 11621 (time 1709787840.98 total 628.55 last 58.02) - open/close 12136 (time 1709787851.02 total 638.59 last 51.30) - open/close 12649 (time 1709787861.02 total 648.59 last 51.30) - open/close 13140 (time 1709787871.04 total 658.62 last 48.99) - open/close 13602 (time 1709787881.06 total 668.63 last 46.13) - open/close 14086 (time 1709787891.07 total 678.65 last 48.32) - open/close 14550 (time 1709787901.08 total 688.66 last 46.36) - open/close 15041 (time 1709787911.09 total 698.66 last 49.07) - open/close 15479 (time 1709787921.10 total 708.68 last 43.74) - open/close 15915 (time 1709787931.11 total 718.68 last 43.58) - open/close 16358 (time 1709787941.11 total 728.68 last 44.30) - open/close 16792 (time 1709787951.11 total 738.68 last 43.38) - open/close 17235 (time 1709787961.12 total 748.70 last 44.25) - open/close 17658 (time 1709787971.13 total 758.70 last 42.28) - open/close 18074 (time 1709787981.14 total 768.71 last 41.56) - open/close 18531 (time 1709787991.14 total 778.71 last 45.70) - open/close 18986 (time 1709788001.16 total 788.73 last 45.41) - open/close 19408 (time 1709788011.17 total 798.75 last 42.13) - open/close 19827 (time 1709788021.19 total 808.76 last 41.84) - open/close 20000 (time 1709788026.19 total 813.76 last 34.61) - open/close 20029 (time 1709788038.11 total 825.68 last 2.43) - open/close 20408 (time 1709788048.14 total 835.71 last 37.78) - open/close 20751 (time 1709788058.14 total 845.71 last 34.29) - open/close 21099 (time 1709788068.17 total 855.75 last 34.69) - open/close 21527 (time 1709788078.17 total 865.75 last 42.80) - open/close 22005 (time 1709788088.18 total 875.75 last 47.76) - open/close 22477 (time 1709788098.21 total 885.79 last 47.05) - open/close 22948 (time 1709788108.22 total 895.79 last 47.09) - open/close 23408 (time 1709788118.23 total 905.80 last 45.96) - open/close 23860 (time 1709788128.24 total 915.81 last 45.16) - open/close 24319 (time 1709788138.24 total 925.81 last 45.89) - open/close 24766 (time 1709788148.25 total 935.83 last 44.62) - open/close 25194 (time 1709788158.26 total 945.83 last 42.77) - open/close 25640 (time 1709788168.27 total 955.85 last 44.54) - open/close 26070 (time 1709788178.29 total 965.86 last 42.95) - open/close 26494 (time 1709788188.29 total 975.86 last 42.38) - open/close 26895 (time 1709788198.31 total 985.88 last 40.02) - open/close 27285 (time 1709788208.32 total 995.89 last 38.96) - open/close 27691 (time 1709788218.33 total 1005.90 last 40.59) - open/close 28110 (time 1709788228.33 total 1015.90 last 41.87) - open/close 28500 (time 1709788238.35 total 1025.92 last 38.94) - open/close 28871 (time 1709788248.38 total 1035.95 last 36.97) - open/close 29220 (time 1709788258.40 total 1045.97 last 34.83) - open/close 29602 (time 1709788268.41 total 1055.99 last 38.15) - open/close 29963 (time 1709788278.44 total 1066.01 last 36.02) - open/close 30324 (time 1709788288.46 total 1076.03 last 36.01) - open/close 30677 (time 1709788298.46 total 1086.04 last 35.29) - open/close 31029 (time 1709788308.48 total 1096.05 last 35.14) - open/close 31380 (time 1709788318.49 total 1106.07 last 35.05) - open/close 31736 (time 1709788328.52 total 1116.09 last 35.52) - open/close 32087 (time 1709788338.54 total 1126.11 last 35.01) - open/close 32439 (time 1709788348.54 total 1136.11 last 35.20) - open/close 32797 (time 1709788358.56 total 1146.13 last 35.74) - open/close 33115 (time 1709788368.57 total 1156.15 last 31.75) - open/close 33449 (time 1709788378.60 total 1166.18 last 33.30) - open/close 33785 (time 1709788388.60 total 1176.18 last 33.60) - open/close 34121 (time 1709788398.63 total 1186.20 last 33.52) - open/close 34451 (time 1709788408.63 total 1196.20 last 33.00) - open/close 34533 (time 1709788422.28 total 1209.85 last 6.01) - open/close 34841 (time 1709788432.31 total 1219.88 last 30.72) - open/close 35162 (time 1709788442.31 total 1229.88 last 32.09) - open/close 35470 (time 1709788452.34 total 1239.91 last 30.71) - open/close 35791 (time 1709788462.34 total 1249.91 last 32.09) - open/close 36105 (time 1709788472.37 total 1259.94 last 31.32) - open/close 36424 (time 1709788482.38 total 1269.95 last 31.88) - open/close 36731 (time 1709788492.39 total 1279.96 last 30.65) - open/close 37032 (time 1709788502.40 total 1289.98 last 30.07) - open/close 37339 (time 1709788512.43 total 1300.00 last 30.61) - open/close 37644 (time 1709788522.45 total 1310.02 last 30.45) - open/close 37954 (time 1709788532.47 total 1320.04 last 30.93) - open/close 38264 (time 1709788542.50 total 1330.07 last 30.92) - open/close 38573 (time 1709788552.52 total 1340.09 last 30.84) - open/close 38874 (time 1709788562.56 total 1350.13 last 29.98) - open/close 39181 (time 1709788572.56 total 1360.14 last 30.68) - open/close 39489 (time 1709788582.57 total 1370.15 last 30.77) - open/close 39794 (time 1709788592.58 total 1380.15 last 30.48) - open/close 40000 (time 1709788599.27 total 1386.85 last 30.76) - open/close 40303 (time 1709788609.28 total 1396.85 last 30.29) - open/close 40610 (time 1709788619.28 total 1406.86 last 30.69) - open/close 40921 (time 1709788629.30 total 1416.87 last 31.04) - open/close 41217 (time 1709788639.32 total 1426.89 last 29.55) - open/close 41529 (time 1709788649.33 total 1436.91 last 31.16) - open/close 41824 (time 1709788659.35 total 1446.92 last 29.46) - open/close 42122 (time 1709788669.35 total 1456.92 last 29.80) - open/close 42399 (time 1709788679.36 total 1466.93 last 27.67) - open/close 42674 (time 1709788689.39 total 1476.96 last 27.42) - open/close 42953 (time 1709788699.42 total 1486.99 last 27.82) - open/close 43234 (time 1709788709.43 total 1497.00 last 28.07) - open/close 43506 (time 1709788719.45 total 1507.03 last 27.13) - open/close 43793 (time 1709788729.45 total 1517.03 last 28.70) - open/close 44079 (time 1709788739.47 total 1527.04 last 28.57) - open/close 44366 (time 1709788749.47 total 1537.04 last 28.68) - open/close 44641 (time 1709788759.50 total 1547.08 last 27.42) - open/close 44902 (time 1709788769.51 total 1557.08 last 26.09) - open/close 45149 (time 1709788779.53 total 1567.10 last 24.65) - open/close 45408 (time 1709788789.55 total 1577.12 last 25.85) - open/close 45659 (time 1709788799.57 total 1587.15 last 25.03) - open/close 45908 (time 1709788809.60 total 1597.17 last 24.83) - open/close 46164 (time 1709788819.62 total 1607.19 last 25.56) - open/close 46411 (time 1709788829.64 total 1617.21 last 24.65) - open/close 46663 (time 1709788839.67 total 1627.24 last 25.11) - open/close 46911 (time 1709788849.69 total 1637.26 last 24.75) - open/close 47153 (time 1709788859.73 total 1647.31 last 24.10) - open/close 47395 (time 1709788869.74 total 1657.31 last 24.19) - open/close 47639 (time 1709788879.76 total 1667.33 last 24.34) - open/close 47897 (time 1709788889.79 total 1677.36 last 25.74) - open/close 48161 (time 1709788899.79 total 1687.36 last 26.39) - open/close 48408 (time 1709788909.82 total 1697.40 last 24.62) - open/close 48662 (time 1709788919.83 total 1707.40 last 25.38) - open/close 48920 (time 1709788929.86 total 1717.44 last 25.71) - open/close 49156 (time 1709788939.88 total 1727.46 last 23.55) - open/close 49395 (time 1709788949.90 total 1737.48 last 23.85) - open/close 49620 (time 1709788959.90 total 1747.48 last 22.50) - open/close 49858 (time 1709788969.92 total 1757.49 last 23.76) total: 50000 open/close in 1763.17 seconds: 28.36 ops/second - unlinked 0 (time 1709788976 ; total 0 ; last 0) - unlinked 10000 (time 1709789463 ; total 487 ; last 487) - unlinked 20000 (time 1709789891 ; total 915 ; last 428) - unlinked 30000 (time 1709790196 ; total 1220 ; last 305) - unlinked 40000 (time 1709790339 ; total 1363 ; last 143) total: 50000 unlinks in 1409 seconds: 35.486160 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 PASS 101a (3225s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg315-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg315-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg315-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 oleg315-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg315-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:46:40 (1709790400) start mds service on oleg315-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock oleg315-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server start ost1 service on oleg315-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6e1c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6e1c800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 PASS 101b (89s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg315-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg315-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 oleg315-server: oleg315-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4041 sec ========== 00:48:35 (1709790515) === conf-sanity: start cleanup 00:48:35 (1709790515) === === conf-sanity: finish cleanup 00:48:35 (1709790515) ===