-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:01 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:26 (1709786486) start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m45s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012c840800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012c840800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:29.000000000 -0500 Modify: 2024-03-06 23:42:29.000000000 -0500 Change: 2024-03-06 23:42:29.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 82 lines, 82 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server PASS 35b (172s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:44:20 (1709786660) start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server 000000 2 000008 oleg229-server: 8+0 records in oleg229-server: 8+0 records out oleg229-server: 4096 bytes (4.1 kB) copied, 0.0046864 s, 874 kB/s start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server stop mds service on oleg229-server unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 PASS 38 (138s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg229-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg229-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg229-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg229-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:46:40 (1709786800) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg229-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg229-server: oleg229-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg229-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server PASS 51 (252s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:57 (1709787057) start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1709787103.88 total 10.35 last 19.81) - open/close 423 (time 1709787114.48 total 20.94 last 20.57) - open/close 691 (time 1709787124.90 total 31.37 last 25.72) - open/close 822 (time 1709787134.92 total 41.38 last 13.08) - open/close 1029 (time 1709787145.72 total 52.19 last 19.16) - open/close 1228 (time 1709787156.74 total 63.20 last 18.06) - open/close 1487 (time 1709787166.74 total 73.21 last 25.89) - open/close 1640 (time 1709787177.19 total 83.65 last 14.65) - open/close 1723 (time 1709787187.42 total 93.89 last 8.11) - open/close 1916 (time 1709787198.40 total 104.87 last 17.58) - open/close 2123 (time 1709787208.42 total 114.88 last 20.67) - open/close 2352 (time 1709787218.42 total 124.89 last 22.88) - open/close 2599 (time 1709787228.54 total 135.00 last 24.42) - open/close 2700 (time 1709787239.20 total 145.67 last 9.47) - open/close 2883 (time 1709787249.41 total 155.87 last 17.93) - open/close 2996 (time 1709787259.70 total 166.16 last 10.98) - open/close 3276 (time 1709787269.71 total 176.18 last 27.96) - open/close 3512 (time 1709787280.48 total 186.95 last 21.92) - open/close 3679 (time 1709787290.56 total 197.03 last 16.57) - open/close 3844 (time 1709787301.29 total 207.76 last 15.37) - open/close 4045 (time 1709787311.46 total 217.93 last 19.77) - open/close 4246 (time 1709787321.48 total 227.95 last 20.07) - open/close 4353 (time 1709787333.25 total 239.72 last 9.09) - open/close 4542 (time 1709787343.26 total 249.72 last 18.89) - open/close 4653 (time 1709787359.56 total 266.03 last 6.81) - open/close 4811 (time 1709787371.18 total 277.64 last 13.61) - open/close 4974 (time 1709787381.92 total 288.38 last 15.18) - open/close 5149 (time 1709787392.60 total 299.06 last 16.38) - open/close 5328 (time 1709787402.61 total 309.08 last 17.87) - open/close 5491 (time 1709787413.09 total 319.56 last 15.55) - open/close 5690 (time 1709787423.10 total 329.57 last 19.88) - open/close 5815 (time 1709787433.64 total 340.10 last 11.87) - open/close 5958 (time 1709787443.64 total 350.11 last 14.30) - open/close 6205 (time 1709787453.64 total 360.11 last 24.69) - open/close 6314 (time 1709787463.93 total 370.40 last 10.59) - open/close 6529 (time 1709787474.12 total 380.59 last 21.11) - open/close 6708 (time 1709787484.14 total 390.61 last 17.86) - open/close 6899 (time 1709787494.17 total 400.63 last 19.05) - open/close 7112 (time 1709787504.39 total 410.86 last 20.83) - open/close 7232 (time 1709787514.80 total 421.26 last 11.54) - open/close 7397 (time 1709787524.82 total 431.28 last 16.47) - open/close 7530 (time 1709787534.86 total 441.32 last 13.24) - open/close 7707 (time 1709787545.61 total 452.08 last 16.46) - open/close 7878 (time 1709787555.62 total 462.09 last 17.09) - open/close 8017 (time 1709787565.63 total 472.09 last 13.89) - open/close 8186 (time 1709787576.14 total 482.61 last 16.07) - open/close 8307 (time 1709787586.15 total 492.62 last 12.09) - open/close 8450 (time 1709787596.17 total 502.63 last 14.27) - open/close 8489 (time 1709787608.28 total 514.74 last 3.22) - open/close 8680 (time 1709787618.32 total 524.78 last 19.02) - open/close 8898 (time 1709787628.41 total 534.88 last 21.60) - open/close 9136 (time 1709787639.42 total 545.89 last 21.61) - open/close 9260 (time 1709787649.43 total 555.90 last 12.38) - open/close 9458 (time 1709787659.46 total 565.92 last 19.75) - open/close 9671 (time 1709787669.54 total 576.00 last 21.13) - open/close 9827 (time 1709787679.55 total 586.01 last 15.59) - open/close 10000 (time 1709787683.71 total 590.17 last 41.61) - open/close 10535 (time 1709787693.72 total 600.18 last 53.45) - open/close 11048 (time 1709787703.73 total 610.20 last 51.23) - open/close 11539 (time 1709787713.74 total 620.21 last 49.04) - open/close 11730 (time 1709787729.54 total 636.01 last 12.09) - open/close 12181 (time 1709787739.54 total 646.01 last 45.09) - open/close 12628 (time 1709787749.58 total 656.05 last 44.52) - open/close 13077 (time 1709787759.60 total 666.07 last 44.81) - open/close 13556 (time 1709787769.63 total 676.09 last 47.79) - open/close 13956 (time 1709787779.64 total 686.11 last 39.94) - open/close 14408 (time 1709787789.66 total 696.12 last 45.13) - open/close 14846 (time 1709787799.67 total 706.13 last 43.77) - open/close 15273 (time 1709787809.67 total 716.14 last 42.67) - open/close 15677 (time 1709787819.68 total 726.15 last 40.37) - open/close 16087 (time 1709787829.69 total 736.16 last 40.96) - open/close 16493 (time 1709787839.70 total 746.17 last 40.56) - open/close 16848 (time 1709787849.77 total 756.24 last 35.25) - open/close 17179 (time 1709787859.79 total 766.25 last 33.05) - open/close 17541 (time 1709787869.80 total 776.26 last 36.15) - open/close 17890 (time 1709787879.81 total 786.28 last 34.84) - open/close 18240 (time 1709787889.83 total 796.29 last 34.95) - open/close 18575 (time 1709787899.84 total 806.31 last 33.45) - open/close 18922 (time 1709787909.85 total 816.32 last 34.66) - open/close 19299 (time 1709787919.89 total 826.35 last 37.59) - open/close 19617 (time 1709787929.91 total 836.37 last 31.73) - open/close 20000 (time 1709787939.65 total 846.12 last 39.30) - open/close 20393 (time 1709787949.67 total 856.13 last 39.26) - open/close 20819 (time 1709787959.68 total 866.15 last 42.52) - open/close 21231 (time 1709787969.69 total 876.16 last 41.16) - open/close 21613 (time 1709787979.70 total 886.17 last 38.17) - open/close 22012 (time 1709787989.71 total 896.17 last 39.88) - open/close 22410 (time 1709787999.72 total 906.19 last 39.74) - open/close 22857 (time 1709788009.73 total 916.20 last 44.65) - open/close 23321 (time 1709788019.73 total 926.20 last 46.40) - open/close 23818 (time 1709788029.75 total 936.22 last 49.61) - open/close 24304 (time 1709788039.75 total 946.22 last 48.59) - open/close 24769 (time 1709788049.76 total 956.22 last 46.48) - open/close 25244 (time 1709788059.77 total 966.23 last 47.46) - open/close 25691 (time 1709788069.78 total 976.25 last 44.62) - open/close 26114 (time 1709788079.79 total 986.25 last 42.29) - open/close 26548 (time 1709788089.80 total 996.27 last 43.34) - open/close 26970 (time 1709788099.81 total 1006.28 last 42.14) - open/close 27381 (time 1709788109.82 total 1016.29 last 41.06) - open/close 27792 (time 1709788119.83 total 1026.29 last 41.08) - open/close 28204 (time 1709788129.84 total 1036.30 last 41.17) - open/close 28612 (time 1709788139.85 total 1046.31 last 40.74) - open/close 29009 (time 1709788149.87 total 1056.33 last 39.62) - open/close 29420 (time 1709788159.87 total 1066.34 last 41.09) - open/close 29825 (time 1709788169.89 total 1076.36 last 40.41) - open/close 30000 (time 1709788174.25 total 1080.71 last 40.18) - open/close 30394 (time 1709788184.26 total 1090.73 last 39.35) - open/close 30772 (time 1709788194.27 total 1100.73 last 37.77) - open/close 31135 (time 1709788204.28 total 1110.75 last 36.24) - open/close 31520 (time 1709788214.30 total 1120.76 last 38.44) - open/close 31905 (time 1709788224.30 total 1130.77 last 38.48) - open/close 32274 (time 1709788234.32 total 1140.79 last 36.83) - open/close 32635 (time 1709788244.35 total 1150.81 last 36.02) - open/close 32992 (time 1709788254.37 total 1160.83 last 35.63) - open/close 33339 (time 1709788264.39 total 1170.86 last 34.61) - open/close 33699 (time 1709788274.40 total 1180.86 last 35.98) - open/close 34037 (time 1709788284.42 total 1190.89 last 33.72) - open/close 34351 (time 1709788294.45 total 1200.91 last 31.33) - open/close 34678 (time 1709788304.45 total 1210.91 last 32.70) - open/close 34994 (time 1709788314.46 total 1220.93 last 31.56) - open/close 35301 (time 1709788324.47 total 1230.94 last 30.66) - open/close 35599 (time 1709788334.49 total 1240.95 last 29.76) - open/close 35893 (time 1709788344.53 total 1250.99 last 29.29) - open/close 36190 (time 1709788354.55 total 1261.01 last 29.64) - open/close 36495 (time 1709788364.56 total 1271.03 last 30.45) - open/close 36786 (time 1709788374.57 total 1281.04 last 29.06) - open/close 37078 (time 1709788384.59 total 1291.06 last 29.15) - open/close 37364 (time 1709788394.62 total 1301.09 last 28.51) - open/close 37665 (time 1709788404.63 total 1311.10 last 30.07) - open/close 37978 (time 1709788414.66 total 1321.12 last 31.22) - open/close 38288 (time 1709788424.67 total 1331.13 last 30.98) - open/close 38613 (time 1709788434.68 total 1341.14 last 32.47) - open/close 38940 (time 1709788444.69 total 1351.16 last 32.65) - open/close 39263 (time 1709788454.70 total 1361.17 last 32.25) - open/close 39546 (time 1709788464.71 total 1371.17 last 28.29) - open/close 39850 (time 1709788474.72 total 1381.19 last 30.36) - open/close 40000 (time 1709788479.55 total 1386.01 last 31.07) - open/close 40320 (time 1709788489.55 total 1396.02 last 32.00) - open/close 40649 (time 1709788499.56 total 1406.03 last 32.87) - open/close 40978 (time 1709788509.57 total 1416.04 last 32.85) - open/close 41301 (time 1709788519.60 total 1426.06 last 32.22) - open/close 41617 (time 1709788529.61 total 1436.08 last 31.56) - open/close 41940 (time 1709788539.62 total 1446.09 last 32.27) - open/close 42261 (time 1709788549.63 total 1456.10 last 32.06) - open/close 42575 (time 1709788559.64 total 1466.11 last 31.37) - open/close 42886 (time 1709788569.66 total 1476.12 last 31.06) - open/close 43189 (time 1709788579.68 total 1486.14 last 30.24) - open/close 43492 (time 1709788589.68 total 1496.15 last 30.28) - open/close 43802 (time 1709788599.68 total 1506.15 last 30.99) - open/close 44112 (time 1709788609.72 total 1516.18 last 30.90) - open/close 44420 (time 1709788619.73 total 1526.19 last 30.76) - open/close 44726 (time 1709788629.74 total 1536.21 last 30.55) - open/close 45015 (time 1709788639.76 total 1546.23 last 28.85) - open/close 45288 (time 1709788649.78 total 1556.24 last 27.26) - open/close 45565 (time 1709788659.80 total 1566.27 last 27.64) - open/close 45862 (time 1709788669.81 total 1576.27 last 29.67) - open/close 46133 (time 1709788679.82 total 1586.28 last 27.07) - open/close 46419 (time 1709788689.83 total 1596.30 last 28.56) - open/close 46705 (time 1709788699.83 total 1606.30 last 28.60) - open/close 46990 (time 1709788709.87 total 1616.33 last 28.40) - open/close 47282 (time 1709788719.90 total 1626.36 last 29.12) - open/close 47574 (time 1709788729.92 total 1636.39 last 29.12) - open/close 47862 (time 1709788739.94 total 1646.41 last 28.75) - open/close 48152 (time 1709788749.98 total 1656.44 last 28.89) - open/close 48436 (time 1709788759.99 total 1666.46 last 28.35) - open/close 48726 (time 1709788770.02 total 1676.48 last 28.93) - open/close 49003 (time 1709788780.02 total 1686.48 last 27.69) - open/close 49278 (time 1709788790.02 total 1696.49 last 27.50) - open/close 49557 (time 1709788800.04 total 1706.51 last 27.84) - open/close 49835 (time 1709788810.06 total 1716.53 last 27.74) total: 50000 open/close in 1722.69 seconds: 29.02 ops/second - unlinked 0 (time 1709788817 ; total 0 ; last 0) - unlinked 10000 (time 1709789242 ; total 425 ; last 425) - unlinked 20000 (time 1709789597 ; total 780 ; last 355) - unlinked 30000 (time 1709789849 ; total 1032 ; last 252) - unlinked 40000 (time 1709789981 ; total 1164 ; last 132) total: 50000 unlinks in 1211 seconds: 41.288193 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 PASS 101a (2992s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg229-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg229-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg229-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg229-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:40:51 (1709790051) start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec oleg229-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a84b2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a84b2000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 PASS 101b (91s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg229-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg229-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3763 sec ========== 00:42:44 (1709790164) === conf-sanity: start cleanup 00:42:45 (1709790165) === === conf-sanity: finish cleanup 00:42:45 (1709790165) ===