-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:47:13 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg251-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:48:26 (1710100106) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server PASS 21a (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:48:54 (1710100134) start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server PASS 21b (212s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:52:28 (1710100348) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880000090000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff880000090000.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 PASS 22 (76s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg251-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:53:48 (1710100428) Testing with 2 OSTs Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9d27800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9d27800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000401 0 34 0x22 0x280000401 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:55:23.000000000 -0400 Modify: 2024-03-10 15:55:23.000000000 -0400 Change: 2024-03-10 15:55:23.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server stop ost1 service on oleg251-server stop mds service on oleg251-server stop mds service on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server PASS 46a (188s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:56:59 (1710100619) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ae4be000.lazystatfs=1 stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server oleg251-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800ae4be000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ae4be000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.202.151@tcp cb49abc9-f7a4-44ed-ae28-bc3bbe693a94 5 1 UP lov lustre-clilov-ffff8800ae4be000 6cd8cc27-1940-4b67-9f57-38df11c1d36c 4 2 UP lmv lustre-clilmv-ffff8800ae4be000 6cd8cc27-1940-4b67-9f57-38df11c1d36c 5 3 UP mdc lustre-MDT0000-mdc-ffff8800ae4be000 6cd8cc27-1940-4b67-9f57-38df11c1d36c 5 4 UP mdc lustre-MDT0001-mdc-ffff8800ae4be000 6cd8cc27-1940-4b67-9f57-38df11c1d36c 5 5 UP osc lustre-OST0000-osc-ffff8800ae4be000 6cd8cc27-1940-4b67-9f57-38df11c1d36c 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7492 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] filesystem_summary: 767232 6272 756864 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (1651) - No such process umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server PASS 50b (60s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:58:01 (1710100681) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg251-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b1cfc000.lazystatfs=1 stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server oleg251-server: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7492 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0001_UUID 382976 4096 376832 2% /mnt/lustre[OST:1] filesystem_summary: 382976 4096 376832 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (4434) - No such process umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg251-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq oleg251-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg251-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server PASS 50c (102s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:59:45 (1710100785) Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server oleg251-server: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6d3b000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7492 start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server PASS 50e (152s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:02:20 (1710100940) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg251-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg251-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server oleg251-server: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012e0d9800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7492 start ost2 service on oleg251-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg251-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg251-server: oleg251-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg251-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server PASS 50f (219s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:06:03 (1710101163) start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 951 (time 1710101210.69 total 10.01 last 95.05) - open/close 1901 (time 1710101220.69 total 20.01 last 95.00) - open/close 2691 (time 1710101230.69 total 30.01 last 78.98) - open/close 3466 (time 1710101240.70 total 40.02 last 77.44) - open/close 4237 (time 1710101250.72 total 50.04 last 76.93) - open/close 4995 (time 1710101260.72 total 60.04 last 75.77) - open/close 5713 (time 1710101270.73 total 70.05 last 71.71) - open/close 6402 (time 1710101280.75 total 80.07 last 68.79) - open/close 7066 (time 1710101290.76 total 90.08 last 66.31) - open/close 7637 (time 1710101300.77 total 100.09 last 57.04) - open/close 8143 (time 1710101310.79 total 110.11 last 50.53) - open/close 8683 (time 1710101320.81 total 120.13 last 53.90) - open/close 9247 (time 1710101330.82 total 130.13 last 56.36) - open/close 9751 (time 1710101340.82 total 140.14 last 50.36) total: 10000 open/close in 145.76 seconds: 68.61 ops/second - unlinked 0 (time 1710101348 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 1059 (time 1710101455.16 total 10.01 last 105.82) - open/close 1980 (time 1710101465.16 total 20.01 last 92.09) - open/close 2850 (time 1710101475.16 total 30.01 last 87.00) - open/close 3658 (time 1710101485.17 total 40.02 last 80.71) - open/close 4398 (time 1710101495.18 total 50.03 last 73.95) - open/close 5137 (time 1710101505.18 total 60.03 last 73.87) - open/close 5763 (time 1710101515.19 total 70.04 last 62.55) - open/close 6398 (time 1710101525.20 total 80.05 last 63.45) - open/close 7080 (time 1710101535.21 total 90.06 last 68.12) - open/close 7744 (time 1710101545.22 total 100.07 last 66.35) - open/close 8383 (time 1710101555.22 total 110.07 last 63.88) - open/close 8976 (time 1710101565.23 total 120.08 last 59.23) - open/close 9609 (time 1710101575.26 total 130.11 last 63.15) - open/close 9777 (time 1710101589.59 total 144.44 last 11.72) total: 10000 open/close in 148.38 seconds: 67.39 ops/second - unlinked 0 (time 1710101595 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 1076 (time 1710101697.55 total 10.01 last 107.51) - open/close 2034 (time 1710101707.55 total 20.01 last 95.80) - open/close 2039 (time 1710101718.63 total 31.09 last 0.45) - open/close 2877 (time 1710101728.63 total 41.09 last 83.78) - open/close 3748 (time 1710101738.64 total 51.09 last 87.06) - open/close 4304 (time 1710101757.34 total 69.80 last 29.72) - open/close 5031 (time 1710101767.36 total 79.81 last 72.61) - open/close 5811 (time 1710101777.36 total 89.82 last 77.99) - open/close 6519 (time 1710101787.38 total 99.83 last 70.67) - open/close 7167 (time 1710101797.38 total 109.84 last 64.78) - open/close 7786 (time 1710101807.38 total 119.84 last 61.88) - open/close 8390 (time 1710101817.39 total 129.85 last 60.35) - open/close 8957 (time 1710101827.40 total 139.86 last 56.65) - open/close 9557 (time 1710101837.40 total 149.86 last 60.00) total: 10000 open/close in 157.31 seconds: 63.57 ops/second - unlinked 0 (time 1710101847 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 965 (time 1710101956.34 total 10.00 last 96.46) - open/close 1786 (time 1710101966.34 total 20.01 last 82.07) - open/close 2658 (time 1710101976.35 total 30.02 last 87.11) - open/close 3394 (time 1710101986.36 total 40.02 last 73.56) - open/close 4204 (time 1710101996.37 total 50.03 last 80.93) - open/close 4918 (time 1710102006.37 total 60.03 last 71.38) - open/close 5639 (time 1710102016.38 total 70.05 last 72.02) - open/close 6286 (time 1710102026.39 total 80.05 last 64.66) - open/close 6903 (time 1710102036.39 total 90.05 last 61.68) - open/close 7557 (time 1710102046.40 total 100.07 last 65.32) - open/close 8172 (time 1710102056.42 total 110.08 last 61.40) - open/close 8753 (time 1710102066.42 total 120.09 last 58.09) - open/close 9368 (time 1710102076.44 total 130.10 last 61.41) - open/close 9975 (time 1710102086.45 total 140.12 last 60.61) total: 10000 open/close in 140.58 seconds: 71.14 ops/second - unlinked 0 (time 1710102088 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 883 (time 1710102212.05 total 19.80 last 44.59) - open/close 1820 (time 1710102222.05 total 29.81 last 93.66) - open/close 2807 (time 1710102232.07 total 39.82 last 98.59) - open/close 3582 (time 1710102242.08 total 49.83 last 77.40) - open/close 4457 (time 1710102252.09 total 59.84 last 87.41) - open/close 5239 (time 1710102262.10 total 69.85 last 78.14) - open/close 5930 (time 1710102272.10 total 79.85 last 69.08) - open/close 6601 (time 1710102282.11 total 89.86 last 67.03) - open/close 7239 (time 1710102292.12 total 99.87 last 63.74) - open/close 7883 (time 1710102302.13 total 109.88 last 64.33) - open/close 8546 (time 1710102312.14 total 119.89 last 66.26) - open/close 9161 (time 1710102322.14 total 129.89 last 61.49) - open/close 9741 (time 1710102332.16 total 139.91 last 57.89) total: 10000 open/close in 144.37 seconds: 69.27 ops/second - unlinked 0 (time 1710102338 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 922 (time 1710102442.14 total 19.54 last 47.19) - open/close 1942 (time 1710102452.15 total 29.55 last 101.86) - open/close 2883 (time 1710102462.16 total 39.56 last 94.03) - open/close 3732 (time 1710102472.17 total 49.57 last 84.82) - open/close 4575 (time 1710102482.17 total 59.57 last 84.28) - open/close 5421 (time 1710102492.17 total 69.57 last 84.59) - open/close 6223 (time 1710102502.17 total 79.57 last 80.17) - open/close 6948 (time 1710102512.18 total 89.58 last 72.45) - open/close 7686 (time 1710102522.19 total 99.59 last 73.75) - open/close 8378 (time 1710102532.19 total 109.59 last 69.17) - open/close 9059 (time 1710102542.20 total 119.60 last 68.06) - open/close 9700 (time 1710102552.25 total 129.65 last 63.77) total: 10000 open/close in 134.33 seconds: 74.45 ops/second - unlinked 0 (time 1710102558 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1036 (time 1710102651.76 total 10.01 last 103.52) - open/close 2016 (time 1710102661.76 total 20.01 last 97.97) - open/close 2936 (time 1710102671.77 total 30.02 last 91.92) - open/close 3869 (time 1710102681.77 total 40.02 last 93.26) - open/close 4712 (time 1710102691.78 total 50.03 last 84.24) - open/close 5638 (time 1710102701.78 total 60.04 last 92.56) - open/close 6400 (time 1710102711.79 total 70.05 last 76.12) - open/close 7151 (time 1710102721.80 total 80.05 last 75.08) - open/close 7847 (time 1710102731.80 total 90.06 last 69.54) - open/close 8575 (time 1710102741.81 total 100.06 last 72.75) - open/close 9200 (time 1710102751.83 total 110.08 last 62.38) - open/close 9865 (time 1710102761.85 total 120.10 last 66.40) total: 10000 open/close in 122.13 seconds: 81.88 ops/second - unlinked 0 (time 1710102765 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 1299 (time 1710102850.57 total 10.00 last 129.84) - open/close 2445 (time 1710102860.58 total 20.01 last 114.55) - open/close 3632 (time 1710102870.59 total 30.02 last 118.56) - open/close 4764 (time 1710102880.59 total 40.02 last 113.16) - open/close 5792 (time 1710102890.59 total 50.03 last 102.78) - open/close 6784 (time 1710102900.60 total 60.03 last 99.17) - open/close 7738 (time 1710102910.60 total 70.03 last 95.38) - open/close 8718 (time 1710102920.61 total 80.04 last 97.94) - open/close 9659 (time 1710102930.61 total 90.04 last 94.08) total: 10000 open/close in 93.84 seconds: 106.56 ops/second - unlinked 0 (time 1710102935 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 2630 (time 1710102996.02 total 10.00 last 262.99) - open/close 4558 (time 1710103006.03 total 20.00 last 192.71) - open/close 6275 (time 1710103016.03 total 30.01 last 171.66) - open/close 7690 (time 1710103026.03 total 40.01 last 141.48) - open/close 9024 (time 1710103036.03 total 50.01 last 133.39) total: 10000 open/close in 57.31 seconds: 174.50 ops/second - unlinked 0 (time 1710103044 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 3206 (time 1710103089.20 total 10.00 last 320.50) - open/close 5842 (time 1710103099.20 total 20.01 last 263.54) - open/close 8168 (time 1710103109.20 total 30.01 last 232.53) total: 10000 open/close in 37.75 seconds: 264.87 ops/second - unlinked 0 (time 1710103117 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.151@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.151@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre On OST0, 20413 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 PASS 69 (2011s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg251-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:39:35 (1710103175) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg251-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg251-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3165 sec ========== 16:39:58 (1710103198) === conf-sanity: start cleanup 16:39:58 (1710103198) === === conf-sanity: finish cleanup 16:39:58 (1710103198) ===